Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev/adjust tcn coco saving #460

Merged
merged 43 commits into from
Oct 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
1989d18
change coco saving to use the annotation
josephvanpeltkw Oct 21, 2024
f3cf62a
Merge branch 'PTG-Kitware:master' into dev/adjust_tcn_coco_saving
josephvanpeltkw Oct 23, 2024
ccf8032
add a debug option to the TCN node in order to see the inputs it has …
josephvanpeltkw Oct 25, 2024
24b3080
adjust saving of coco output and add score
josephvanpeltkw Oct 25, 2024
a1e3bef
add collection of no activity classification
josephvanpeltkw Oct 25, 2024
6ef0d71
add a note about usage to the video/image to bag conversion script
josephvanpeltkw Oct 25, 2024
f242ea2
black formatting a clean copy error
josephvanpeltkw Oct 25, 2024
54e8104
check the frame number first before ignoring a frame (for playing bac…
josephvanpeltkw Oct 25, 2024
3650376
drop unneeded argument
josephvanpeltkw Oct 25, 2024
afec82d
change coco saving to add an image regardless of whether the activity…
josephvanpeltkw Oct 25, 2024
cc257d8
fix for beginning frame issue
josephvanpeltkw Oct 25, 2024
aec3eb1
simplify handling images that were already added
josephvanpeltkw Oct 25, 2024
4736376
cleanup debug input so it is easier to read in normal viewers
josephvanpeltkw Oct 28, 2024
81dc612
change the way that timestamps are saved to bags
josephvanpeltkw Oct 29, 2024
36d97d9
remove the need to check frame number now that converting videos to b…
josephvanpeltkw Oct 29, 2024
c40adfc
black formatting
josephvanpeltkw Oct 29, 2024
6105312
Incremental updates to train README for data and precursor data gener…
Purg Oct 21, 2024
412b8b7
Move TCN vectorization into the TCN package
Purg Oct 21, 2024
3d31631
Add colorlog dep to support TCN training
Purg Oct 22, 2024
c6a36f1
Minor dep order reorg, pin hydra-colorlog version like in tcn-hpl
Purg Oct 23, 2024
2f1c2a3
Updates to training README
Purg Oct 23, 2024
d8bff16
Fix archive extraction script
Purg Oct 23, 2024
502322e
Add final grep to probe helper script
Purg Oct 23, 2024
7fd0a26
Revert TensorRT Engine model usage
Purg Oct 24, 2024
21d4423
Update TCN related things to use common dataset and vector computation
Purg Oct 24, 2024
c2bc709
Update yolo v7 submodule for CLI updates
Purg Oct 25, 2024
1110b8c
Fix formatting
Purg Oct 25, 2024
ffb0d43
Finish documentation sentence
Purg Oct 25, 2024
7c4d693
Cache old configurations into an "old" directory for now
Purg Oct 25, 2024
65eb715
add a debug option to the TCN node in order to see the inputs it has …
josephvanpeltkw Oct 25, 2024
0839d1c
cleanup debug input so it is easier to read in normal viewers
josephvanpeltkw Oct 28, 2024
fd55d1f
Update TCN related things to use common dataset and vector computation
Purg Oct 24, 2024
59c5598
cleanup debug input so it is easier to read in normal viewers
josephvanpeltkw Oct 28, 2024
b6895f0
Update TCN related things to use common dataset and vector computation
Purg Oct 24, 2024
fe065ef
cleanup debug input so it is easier to read in normal viewers
josephvanpeltkw Oct 28, 2024
49319ff
Merge branch 'master' into dev/adjust_tcn_coco_saving
josephvanpeltkw Oct 29, 2024
64ae6c0
Merge branch 'PTG-Kitware:master' into dev/adjust_tcn_coco_saving
josephvanpeltkw Oct 29, 2024
7d5230e
fix for classes not in model
josephvanpeltkw Oct 29, 2024
ef38e74
Update ros/angel_system_nodes/angel_system_nodes/activity_classificat…
josephvanpeltkw Oct 30, 2024
a3ce112
remove unneeded code and change var name
josephvanpeltkw Oct 30, 2024
3441c18
change to collect image outside of try statement so it always collects
josephvanpeltkw Oct 30, 2024
c1f7fbb
Update ros/angel_system_nodes/angel_system_nodes/activity_classificat…
josephvanpeltkw Oct 30, 2024
db2820e
Update ros/angel_system_nodes/angel_system_nodes/activity_classificat…
josephvanpeltkw Oct 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 44 additions & 14 deletions angel_system/activity_classification/tcn_hpl/predict.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,36 +56,66 @@ def set_video(self, video_name: str) -> None:
else:
self._vid = self._dset.add_video(name=video_name)

def collect(
def add_image(
self,
frame_index: int,
activity_pred: int,
activity_conf_vec: Sequence[float],
name: Optional[str] = None,
file_name: Optional[str] = None,
activity_gt: Optional[int] = None,
) -> None:
) -> int:
"""
See `CocoDataset.add_image` for more details.
Add an image to the dataset. Returns the global image id.
If the image was already added (by name or file name), returns -1.
"""
with self._lock:
if self._vid is None:
raise RuntimeError(
"No video set before results collection. See `set_video` method."
)
packet = dict(

# get the global id for the image from the frame number
# add the image
img = dict(
video_id=self._vid,
frame_index=frame_index,
activity_pred=activity_pred,
activity_conf=list(activity_conf_vec),
)
if name is not None:
packet["name"] = name
img["name"] = name
if file_name is not None:
packet["file_name"] = file_name
if activity_gt is not None:
packet["activity_gt"] = activity_gt
self._dset.add_image(**packet)
img["file_name"] = file_name
# save the gid from the image to link to the annot
try:
gid = self._dset.add_image(**img)
except Exception:
return -1 # image already exists

return gid

def collect(
self,
gid: int,
activity_pred: int,
activity_conf_vec: Sequence[float],
) -> None:
"""
See `CocoDataset.add_image` for more details.

:param gid: Global image id.
:param activity_pred: Predicted activity class index.
:param activity_conf_vec: Confidence vector for all activity classes.
"""
with self._lock:
if self._vid is None:
raise RuntimeError(
"No video set before results collection. See `set_video` method."
)

# add the annotation
self._dset.add_annotation(
image_id=gid,
category_id=activity_pred,
score=activity_conf_vec[activity_pred],
prob=list(activity_conf_vec),
)

def write_file(self):
"""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from typing import Callable
from typing import List
from typing import Optional
import re
from typing import Tuple

import kwcoco
Expand Down Expand Up @@ -99,6 +100,10 @@
# activity prediction for the "live" image will not occur until object
# detections are predicted for that frame.
PARAM_WINDOW_LEADS_WITH_OBJECTS = "window_leads_with_objects"
# Debug file saved out to the filesystem for understanding the node's
# inputs when it decides not to create an activity classification.
# the format will be csv with a list of the object detections and the pose
PARAM_DEBUG_FILE = "debug_file"


class NoActivityClassification(Exception):
Expand Down Expand Up @@ -156,6 +161,7 @@ def __init__(self):
(PARAM_TOPIC, "medical"),
(PARAM_POSE_REPEAT_RATE, 0),
(PARAM_WINDOW_LEADS_WITH_OBJECTS, False),
(PARAM_DEBUG_FILE, ""),
],
)
self._img_ts_topic = param_values[PARAM_IMG_TS_TOPIC]
Expand Down Expand Up @@ -512,6 +518,7 @@ def _window_criterion_correct_size(self, window: InputBuffer) -> bool:
f"Window is not the appropriate size "
f"(actual:{len(window)} != {self._window_size}:expected)"
)

return window_ok

def _window_criterion_new_leading_frame(self, window: InputWindow) -> bool:
Expand Down Expand Up @@ -585,6 +592,9 @@ def rt_loop(self):
have_leading_object=self._window_lead_with_objects,
)

window_end_frame = window.frames[-1][0]
image_gid = self._collect_image(window_end_frame)

# log.info(f"buffer contents: {window.obj_dets}")

# if enable_time_trace_logging:
Expand Down Expand Up @@ -615,7 +625,7 @@ def rt_loop(self):
act_msg = self._process_window(window)
# log.info(f"activity message: {act_msg}")

self._collect_results(act_msg)
self._collect_results(act_msg, image_gid)
# set the header right before publishing so that the time is after processing
act_msg.header.frame_id = "Activity Classification"
act_msg.header.stamp = self.get_clock().now().to_msg()
Expand All @@ -628,6 +638,14 @@ def rt_loop(self):
"not yield an activity classification for "
"publishing."
)
if self._debug_file != "":
# save the info for why this window was not processed
repr = window.__repr__()
# clean this output for easier viewing (CSV)
repr = "index" + repr # add a column for the index
repr = re.sub(" +", ",", repr) # replace spaces with commas
with open(self._debug_file, "a") as f:
f.write(f"{repr}\n")

# This window has completed processing - record its leading
# timestamp now.
Expand Down Expand Up @@ -762,7 +780,7 @@ def _process_window(self, window: InputWindow) -> ActivityDetection:

return activity_msg

def _collect_results(self, msg: ActivityDetection):
def _collect_image(self, end_frame_time: Time) -> int:
"""
Collect into our ResultsCollector instance from the produced activity
classification message if we were initialized to do that.
Expand All @@ -777,11 +795,31 @@ def _collect_results(self, msg: ActivityDetection):
# Use window end timestamp nanoseconds as the frame index.
# When reading from an input COCO file, this aligns with the input
# `image` `frame_index` attributes.
frame_index = time_to_int(msg.source_stamp_end_frame)
pred_cls_idx = int(np.argmax(msg.conf_vec))
rc.collect(
frame_index = time_to_int(end_frame_time)
gid = rc.add_image(
frame_index=frame_index,
name=f"ros-frame-nsec-{frame_index}",
)
return gid
return -1

def _collect_results(self, msg: ActivityDetection, gid: int) -> None:
"""
Collect into our ResultsCollector instance from the produced activity
classification message if we were initialized to do that.

This method does nothing if this node has not been initialized to
collect results.

:param msg: ROS2 activity classification message that would be output.
:param gid: Global ID of the image associated with the activity
"""
rc = self._results_collector
if rc is not None:
# use the gid that was created when the image was added
pred_cls_idx = int(np.argmax(msg.conf_vec))
rc.collect(
gid=gid,
activity_pred=pred_cls_idx,
activity_conf_vec=list(msg.conf_vec),
)
Expand Down
25 changes: 17 additions & 8 deletions ros/angel_utils/scripts/convert_video_to_ros_bag.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
#!/usr/bin/env python3
"""
Convert a video (mp4) or a series of images into a ROS bag.

Example running (inside ROS environment):
ros2 run angel_utils convert_video_to_ros_bag.py \
--video-fn video.mp4 \
--output-bag-folder ros_bags/new_bag
"""
import argparse
from glob import glob
from pathlib import Path
Expand Down Expand Up @@ -104,31 +112,32 @@ def convert_video_to_bag(

# Starting at this so our first increment starts us at frame ID 0.
frame_id = -1
start_ts = rclpy.time.Time(nanoseconds=time.time_ns())
for frame, frame_rel_ts in frame_iter:
frame_id += 1
# Only proceed if we don't have a down-sample rate specified or if the
# current frame aligns with the down-sample rate.
if downsample_rate is not None and frame_id % downsample_rate != 0:
continue
print(f"==== FRAME {frame_id} ====")
# Create timestamp

frame_ts = start_ts + rclpy.duration.Duration(seconds=frame_rel_ts)
frame_ts_msg = frame_ts.to_msg()
print("timestamp", frame_ts)

# Create image message
image_msg = bridge.cv2_to_imgmsg(frame, encoding="bgr8")
image_msg.header.stamp = frame_ts_msg
# split the frame timestamp into sec and nsec
seconds = frame_rel_ts
nsec = int((seconds - int(seconds)) * 1_000_000_000)
seconds = int(seconds)
image_msg.header.stamp.sec = seconds
image_msg.header.stamp.nanosec = nsec
print(f"timestamp: {image_msg.header.stamp}")

image_msg.header.frame_id = "PVFramesBGR"

# Write to bag
try:
bag_writer.write(
output_image_topic,
serialize_message(image_msg),
frame_ts.nanoseconds,
image_msg.header.stamp.nanosec,
)
except Exception as err:
# Truncating the error message because it printed out the whole image_msg input
Expand Down
Loading