feat(multi_object_tracker): multi object input #6820

technolojin · 2024-04-16T00:37:46Z

Description

Enable multi_object_tracker subscribe multiple detected objects topics.

This PR contains the following components/updates.

Channel configuration: Provide all available sources of detections.
Input manager: Manage multiple detected object message channels.
Process timing algorithm: Operate the tracking process in an optimal set of objects to reduce latency.
Existence model: Model the existence probability of each detection source to understand the contributions of detections.
Debugger: Visualize association results and existence probability on each channel.

The following is an example of possible perception pipeline diagram (the PR do not contain changes of the pipeline).

Tests performed

multi_input_mot_test1.mp4

debug message
- topic: /perception/object_recognition/tracking/multi_object_tracker/debug/objects_markers
- type: Marker
- Lines: Association
- Cubic
  - Colored: detection position
  - White: tracked object position
- Detections (In this example) name, short name, color-code
  - Lidar Centterpoint, Lcp, blue
  - Detection by Tracker, dbT, green
  - Radar far, Rf, yellow
Tests on Autoware enviornments
Tests on the simulations provided by the Autoware tutorials were operated and the system worked.
- Rosbag replay simulation sample data https://autowarefoundation.github.io/autoware-documentation/main/tutorials/ad-hoc-simulation/rosbag-replay-simulation/
- AWSIM https://tier4.github.io/AWSIM/GettingStarted/QuickStartDemo/

Notes for reviewers

prior PR on autoware_launch.
This PR will be merged after the launcher is updated.

Interface changes

Additional configuration of input channel selection is added.

Effects on system behavior

Default value is single input.
The overall perception pipeline will work as same as before.

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

I've confirmed the contribution guidelines.
The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

The PR follows the pull request guidelines.
The PR has been properly tested.
The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

There are no open discussions or they are tracked via tickets.
The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

Signed-off-by: Taekjin LEE <[email protected]>

perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp

…ebugger/debug_object.hpp Co-authored-by: Shunsuke Miura <[email protected]> Signed-off-by: Taekjin LEE <[email protected]>

Signed-off-by: Taekjin LEE <[email protected]>

perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp

perception/multi_object_tracker/src/debugger/debugger.cpp

perception/multi_object_tracker/src/processor/input_manager.cpp

Signed-off-by: Taekjin LEE <[email protected]>

YoshiRi

LGTM with minor comments

perception/multi_object_tracker/src/processor/input_manager.cpp

perception/multi_object_tracker/src/tracker/model/tracker_base.cpp

Signed-off-by: Taekjin LEE <[email protected]>

wep21 · 2024-05-26T13:21:31Z

perception/multi_object_tracker/src/processor/input_manager.cpp

+  const rclcpp::Time & object_latest_time, const rclcpp::Time & object_oldest_time,
+  ObjectsList & objects_list)
+{
+  assert(object_latest_time.nanoseconds() > object_oldest_time.nanoseconds());


@YoshiRi @technolojin @yukkysaito @miursh When launching planning simulator, this assertion always occurs at the start time in my environment. I appreciate it if someone could check if this assertion is valid.

@wep21 Thanks.
@technolojin May I ask for confirmation?

@wep21 @yukkysaito
Thank you for your report. I will investigate it.

@wep21
I made a fix. #7127
However, I couldn't reproduce the situation with the planning simulator.

Can you test this modification and let me know if the issue is resolved?

* refactor: frequently used types, namespace Signed-off-by: Taekjin LEE <[email protected]> * test: multiple inputs Signed-off-by: Taekjin LEE <[email protected]> * feat: check latest measurement time Signed-off-by: Taekjin LEE <[email protected]> * feat: define input manager class Signed-off-by: Taekjin LEE <[email protected]> * feat: interval measures Signed-off-by: Taekjin LEE <[email protected]> * feat: store and sort inputs PoC Signed-off-by: Taekjin LEE <[email protected]> * chore: rename classes Signed-off-by: Taekjin LEE <[email protected]> * feat: object collector Signed-off-by: Taekjin LEE <[email protected]> * impl input manager, no subscribe Signed-off-by: Taekjin LEE <[email protected]> * fix: subscribe and trigger callback Signed-off-by: Taekjin LEE <[email protected]> * fix: subscriber and callbacks are working Signed-off-by: Taekjin LEE <[email protected]> * fix: callback object is fixed, tracker is working Signed-off-by: Taekjin LEE <[email protected]> * fix: get object time argument revise Signed-off-by: Taekjin LEE <[email protected]> * feat: back to periodic publish, analyze input latency and timings Signed-off-by: Taekjin LEE <[email protected]> * fix: enable timing debugger Signed-off-by: Taekjin LEE <[email protected]> * fix: separate object interval function Signed-off-by: Taekjin LEE <[email protected]> * feat: prepare message triggered process Signed-off-by: Taekjin LEE <[email protected]> * feat: trigger tracker by main message arrive Signed-off-by: Taekjin LEE <[email protected]> * chore: clean-up, set namespace Signed-off-by: Taekjin LEE <[email protected]> * feat: object lists with detector index Signed-off-by: Taekjin LEE <[email protected]> * feat: define input channel struct Signed-off-by: Taekjin LEE <[email protected]> * fix: define type for object list Signed-off-by: Taekjin LEE <[email protected]> * feat: add channel wise existence probability Signed-off-by: Taekjin LEE <[email protected]> * fix: relocate debugger Signed-off-by: Taekjin LEE <[email protected]> * fix: total existence logic change Signed-off-by: Taekjin LEE <[email protected]> * feat: publishing object debug info, need to fix marker id Signed-off-by: Taekjin LEE <[email protected]> * feat: indexing marker step 1 Signed-off-by: Taekjin LEE <[email protected]> * fix: uuid management Signed-off-by: Taekjin LEE <[email protected]> * feat: association line fix Signed-off-by: Taekjin LEE <[email protected]> * feat: print channel names Signed-off-by: Taekjin LEE <[email protected]> * feat: association lines are color-coded Signed-off-by: Taekjin LEE <[email protected]> * fix: association debug marker bugfix Signed-off-by: Taekjin LEE <[email protected]> * style(pre-commit): autofix Signed-off-by: Taekjin LEE <[email protected]> * feat: add option for debug marker Signed-off-by: Taekjin LEE <[email protected]> * feat: skip time statistics update in case of outlier Signed-off-by: Taekjin LEE <[email protected]> * feat: auto-tune latency band Signed-off-by: Taekjin LEE <[email protected]> * feat: pre-defined channels, select on launcher Signed-off-by: Taekjin LEE <[email protected]> * feat: add input channels Signed-off-by: Taekjin LEE <[email protected]> * fix: remove marker idx map Signed-off-by: Taekjin LEE <[email protected]> * fix: to do not miss the latest message of the target stream Signed-off-by: Taekjin LEE <[email protected]> * fix: remove priority, separate timing optimization Signed-off-by: Taekjin LEE <[email protected]> * fix: time interval bug fix Signed-off-by: Taekjin LEE <[email protected]> * chore: refactoring timing state update Signed-off-by: Taekjin LEE <[email protected]> * fix: set parameters optionally Signed-off-by: Taekjin LEE <[email protected]> * feat: revise object time range logic Signed-off-by: Taekjin LEE <[email protected]> * fix: launcher to set input channels Signed-off-by: Taekjin LEE <[email protected]> * fix: exempt spell check 'pointpainting' Signed-off-by: Taekjin LEE <[email protected]> * feat: remove expected interval Signed-off-by: Taekjin LEE <[email protected]> * feat: implement spawn switch Signed-off-by: Taekjin LEE <[email protected]> * fix: remove debug messages Signed-off-by: Taekjin LEE <[email protected]> * chore: update readme Signed-off-by: Taekjin LEE <[email protected]> * fix: change tentative object topic Signed-off-by: Taekjin LEE <[email protected]> * Revert "fix: remove debug messages" This reverts commit 725a49e. Signed-off-by: Taekjin LEE <[email protected]> * fix: reset times when jumps to past Signed-off-by: Taekjin LEE <[email protected]> * fix: check if interval is negative Signed-off-by: Taekjin LEE <[email protected]> * fix: missing config, default value Signed-off-by: Taekjin LEE <[email protected]> * fix: remove debug messages Signed-off-by: Taekjin LEE <[email protected]> * fix: change no-object message level Signed-off-by: Taekjin LEE <[email protected]> * Update perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp Co-authored-by: Shunsuke Miura <[email protected]> Signed-off-by: Taekjin LEE <[email protected]> * chore: Update copyright to uppercase Signed-off-by: Taekjin LEE <[email protected]> * chore: fix readme links to config files Signed-off-by: Taekjin LEE <[email protected]> * chore: move and rename uuid functions Signed-off-by: Taekjin LEE <[email protected]> * chore: fix debug topic to use node name Signed-off-by: Taekjin LEE <[email protected]> * chore: express meaning of threshold Signed-off-by: Taekjin LEE <[email protected]> * feat: revise decay rate, update function Signed-off-by: Taekjin LEE <[email protected]> * fix: define constants with explanation Signed-off-by: Taekjin LEE <[email protected]> * style(pre-commit): autofix --------- Signed-off-by: Taekjin LEE <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Shunsuke Miura <[email protected]>

technolojin self-assigned this Apr 16, 2024

github-actions bot added the component:perception Advanced sensor data processing and environment understanding. (auto-assigned) label Apr 16, 2024

technolojin force-pushed the feat/mot-multi-object-input branch from cad733b to d8d35cd Compare May 8, 2024 04:02

github-actions bot added the component:launch Launch files, scripts and initialization tools. (auto-assigned) label May 8, 2024

technolojin force-pushed the feat/mot-multi-object-input branch from d8d35cd to d0bd1b2 Compare May 9, 2024 08:24

technolojin linked an issue May 9, 2024 that may be closed by this pull request

Replace serialized detected object mergers to reduce complexity #6958

Closed

3 tasks

technolojin force-pushed the feat/mot-multi-object-input branch from 13dfe38 to 1f533f0 Compare May 13, 2024 06:33

github-actions bot added the type:documentation Creating or refining documentation. (auto-assigned) label May 13, 2024

technolojin mentioned this pull request May 14, 2024

feat(multi_object_tracker): add multi object input config file autowarefoundation/autoware_launch#989

Merged

4 tasks

technolojin added 21 commits May 14, 2024 15:03

refactor: frequently used types, namespace

5412a55

Signed-off-by: Taekjin LEE <[email protected]>

test: multiple inputs

4176cec

Signed-off-by: Taekjin LEE <[email protected]>

feat: check latest measurement time

96be0cc

Signed-off-by: Taekjin LEE <[email protected]>

feat: define input manager class

fd086dc

Signed-off-by: Taekjin LEE <[email protected]>

feat: interval measures

1462eef

Signed-off-by: Taekjin LEE <[email protected]>

feat: store and sort inputs PoC

7757212

Signed-off-by: Taekjin LEE <[email protected]>

chore: rename classes

2a443e7

Signed-off-by: Taekjin LEE <[email protected]>

feat: object collector

866b656

Signed-off-by: Taekjin LEE <[email protected]>

impl input manager, no subscribe

4b71b57

Signed-off-by: Taekjin LEE <[email protected]>

fix: subscribe and trigger callback

a9fa209

Signed-off-by: Taekjin LEE <[email protected]>

fix: subscriber and callbacks are working

cf2874d

Signed-off-by: Taekjin LEE <[email protected]>

fix: callback object is fixed, tracker is working

246cb4a

Signed-off-by: Taekjin LEE <[email protected]>

fix: get object time argument revise

9ebaf6d

Signed-off-by: Taekjin LEE <[email protected]>

feat: back to periodic publish, analyze input latency and timings

c496a25

Signed-off-by: Taekjin LEE <[email protected]>

fix: enable timing debugger

969483c

Signed-off-by: Taekjin LEE <[email protected]>

fix: separate object interval function

ceaacc3

Signed-off-by: Taekjin LEE <[email protected]>

feat: prepare message triggered process

c556460

Signed-off-by: Taekjin LEE <[email protected]>

feat: trigger tracker by main message arrive

4afab3f

Signed-off-by: Taekjin LEE <[email protected]>

chore: clean-up, set namespace

23a9df4

Signed-off-by: Taekjin LEE <[email protected]>

feat: object lists with detector index

787d48b

Signed-off-by: Taekjin LEE <[email protected]>

feat: define input channel struct

b4993c2

Signed-off-by: Taekjin LEE <[email protected]>

technolojin marked this pull request as ready for review May 14, 2024 08:29

technolojin requested review from YoshiRi, yukkysaito and miursh as code owners May 14, 2024 08:29

miursh reviewed May 15, 2024

View reviewed changes

perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp Outdated Show resolved Hide resolved

technolojin added the run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label May 16, 2024

technolojin and others added 3 commits May 16, 2024 16:54

Update perception/multi_object_tracker/include/multi_object_tracker/d…

f873ab8

…ebugger/debug_object.hpp Co-authored-by: Shunsuke Miura <[email protected]> Signed-off-by: Taekjin LEE <[email protected]>

chore: Update copyright to uppercase

1ba2c79

Signed-off-by: Taekjin LEE <[email protected]>

chore: fix readme links to config files

9edd2e0

Signed-off-by: Taekjin LEE <[email protected]>

technolojin force-pushed the feat/mot-multi-object-input branch from 827ead6 to 9edd2e0 Compare May 16, 2024 07:55

yukkysaito reviewed May 16, 2024

View reviewed changes

perception/multi_object_tracker/include/multi_object_tracker/debugger/debug_object.hpp Outdated Show resolved Hide resolved

yukkysaito reviewed May 16, 2024

View reviewed changes

perception/multi_object_tracker/src/debugger/debugger.cpp Outdated Show resolved Hide resolved

yukkysaito reviewed May 16, 2024

View reviewed changes

perception/multi_object_tracker/src/processor/input_manager.cpp Outdated Show resolved Hide resolved

technolojin added 3 commits May 17, 2024 10:21

chore: move and rename uuid functions

cf6cf3c

Signed-off-by: Taekjin LEE <[email protected]>

chore: fix debug topic to use node name

d5efe10

Signed-off-by: Taekjin LEE <[email protected]>

chore: express meaning of threshold

4445c45

Signed-off-by: Taekjin LEE <[email protected]>

technolojin force-pushed the feat/mot-multi-object-input branch from 2ffb7d8 to 4445c45 Compare May 17, 2024 01:22

technolojin requested a review from yukkysaito May 17, 2024 01:25

YoshiRi approved these changes May 23, 2024

View reviewed changes

perception/multi_object_tracker/src/processor/input_manager.cpp Outdated Show resolved Hide resolved

perception/multi_object_tracker/src/tracker/model/tracker_base.cpp Outdated Show resolved Hide resolved

technolojin and others added 3 commits May 23, 2024 11:55

feat: revise decay rate, update function

f218f84

Signed-off-by: Taekjin LEE <[email protected]>

fix: define constants with explanation

b623993

Signed-off-by: Taekjin LEE <[email protected]>

style(pre-commit): autofix

13adc73

technolojin merged commit 3aa84b0 into autowarefoundation:main May 23, 2024
22 of 24 checks passed

wep21 reviewed May 26, 2024

View reviewed changes

technolojin mentioned this pull request May 27, 2024

fix(multi_object_tracker): input stream timimg functions #7127

Merged

4 tasks

technolojin mentioned this pull request Jun 12, 2024

feat(tier4_perception_launch): enable multi channel tracker merger #7459

Merged

4 tasks

technolojin deleted the feat/mot-multi-object-input branch July 26, 2024 02:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(multi_object_tracker): multi object input #6820

feat(multi_object_tracker): multi object input #6820

technolojin commented Apr 16, 2024 •

edited

Loading

YoshiRi left a comment

wep21 May 26, 2024 •

edited

Loading

yukkysaito May 26, 2024

technolojin May 27, 2024

technolojin May 27, 2024

feat(multi_object_tracker): multi object input #6820

feat(multi_object_tracker): multi object input #6820

Conversation

technolojin commented Apr 16, 2024 • edited Loading

Description

Related links

Tests performed

Notes for reviewers

Interface changes

Effects on system behavior

Pre-review checklist for the PR author

In-review checklist for the PR reviewers

Post-review checklist for the PR author

YoshiRi left a comment

Choose a reason for hiding this comment

wep21 May 26, 2024 • edited Loading

Choose a reason for hiding this comment

yukkysaito May 26, 2024

Choose a reason for hiding this comment

technolojin May 27, 2024

Choose a reason for hiding this comment

technolojin May 27, 2024

Choose a reason for hiding this comment

technolojin commented Apr 16, 2024 •

edited

Loading

wep21 May 26, 2024 •

edited

Loading