How to connect the points predicted?

[wallaceloos]

Hello,

I am trying to understand how the points from the pose estimation are connected.
There is a function called vis_pose_result (https://github.com/open-mmlab/mmpose/blob/0.x/mmpose/apis/inference.py) and then inside of this function the object model calls show_result to connect the points extracted previewly. How does it know how to connect them? Anyone know anything about it?

Thank you.

[gelbling]

Hi @wallaceloos, I am not an official contributor but I've done exactly what you're asking before. Hopefully I can help!

The mmpose visualizer handles the connections between keypoints during inference (running the model with visuals). Not sure if you are using a custom dataset or not, but there is a "dataset_info" dictionary that specifies the connections between keypoints. The visualizer uses these mappings to plot the predicted result.

For example, for the COCO dataset (found here you can see the exact keypoints they use and their ids as well as the skeleton connections between them.

It is also worth noting that the skeleton keypoint connections are also specified in your annotations for training/val data.
Here is an example of the COCO dataset annotations skeleton and keypoints.

"categories": [ { "supercategory": "person", "id": 1, "name": "person", "keypoints": [ "nose", "left_eye", "right_eye", "left_ear", "right_ear", "left_shoulder", "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist", "left_hip", "right_hip", "left_knee", "right_knee", "left_ankle", "right_ankle" ], "skeleton": [ [ 16, 14 ], [ 14, 12 ], [ 17, 15 ], [ 15, 13 ], [ 12, 13 ], [ 6, 12 ], [ 7, 13 ], [ 6, 7 ], [ 6, 8 ], [ 7, 9 ], [ 8, 10 ], [ 9, 11 ], [ 2, 3 ], [ 1, 2 ], [ 1, 3 ], [ 2, 4 ], [ 3, 5 ], [ 4, 6 ], [ 5, 7 ] ] }

[wallaceloos]

Thank you very much, @gelbling.

However, I am wondering how it handles when not all points are detected. Because, when we have all points we can follow the skeleton dictionary, but how do we proceed when there are missing points? Right now, I am assuming if we have missing points, some of them will be connected in a wrong way.

[gelbling]

I'm not sure if I understand your question, my bad. Could you provide some more detail/context?

Are you asking about a situation in which some keypoints are not in frame? (For example a persons head isn't visible, or labeled in the annotations)

[LareinaM]

Hi, thanks for using MMPose. Usually, the model would predict all keypoints (e.g. 17 for Coco Dataset), but at the same time provide confidence scores. During visualization, links with a low confidence or invalid (out of the image) would be ignored.

[wallaceloos]

Okay, if the model always returns all key points, then no problem.
Thank you, @gelbling and @LareinaM