Receiving double JSON in one UDP message

[muella91]

Hello,

Right now I am working with single camera pose tracking with a Kinect v2.
I have three issues:

Issue 1:

I realized that I sometimes receive two JSONs in a row in one UDP message (what throws an JSON parse error).
The two JSON frames have the same sequence number, but different joint data.
The error mainly occurs when I enter the field of view of the camera (or leave it and enter it again).
const char* jsonString = (char*)udpBuffer gets me the following string:

{"header":{"seq":892,"stamp":{"sec":1580147986,"nsec":879454837},"frame_id":"world"},"pose_tracks":[{"id":10,"height":0.0473687,"orientation":-1.66644,"age":71.1696,"predicted_pose_name":"unknown","predicted_pose_id":-1,"prediction_score":-1,"poses":[{"pose_name":"arms_mid","pose_id":0,"prediction_score":2.75134},{"pose_name":"right_arm_up","pose_id":1,"prediction_score":3.80482},{"pose_name":"arms_up","pose_id":2,"prediction_score":4.03893},{"pose_name":"left_arm_pointing","pose_id":3,"prediction_score":3.77611}],"joints":{"HEAD":{"x":2.82864,"y":0.424074,"z":0.701934,"confidence":1},"NECK":{"x":2.44813,"y":1.44401,"z":0.537648,"confidence":1},"RIGHT_SHOULDER":{"x":2.65279,"y":1.32239,"z":0.539224,"confidence":1},"RIGHT_ELBOW":{"x":2.78928,"y":1.3434,"z":0.258426,"confidence":1},"RIGHT_WRIST":{"x":2.82219,"y":1.49604,"z":0.0990536,"confidence":1},"LEFT_SHOULDER":{"x":2.2932,"y":1.29466,"z":0.517954,"confidence":1},"LEFT_ELBOW":{"x":2.32901,"y":1.42079,"z":0.24244,"confidence":1},"LEFT_WRIST":{"x":2.67842,"y":0.345023,"z":0.114418,"confidence":1},"RIGHT_HIP":{"x":2.57418,"y":1.41502,"z":0.0505699,"confidence":1},"RIGHT_KNEE":{"x":2.72625,"y":1.5835,"z":-0.328984,"confidence":1},"RIGHT_ANKLE":{"x":2.78874,"y":1.53666,"z":-0.750232,"confidence":1},"LEFT_HIP":{"x":2.423,"y":1.57372,"z":0.0480985,"confidence":1},"LEFT_KNEE":{"x":2.4547,"y":1.1704,"z":-0.34796,"confidence":1},"LEFT_ANKLE":{"x":2.47469,"y":1.11521,"z":-0.750577,"confidence":1},"CHEST":{"x":2.47295,"y":1.47746,"z":0.214173,"confidence":1}}}]}{"header":{"seq":892,"stamp":{"sec":1580147986,"nsec":879454837},"frame_id":"world"},"pose_tracks":[{"id":11,"height":0.0785036,"orientation":"NaN","age":0.248722,"predicted_pose_name":"unknown","predicted_pose_id":-1,"prediction_score":-1,"poses":[{"pose_name":"arms_mid","pose_id":0,"prediction_score":"nan"},{"pose_name":"","pose_id":0,"prediction_score":0},{"pose_name":"","pose_id":0,"prediction_score":0},{"pose_name":"","pose_id":0,"prediction_score":0}],"joints":{"HEAD":{"x":6.93501e-310,"y":2.34837e-316,"z":2.17977e+243,"confidence":1},"NECK":{"x":6.93501e-310,"y":2.44433e-316,"z":0.554901,"confidence":1},"RIGHT_SHOULDER":{"x":0,"y":0,"z":0,"confidence":1},"RIGHT_ELBOW":{"x":0,"y":0,"z":0,"confidence":1},"RIGHT_WRIST":{"x":0,"y":0,"z":0,"confidence":1},"LEFT_SHOULDER":{"x":0,"y":0,"z":0,"confidence":1},"LEFT_ELBOW":{"x":0,"y":0,"z":0,"confidence":1},"LEFT_WRIST":{"x":0,"y":0,"z":0,"confidence":1},"RIGHT_HIP":{"x":0,"y":0,"z":0,"confidence":1},"RIGHT_KNEE":{"x":0,"y":0,"z":0,"confidence":1},"RIGHT_ANKLE":{"x":0,"y":0,"z":0,"confidence":1},"LEFT_HIP":{"x":0,"y":0,"z":0,"confidence":1},"LEFT_KNEE":{"x":0,"y":0,"z":0,"confidence":1},"LEFT_ANKLE":{"x":0,"y":0,"z":0,"confidence":1},"CHEST":{"x":0,"y":0,"z":0,"confidence":1}}}]}

The second JSON in this message if filled with zeros, but there are also messages filled with different values. Do you know this error?
My idea is to cut the string, I am just not sure which JSON I should throw away.

Issue 2:

Sometimes I get JSON messages where the joints are filled with zeros (like the second JSON above).
Anyway, why does the confidence of that joint still says "confidence":1, when the position is obviously not known?
I realized those messages have a lot of NaN in the attributes like "orientation" or "prediction_score".
Can I assume that messages with such NaNs are not valid?

Issue 3:

I receive sometimes frames with the "frame_id":"heartbeat".
I expect a heartbeat to repeat at a certain frequency. But this one seems only to occur like an echo, if no poses are anymore detected.

Thanks!

[bsubbaraman]

1. I haven't seen this before, thanks for bringing it to our attention. Are there two skeletons being detected when this occurs? If so, I'm not sure which skeleton is 'correct'/should be thrown away.
2. You can assume these aren't valid (see NaN, very large and very small numbers for joint positions  #12 for more info on NaNs and zeros). The confidence values are generated by rt_pose, so unfortunately I don't have a good answer for why the confidence outputs 1 in these cases.
3. I'm not sure, but my guess is that the hearbeat is published when there are tracks, and there is a slight delay between when the person leaves the camera view and the last detection 'disappears'. See ros2udp_converter.cpp for more info

[muella91]

Thanks for your answer!

1. So, it seems there are two skeletons detected (because there are two IDs). But for some reason, OPT does not put them together in the "pose_tracks" JSON array, but as two JSONs (with two headers) in one UDP message. Anyway, there was only one person in the room/camera view.
   I realized that the joints of one skeleton are always filled with zeros (so this is the invalid one). So far, this was always the second JSON.
   

Maybe the error disappears with Multi-camera-pose tracking.