Failed to replay the trajectory when setting the sim_backend GPU?

[JuilieZ]

I met an issue that the trajectory wont succeed if I set the sim_backend 'GPU' when I replay the traj.
More seriously ,I completey got the wrong trajectory when I inputed(actually even I didn't add the '-b gpu',it didn't make sense ,either):
python -m mani_skill.trajectory.replay_trajectory --traj-path /home/server/.maniskill/demos/PushT-v1/rl/trajectory.none.pd_joint_delta_pos.cuda.h5 --render-mode="rgb_array" --save-video --count 10 -b gpu

panda_pusht.mp4

I got the similar issue when I was in other tasks(pickup,pushcube,an so on)
Could u tell me the reason caused the problem?Thank u in advance~

[StoneT2000]

I think our trajectory replay converter needs a massive upgrade. Currently there are some bugs with replaying in parallel on the GPU at the moment that needs to be fixed, the issue is tracked here #564

[StoneT2000]

We just upgraded the replay conversion tool to better support GPU sim. Not on pypi yet but on the main branch

Also for PushT and other highly precise tasks, you sometimes may want to add the --use-env-states option to the replay tool. Due to how precise the task is, sometimes small floating point imprecisions can actually lead to different behaviors when taking the same actions, especially with GPU simulation.

[JuilieZ]

We just upgraded the replay conversion tool to better support GPU sim. Not on pypi yet but on the main branch

Also for PushT and other highly precise tasks, you sometimes may want to add the --use-env-states option to the replay tool. Due to how precise the task is, sometimes small floating point imprecisions can actually lead to different behaviors when taking the same actions, especially with GPU simulation.

Thx for your reply, but actually the failure just happened when the trajectory is from RL evaluating.Now I'm confused why the different qpos results occurred when the same action steped(RL evaluation trajectory>>>just Replay it with no control mode changing)?Hope you help this problem, thx in advance~

[StoneT2000]

hi @JuilieZ can you clarify your question?

Different qpos can happen despite the same starting env state and action sequence if theres a

1. different sim backend
2. different number of parallel GPU envs

the errors accumulate due to floating precision differences and probably some things in the physx engine that I don't know about. For highly precise tasks where there is a lot of contact like pushT this error is more problematic.

python -m mani_skill.trajectory.replay_trajectory \
  --traj-path ~/.maniskill/demos/PushT-v1/rl/trajectory.none.pd_ee_delta_pose.physx_cuda.h5 \
  --use-env-states -c pd_ee_delta_pose -o state \
  --save-traj --num-envs 1024 -b physx_cuda

a script like this should work though

[JuilieZ]

hi @JuilieZ can you clarify your question?

Different qpos can happen despite the same starting env state and action sequence if theres a

1. different sim backend

2. different number of parallel GPU envs

the errors accumulate due to floating precision differences and probably some things in the physx engine that I don't know about. For highly precise tasks where there is a lot of contact like pushT this error is more problematic.

python -m mani_skill.trajectory.replay_trajectory \
  --traj-path ~/.maniskill/demos/PushT-v1/rl/trajectory.none.pd_ee_delta_pose.physx_cuda.h5 \
  --use-env-states -c pd_ee_delta_pose -o state \
  --save-traj --num-envs 1024 -b physx_cuda

a script like this should work though

thx for ur reply !!! actually recently I figure it out . the true reason is that the difference IK result between the two simbackend(CPU&GPU).by the way,I have another request:could u provide more tasks and demonstrations for franka from Maniskill2 and 3.such as assemblingkits,pick clutterYCB so on.