How to use scene_lib for interaction tasks training?

[mulplue]

Hi! I'd like to use the scene lib to train some interaction tasks, but I couldn't find any example scripts for using scene lib in the readme. I tried using:

python phys_anim/train_agent.py headless=False \
+exp=amp experiment_name=amp_smpl_walk_gym_test \
+robot=smpl motion_file=data/motions/smpl_humanoid_walk.npy \
+scenes=samp scene_file='data/yaml_files/samp_scenes_test.yaml' \
+backbone=isaacgym

but encountered a list overflow error.

I'm not sure if there's an issue with this script. Could you provide an example script for proper usage? Thanks very much! (Since the scene/object/robot spawn logic and offsets need to be consistent, I find it hard to add some objects without going through scene_lib)

[tesslerc]

Running the full_body_tracker experiment doesn't work?
That should spawn the SAMP scenes and provide an example for how the scene files are constructed.

Basically, a scene file is a list of scenes with unique ids.
Each scene is a list of objects.
Each object has a set of parameters, for example:

  - is_static: true  # is this fixed in place, or can the humanoid move it
    motion: data/assets/scenes/train/SAMP/Tables/836157300c030c9a246b9f2ca347e8e3.npy  # a file determining the object motion (examples are only for single pose = static objects)
    path: data/assets/scenes/train/SAMP/Tables/836157300c030c9a246b9f2ca347e8e3.urdf  # path to object
    object_options:  # parameters for convex decomposition
      vhacd_enabled: True
      vhacd_params:
        resolution: 50000
      fix_base_link: True  # identical to is_static. this is duplicated, should fix
      participates_in_heightmap: True  # whether the lidar projection (heightmap) observes this object. it's useful for large objects like sofa/chair. probably not very useful for small dynamic objects

[mulplue]

Hi @tesslerc , I tried to run the full_body_tracker using the following scripts:

python phys_anim/train_agent.py \
+exp=full_body_tracker +robot=smpl +backbone=isaacgym \
motion_file=data/motions/smpl_humanoid_walk.npy \
+scenes=samp scene_file='data/yaml_files/samp_scenes_test.yaml' \
num_envs=256 algo.config.batch_size=8192

It works like this:

the actors are not in scenes, I checked it and found that motion_ids and scene_ids are all 0/-1

I want to implement this: have actors interact with objects in specific scenes. For example, actor1 interacts with a sofa in scene1, actor2 interacts with a chair in scene2(may not need specific motion-scene pairs using amp). Is there a good way to do this? I checked the code but found the motion_lib and scene_lib to be pretty complex and have many interconnections. Thanks!

[tesslerc]

Hey, sorry for the late response.
Look at https://github.com/NVlabs/ProtoMotions/blob/main/data/yaml_files/samp_scenes_single.yaml for an example.
The motion file needs to define which scene it goes along with.

We will try and make this easier to work with in the future version such that you can define the expected behavior through code.

I agree that SceneLib is a bit complex at the moment, and we're actively trying to figure out what would be the best way to handle it.
One the one hand -- it should support complex scenarios with multiple objects, and objects that move around etc..
On the other hand -- it should be very easy to build new scenarios and code stuff via RL objectives.