Questions about data preprocessing

[zhujiagang]

Thanks for sharing your excellent work.
I've encountered difficulties in preprocessing data using your script.

python preprocess.py \
    --data_root data/waymo/raw/ \
    --target_dir data/waymo/processed \
    --split training \
    --process_keys images lidar calib pose dynamic_masks segmentation \
    --workers 2 \
    --scene_ids 114 700

First of all, it should be preprocess_main.py.
Secondly, I cannot get full result using --workers 2. The code seems to stuck, Using --workers 1, I can get the final result, though it is slow. Have you encountered same problem before?

[LuPaoPao]

Thanks for sharing your excellent work. I've encountered difficulties in preprocessing data using your script.

python preprocess.py \
    --data_root data/waymo/raw/ \
    --target_dir data/waymo/processed \
    --split training \
    --process_keys images lidar calib pose dynamic_masks segmentation \
    --workers 2 \
    --scene_ids 114 700

First of all, it should be preprocess_main.py. Secondly, I cannot get full result using --workers 2. The code seems to stuck, Using --workers 1, I can get the final result, though it is slow. Have you encountered same problem before?

preprocess_main.py is right! Thank you for your kind reminder!

Yes, it is very slow. Multiple workers are fine, but no more than 8 are recommended. At the same time, you can open several scripts, which will increase the speed! You can handle 0-200 with one code, 200-400 with the second code... Of course, the main results in our paper only need to be trained in 64 scenes to get the such performance!