FSDP Mode

[less-and-less-bugs]

accelerate launch
--num_processes=1
--use_fsdp
--fsdp_offload_params false
--fsdp_sharding_strategy SHARD_GRAD_OP
--fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP
--fsdp_transformer_layer_cls_to_wrap Phi3DecoderLayer
--fsdp_state_dict_type FULL_STATE_DICT
--fsdp_forward_prefetch false
--fsdp_use_orig_params True
--fsdp_cpu_ram_efficient_loading false
--fsdp_sync_module_states True
train.py
--model_name_or_path Shitao/OmniGen-v1
--json_file ./toy_data/toy_data.jsonl
--image_path ./toy_data/images
--batch_size_per_device 1
--lr 2e-5
--keep_raw_resolution
--max_image_size 1024
--gradient_accumulation_steps 1
--ckpt_every 50
--epochs 200
--log_every 1
--results_dir ./results/toy_finetune

When running this command, I encountered an AttributeError: 'OmniGen' object has no attribute '_tied_weights_keys.' It seems you have not implemented fsdp mode for Omonigen. Could you double-check this problem, please?

[staoxiao]

You can solve it by re-installing accelerate of a lower version: pip install accelerate==0.26.1, as shown in setup.py https://github.com/VectorSpaceLab/OmniGen/blob/main/setup.py#L20