New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REQUEST] option to shard weights only in each node #7019

Open

cyr0930 opened this issue Feb 8, 2025 · 1 comment

Labels

cyr0930 commented Feb 8, 2025

Is your feature request related to a problem? Please describe.
Multi-node training with stage3 is too slow.

Describe the solution you'd like
Avoid weight sync between inter-GPUs and shard weights only between intra-GPUs

cyr0930 added the enhancement label

Collaborator

loadams commented Feb 10, 2025

@cyr0930 - could you explain more about your problem/what is slow? Can you share any more details about your system/model?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment