Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimized set_initialized_submodules. #35493

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

LagPixelLOL
Copy link

Fixed an insanely slow function set_initialized_submodules, when loading DeepSeek V3, since it has so many experts, the amount of state_dict_keys are huge and also there are a lot of named_modules.

loaded_keys = {k.replace(f"{module_name}.", "") for k in state_dict_keys if k.startswith(f"{module_name}.")}

This line, especially k.startswith(f"{module_name}."), gets called at least 1 million times, which is insanely slow, it takes at least 10 minutes runtime.

My optimization reduces the runtime to less than 10 seconds by not iterating through state_dict_keys and deleting module_name from the start for every named_module, instead, it prepends the module_name before the module.state_dict() keys.

cc: @ArthurZucker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant