New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Great contribution ! #2

Open

johnr14 opened this issue Jan 24, 2025 · 1 comment

johnr14 commented Jan 24, 2025

Hi, I made some changes #1 to get it to work on larger models.

My 2x3090 did not help so I had to opt to do per-layer-loading.

Thanks for your work.

I also gave a few ideas for MoE there :

Parameter-Efficient-MoE
Mergekit

Thanks again and keep it up !

Great to see that you publish papers AND code !

Owner

peijunallin commented Jan 28, 2025

Thank you so much for your feedback and contributions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment