New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request]: MiniCPM3中是否还使用了“粗质量Stable+高质量Decay”？ #248

Open

huyiwen opened this issue Sep 28, 2024 · 0 comments

Labels

feature

huyiwen commented Sep 28, 2024

Feature request / 功能建议

MiniCPM技术报告中提到“在预训练阶段只使用通用、量大的预训练粗质量数据，而在退火阶段，使用非常广泛的高质量知识和能力数据以及SFT的高质量数据，混合入预训练数据进行退火。”

请问在MiniCPM3中是否采用了相同的训练方法？是否有尝试过在Stable阶段加入高质量数据（或者换成Cosine降低学习率）？

huyiwen added the feature label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment