New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

questions about NIAH #35

Open

Cooperx521 opened this issue Dec 4, 2024 · 1 comment

Cooperx521 commented Dec 4, 2024 •

edited

Loading

Congrats for the insightful paper!

I noticed a few points in the figure in the appendix that I find a bit confusing, and here are two questions:

Since the 'Training Long Language Model' step uses a context length of only 224k, why does the model still show high accuracy even when the context length reaches 512k?
I observed that when the distractor is set to 5, the distribution of the NIAH results appears unusual. It seems that the context length of 224k performs better than the context length of 64k, which is quite different from what is typically seen in NIAH results for other models.

Looking forward to your insights on these points
Best regards

Cooperx521 changed the title ~~questions about~~ questions about NIAH

Collaborator

jzhang38 commented Dec 5, 2024

The first phenomenon is generally observed across long context training for across various models.
I do not have a good answer for the second question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment