mme-realworld #266
Merged
mme-realworld #266
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi thanks for this PR! |
kcz358
reviewed
Sep 21, 2024
kcz358
reviewed
Sep 21, 2024
kcz358
reviewed
Sep 21, 2024
|
and please fix the lint issue. |
kcz358
requested changes
Sep 21, 2024
There was a problem hiding this comment.
Hi @yfzhang114 , thank you for your contribution!
I would like to ask a few questions after reviewing the code and kindly ask if some changes can be made before merging.
- The split of dataset currently is hardcoded. This is not a recommended way to do so. Is it possible that you can split your dataset into two splits or two subsets in the huggingface and specify in the yaml?
- As I noticed that the two utils are almost the same instead of the doc_to_text, is it possible that you can merge the two utils into one file? You can specify which doc_to_text to use in the yaml config also.
- This one is not necessary but recommended to do so. I noticed that you convert your dataset from base64 to PIL.Image manually after download. Actually you can upload the dataset in the format of
{"bytes" : <base64 image>, "path" : <a dummy path>}and set the features toImage()from huggingfacedatasetso that the image will show up in your dataset viewer also.
Thank you again for your contribution and please share your thoughts on whether these changes can be made
kcz358
reviewed
Sep 21, 2024
|
kcz358
approved these changes
Sep 22, 2024
There was a problem hiding this comment.
@yfzhang114 , Thank you for your contribution! LGTM
If it is possible, can you share a screenshot of the evaluation result using the mme-realworld dataset?
And seems like the lint check have failed again. Can you run an auto fixed using pre-commit?
pip install pre-commit
pre-commit install
pre-commit run --all-files
|
@kcz358 of course, here is the results of qwen0.5B_clip_vit, the number of all subtasks has been carefully checked.
|
KairuiHu
pushed a commit
that referenced
this pull request
Sep 23, 2024
* mme-realworld * mme-realworld * mme-realworld
KairuiHu
pushed a commit
that referenced
this pull request
Oct 24, 2024
* mme-realworld * mme-realworld * mme-realworld
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MME-RealWorld is the largest manually annotated benchmark to date, featuring the highest resolution and a targeted focus on real-world applications. https://mme-realworld.github.io/