Add model jxm/cde-small-v1 #1521
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| from __future__ import annotations | ||
|
|
||
| import mteb | ||
|
|
||
| model = mteb.get_model( | ||
|
There was a problem hiding this comment. I'm not sure if I understand this correctly, but it seems like you did not add a model implementation or model metadata for CDEs. I'm also unsure whether this would work or not. I believe their official guide on how to use CDE is a bit more complicated than this, since they have a first and a sceond stage in all of their guides where they first produce a corpus embedding and then pass it along to the model when embedding new documents. There was a problem hiding this comment. They have evaluation script, but it a bit complecated https://github.com/jxmorris12/cde/blob/0de4e6c116c8e8223075a2b56277d69e04a2ab7c/evaluate_mteb.py#L26 There was a problem hiding this comment. I see, but I guess it's a better choice still not to implement the model incorrectly here, and maybe just add metadata on it, then ask the CDE team to upload their results to the results repository. There was a problem hiding this comment. I agree with you. I added it evaluation script just for information and show author's implementation There was a problem hiding this comment. I didn't explicitly define the model metadata because when I ran the mteb.get_model_meta command, the output seemed correct. However, I may have misunderstood and overlooked the need to explicitly define the model metadata. I also have the results repository from when I ran the script. Should I disregard that? I'm a bit unsure about the next steps I should take. I would appreciate your guidance—thank you! There was a problem hiding this comment. @jxmorris12 Awesome, I look forward to working with you in the new year! There was a problem hiding this comment. Hi @jxmorris12, Happy New Year! I hope you’re doing well. I wanted to follow up and see if you’ve had a chance to take a deeper look at my implementation of your model. I’d greatly appreciate any feedback or suggestions for improvement to ensure we can properly integrate CDE into MTEB. Looking forward to hearing your thoughts—thank you in advance! There was a problem hiding this comment. Hi. I'm currently in the process of uploading cde-small-v2, which should happen this week. Once that is finished we can update this PR since it should be easier to use. Should be available within a few days. There was a problem hiding this comment. Hi @YashDThapliyal – can you (1) update your code to use cde-small-v2 (https://huggingface.co/jxm/cde-small-v2) and (2) update the code to actually grab contextual documents from each corpus? I I've actually done the work for you of figuring out how to get documents from each dataset type, so you should be essentially copy the approach in the CDE repo: https://github.com/jxmorris12/cde/blob/main/evaluate_mteb.py Let us know once you've done that and we can all look over the results. There was a problem hiding this comment. @jxmorris12 sure, I will get on that |
||
| "jxm/cde-small-v1", | ||
| trust_remote_code=True, | ||
| model_prompts={"query": "search_query: ", "passage": "search_document: "}, | ||
| ) | ||
| tasks = mteb.get_tasks( | ||
| tasks=[ | ||
| # classification | ||
| "AmazonCounterfactualClassification", | ||
| # clustering | ||
| "RedditClustering", | ||
| # pair classification | ||
| "TwitterSemEval2015", | ||
| # reranking | ||
| "AskUbuntuDupQuestions", | ||
| # retrieval | ||
| "SCIDOCS", | ||
| # # sts | ||
| "STS22", | ||
| # # summarization | ||
| "SummEval", | ||
| ] | ||
| ) | ||
| evaluation = mteb.MTEB(tasks=tasks) | ||
| results = evaluation.run( | ||
| model, | ||
| output_folder="results", | ||
| extra_kwargs={"batch_size": 8}, | ||
| overwrite_results=True, | ||
| ) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since your PR is not concerned with the leaderboard, you probably shouldn't put changes in it related to that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I believe that was a result of running make lint, however I can leave that out.