Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support for handling Benchmarks V2 #234

Merged
merged 9 commits into from
Dec 17, 2024
Merged

feat: Support for handling Benchmarks V2 #234

merged 9 commits into from
Dec 17, 2024

Conversation

jstlaurent
Copy link
Contributor

Changelogs

  • Make BenchmarkSpecification into an abstract class, with BenchmarkV1Specification and BenchmarkV2Specification implementations. The abstract class defines a series of abstract methods and properties
    for the subclasses to implement.
  • Implemented a new type of split for Benchmark V2, using the Roaring Bitmap format to efficiently store extremely large sets of indicices.
  • Improved the type definition in the Pydantic models, to reflect the post-validation types. Where we are flexible on the input type, validators with mode="before" ensure that we convert the input to a
    single internal representation.
  • Added upload/dowload methods to the Hub client, to support interacting with the Hub storage.

Checklist:

  • Was this PR discussed in an issue? It is recommended to first discuss a new feature into a GitHub issue before opening a PR.
  • Add tests to cover the fixed bug(s) or the newly introduced feature(s) (if appropriate).
  • Update the API documentation if a new function is added, or an existing one is deleted.
  • Write concise and explanatory changelogs above.
  • If possible, assign one of the following labels to the PR: feature, fix, chore, documentation or test (or ask a maintainer to do it for you).

The core changes are ready for review. I need to add some tests and ensure the doc is updated as well.

@jstlaurent jstlaurent requested a review from cwognum as a code owner December 16, 2024 14:59
@jstlaurent jstlaurent added the feature Annotates any PR that adds new features; Used in the release process label Dec 16, 2024
@jstlaurent jstlaurent self-assigned this Dec 16, 2024
Copy link
Collaborator

@cwognum cwognum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jstlaurent and @mercuryseries ! This is looking good! 🚀

Copy link
Contributor

@mercuryseries mercuryseries left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, @jstlaurent! I added some minor comments here and there.

try:
    while True:
        print("Thank you! (Until Julien is tired)")
except KeyboardInterrupt:
    print("Gratitude interrupted, but it’s endless in spirit! :)")

@jstlaurent jstlaurent merged commit abb9b29 into main Dec 17, 2024
10 checks passed
@jstlaurent jstlaurent deleted the xl-benchmarks branch December 17, 2024 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Annotates any PR that adds new features; Used in the release process
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants