Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Support for Model artifacts #265

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

feat: Support for Model artifacts #265

wants to merge 8 commits into from

Conversation

mercuryseries
Copy link
Contributor

@mercuryseries mercuryseries commented Feb 3, 2025 •
edited
Loading

Changelogs

  • Add a new Model class to represent a model artifact.
  • Add a new load_model utility function to load a model.

Checklist:

  • Was this PR discussed in an issue? It is recommended to first discuss a new feature into a GitHub issue before opening a PR.
  • Add tests to cover the fixed bug(s) or the newly introduced feature(s) (if appropriate).
  • Update the API documentation if a new function is added, or an existing one is deleted.
  • Write concise and explanatory changelogs above.
  • If possible, assign one of the following labels to the PR: feature, fix, chore, documentation or test (or ask a maintainer to do it for you).

TODO

  • Add some tests

@mercuryseries mercuryseries added the feature Annotates any PR that adds new features; Used in the release process label Feb 3, 2025
@mercuryseries mercuryseries self-assigned this Feb 3, 2025
cwognum
cwognum approved these changes Feb 3, 2025
View reviewed changes
Copy link
Collaborator

@cwognum cwognum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mercuryseries ! 🙏

polaris/model/__init__.py
from polaris.utils.types import AccessType, HubOwner


class Model(BaseArtifactModel):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Could we add a docstring here with some details about this class? And could we add this class to our mkdocs documentation site?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea.

polaris/evaluate/_results.py Show resolved Hide resolved
polaris/evaluate/_results.py Show resolved Hide resolved
polaris/evaluate/_results.py Outdated Show resolved Hide resolved
polaris/hub/client.py Show resolved Hide resolved
@cwognum
Copy link
Collaborator

cwognum commented Feb 3, 2025

@mercuryseries For the failing test case, looks like you were just unlucky! We should set the random seed for that test case so that doesn't happen again.

jstlaurent
jstlaurent reviewed Feb 3, 2025
View reviewed changes
Copy link
Contributor

@jstlaurent jstlaurent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Short and sweet. Very nice. 😄

I do agree with Cas' comment, about making this only available on Results V2, for Benchmark V2s.

polaris/hub/client.py Outdated Show resolved Hide resolved
polaris/model/__init__.py
from polaris.utils.types import AccessType, HubOwner


class Model(BaseArtifactModel):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea.

polaris/hub/client.py Show resolved Hide resolved
polaris/evaluate/_results.py Show resolved Hide resolved
@mercuryseries mercuryseries force-pushed the model-artifact branch 4 times, most recently from dded7f3 to ec89949 Compare February 7, 2025 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jstlaurent jstlaurent jstlaurent left review comments

@cwognum cwognum cwognum approved these changes

@Andrewq11 Andrewq11 Awaiting requested review from Andrewq11

@achchala achchala Awaiting requested review from achchala

@roselynh100 roselynh100 Awaiting requested review from roselynh100

Assignees

@mercuryseries mercuryseries

Labels
feature Annotates any PR that adds new features; Used in the release process
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mercuryseries @cwognum @jstlaurent