-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[tuner] Run baseline benchmarks first with reusable helper function and regression detection #789
base: main
Are you sure you want to change the base?
Conversation
bd88a9c
to
46d6a87
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall looks good. Just a few nits about the logging and code reuse. Could you also add a more informative PR description (essentially explain the reasoning for benchmarking baselines twice like in the linked issue).
46d6a87
to
222f783
Compare
222f783
to
4b2165c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, thanks! Just one nit.
Ideally, these corner cases should be covered by unit tests. |
WIP: need to rebase on top of #799, and continue to work. |
9593df8
to
f139ee1
Compare
30ab7e4
to
6c88791
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you edit the PR description and explain what error conditions you considered and how they are handled?
f4be426
to
6966557
Compare
Signed-off-by: Bangtian Liu <[email protected]>
Signed-off-by: Bangtian Liu <[email protected]>
Signed-off-by: Bangtian Liu <[email protected]>
Signed-off-by: Bangtian Liu <[email protected]>
6966557
to
4c52419
Compare
This PR is relevant to address the task in: #783.
This PR adds a reusable helper function that takes a list of candidate indices and returns benchmarking results, allowing the same function to be used for baseline and candidate benchmark results.
Furthermore, this PR adds detection and logging for several conditions:
Also, unit tests are added to show the corner cases.