New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve transparency of the evaluation framework #2

Open

payoto wants to merge 5 commits into openai-api-compatability from extract-all-data-from-api-run

Collaborator

payoto commented Feb 20, 2024 •

edited

Loading

Saves all prompt transformation steps to the JSON logs
Supports passing arbitrary prompt formats
Improves docker build speed by pre-caching requirements
Output results for individual samples

...

This PR is not to be merged

payoto added 5 commits

February 20, 2024 10:09


          Remove whitespace


          Save all the stages of data transformation when evaluating the API

2b441df


          Support arbitrary prompt formats for HumanEvalPack

0d30df6


          Cache requirements for the docker build

ab794af


          Output individual sample results in the result output

d349ff6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet