Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[sharktank] Add perplexity CI to sharktank dashboard #466

Merged
merged 34 commits into from
Nov 22, 2024

Conversation

archana-ramalingam
Copy link
Collaborator

@archana-ramalingam archana-ramalingam commented Nov 9, 2024

Add perplexity CI to sharktank dashboard
Rename _vmfb to _iree
Add Perplexity scoreboard & description
Add descriptive errors for better logging

@archana-ramalingam
Copy link
Collaborator Author

archana-ramalingam commented Nov 9, 2024

@saienduri Would like to add Perplexity CI results to this dashboard too. pytest-html doesn't auto-merge existing html files. Do we need a utility function to do this separately?

.github/workflows/ci_eval.yaml Outdated Show resolved Hide resolved
.github/workflows/ci_eval.yaml Outdated Show resolved Hide resolved
.github/workflows/ci_eval.yaml Outdated Show resolved Hide resolved
@saienduri
Copy link
Contributor

I don't think we need to merge the html files themselves. Each one will live on it's own subsite with an overarching index.html that points to each of them was what I was thinking. I am planning to explore this when I get some time today

@archana-ramalingam
Copy link
Collaborator Author

I don't think we need to merge the html files themselves. Each one will live on it's own subsite with an overarching index.html that points to each of them was what I was thinking. I am planning to explore this when I get some time today

Makes sense. I tried to push the perplexity.html to a new folder perplexity. But it seems to have replaced the existing index.html at root, the last time I ran it.

Each folder in gh-pages branch rep a subsite IIUC. We need IREE perplexity and Torch perplexity as the subsites/folders.

sharktank/sharktank/evaluate/README.md Outdated Show resolved Hide resolved
@sogartar sogartar mentioned this pull request Nov 18, 2024
@archana-ramalingam archana-ramalingam merged commit 779adc3 into main Nov 22, 2024
6 checks passed
@archana-ramalingam archana-ramalingam deleted the update-perplexity-ci branch November 22, 2024 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants