Bencher Historical Baseline #666
Conversation
|
cscs-ci run baseline |
|
cscs-ci run baseline |
|
@edopao is there any better way to get the name of |
kotsaloscv
changed the title
|
cscs-ci run benchmark-bencher-baseline |
I don't know how to get "gh200", but you could get the vCluster name from the variable |
ci/benchmark_bencher_baseline.yml
Outdated
| --err \ | ||
| --adapter python_pytest \ | ||
| --file results.json \ | ||
| "pytest --benchmark-json results.json -sv --benchmark-only --backend=$BACKEND --grid=$GRID ./model" |
There was a problem hiding this comment.
Is -s to stream the stdout? If so I suggest to remove it, since this is a benchmark.
There was a problem hiding this comment.
The above command is the same as the one called by nox -> https://github.com/C2SM/icon4py/blob/main/noxfile.py#L49
Although I am wondering if both should be removed.
There was a problem hiding this comment.
I think -v is fine to keep, it's about verbosity of pytest command itself. -s instead is maybe better to skip, in benchmark mode.
|
Mandatory Tests Please make sure you run these tests via comment before you merge!
Optional Tests To run benchmarks you can use:
To run tests and benchmarks with the DaCe backend you can use:
In case your change might affect downstream icon-exclaim, please consider running
For more detailed information please look at CI in the EXCLAIM universe. |
This pipeline runs every day at 21 (for the main branch), and pushes the benchmark results to Bencher cloud. In this way, we are creating a historical baseline such that we can compare every feature branch with this baseline.
All the parameters for bencher cloud are set in https://cicd-ext-mw.cscs.ch (currently I am pushing everything in a test project, while an icon4py-bencher project already exists and will be used once we merge this PR).
This link shows some graphs for this test bencher project.
Once we move to the icon4py, I will pin more meaningful graphs.