Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add profiling & benchmarks for Indexset and Parameter -- clean #155

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

glatterf42
Copy link
Member

This PR adds profiling tests for Parameters and Indexsets. It also increases performance by avoiding the copy.deepcopy() calls when validating optimization data and adjusts the validation function accordingly.

Unfortunately, #150 has fallen out of sync with main. When trying to do a git rebase there, git wants to rebase all commits, which in this case means all 118 commits in the whole repository, creating lots of unnecessary conflicts.
So instead of dealing with that, I looked at the largest files in our git history (thanks to stackoverflow) in a fresh clone and found that it doesn't contain the big parameterdata file that was committed by accident.

List of ten largest files in our git history right now
d305e273e990  425KiB tests/fixtures/optimization/medium/parameterdata.csv
b781df98acee  716KiB tests/fixtures/optimization/medium/parameterdata.csv
a07bfd345778  996KiB tests/fixtures/medium/datapoints.csv
92fe972a2a27  2,2MiB tests/fixtures/big/parameterdata.csv
6716fc873ab2  6,5MiB tests/fixtures/big/datapoints.csv
38d78752a53b  6,6MiB tests/fixtures/optimization/big/indexsetdata.csv
6f4595a13944   15MiB tests/fixtures/optimization/big/indexsets.csv
e2a9ff495a0b   16MiB tests/fixtures/optimization/big/parameters.csv
d8a3bb70b4d4   32MiB tests/fixtures/big/upsert_parameterdata_vk.csv
762b69cd8c81   32MiB tests/fixtures/big/insert_parameterdata_vk.csv

Which means: the branch was deleted and never merged, thus the file was never present and we don't need to invoke git filter-repo to clean up. All we need to do is open a new PR based on main and cherry-pick the commits we're interested in, which is what this PR does.

Note

The two largest files come from the branch #112, which will likely not be merged anytime soon. It also contains the script with which these files were created, so we can already cut repo size quite significantly by going to that branch and removing these files (first try a new commit doing that, then checking files sizes in a fresh clone again, if that didn't work, re-work the commit that introduced the files | alternatively, delete this branch, but I'd like to keep it for now in case we want to revisit that topic).

Copy link

codecov bot commented Jan 30, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.0%. Comparing base (c7ea1f3) to head (170f7fc).

Additional details and impacted files
@@           Coverage Diff           @@
##            main    #155     +/-   ##
=======================================
- Coverage   87.0%   87.0%   -0.1%     
=======================================
  Files        230     230             
  Lines       8170    8156     -14     
=======================================
- Hits        7112    7098     -14     
  Misses      1058    1058             
Files with missing lines Coverage Δ
ixmp4/data/db/optimization/equation/model.py 100.0% <100.0%> (ø)
ixmp4/data/db/optimization/parameter/model.py 100.0% <100.0%> (ø)
ixmp4/data/db/optimization/table/model.py 100.0% <100.0%> (ø)
ixmp4/data/db/optimization/utils.py 100.0% <100.0%> (ø)
ixmp4/data/db/optimization/variable/model.py 100.0% <100.0%> (ø)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant