failing enzo answer tests

[chrishavlin]

Both #4964 and #4965 are showing the same failing answer tests for enzo in the full test suite runs, unrelated to changes in the PRs. e.g., https://tests.yt-project.org/job/yt_py310_git/8375/

all (at least all that I looked at...) 225 failures are for enzo with similar errors:

Traceback (most recent call last):
  File "/usr/lib/python3.10/shelve.py", line 111, in __getitem__
    value = self.cache[key]
KeyError: 'galaxy0030'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib/python3.10/unittest/case.py", line 591, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
    method()
  File "/home/fido/workspace/yt_py310_git/.venv/lib/python3.10/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/fido/workspace/yt_py310_git/yt/utilities/answer_testing/framework.py", line 418, in __call__
    dd = self.reference_storage.get(self.storage_name)
  File "/home/fido/workspace/yt_py310_git/yt/utilities/answer_testing/framework.py", line 295, in get
    result = ds[answer_name]
  File "/usr/lib/python3.10/shelve.py", line 114, in __getitem__
    value = Unpickler(f).load()
ModuleNotFoundError: No module named 'numpy._core.numeric'

maybe a pickle/numpy version change issue?

[neutrinoceros]

maybe a pickle/numpy version change issue?

Hum, numpy itself didn't change between the last successful run and these (it was and still is at 2.0.1), and we didn't upgrade Python itself (including the standard lib) for a year or two, so unfortunately I think it's not that clear what's happening.

what did change in the interval was sympy (1.13.1->1.13.2) which has historically often required use to bump some pickles, and PyYaml (6.0.1->6.0.2), but I think this one is a lot less suspicious in this context.
I'll try to set an upper cap on both to see if that fixes it, and if so, we'll be able to bump these answers knowing that we're not just working around an unknown issue.

[Xarthisius]

Those test started failing on main on 08/13 (see #testing on slack)

[Xarthisius]

Comparing last passed run and failed run the thing that changes is this:

Collecting numpy<3,>=1.19.3 (from yt==4.4.dev0)
  Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)

something is pulling wrong, old version of numpy

[Xarthisius]

The culprit is this dependency:

-  Downloading imageio-2.34.2-py3-none-any.whl.metadata (4.9 kB)
+  Downloading imageio-2.35.0-py3-none-any.whl.metadata (5.0 kB)

which has numpy<2 and was released on 08/12. Issue was noticed and fixed upstream: imageio/imageio#1098
but I guess they haven't released new version yet.

[chrishavlin]

awesome, thanks for digging @Xarthisius ! So pinning imagio!=2.35.0 should do the trick. I'll push up a PR to try that out.

[neutrinoceros]

Thanks @Xarthisius for pinning it down ! I did look to see if numpy was somehow downgraded to 1.x but missed it because it can actually end up installed multiple times and the first one was 2.0.1 . I feel like we should maybe add a pip freeze or pip list call after all installs are done and before tests are run so it's easier to diff that part of the logs. Is this something we can do on the repo or is it a private script ?