Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow newer versions of xarray #1140

Closed
heikoklein opened this issue Apr 30, 2024 · 3 comments
Closed

Allow newer versions of xarray #1140

heikoklein opened this issue Apr 30, 2024 · 3 comments
Labels
dependencies Issues related to pyaerocom dependencies enhancement ✨ New feature or request
Milestone
m2024-05

Comments

@heikoklein
Copy link
Member

Is your feature request related to a problem? Please describe.
xarray is currently restricted to <=2022.10. pyaerocom should work with latest releases of xarray.

This issue is related to #1066 .

Describe the solution you would like to see
A clear and concise description of what you want to happen.

  • xarray should only depend on a minimum, but not on a maximum version, e.g. xarray >= 2020.10
@heikoklein heikoklein added the enhancement ✨ New feature or request label Apr 30, 2024
@heikoklein heikoklein added this to the m2024-06 milestone Apr 30, 2024
@heikoklein heikoklein added the dependencies Issues related to pyaerocom dependencies label Apr 30, 2024
@thorbjoernl
Copy link
Collaborator

thorbjoernl commented May 2, 2024 •
edited
Loading

Upgrading to the most recent release of xarray (2024.3.0) runs into a Segmentation Fault:

Fatal Python error: Segmentation fault

[...]

Current thread 0x00007f74d81a4140 (most recent call first):
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 217 in _acquire_with_cache_info
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 199 in acquire_context
  File "/modules/rhel8/user-apps/fou-modules/python/python_3.11.3/lib/python3.11/contextlib.py", line 137 in __enter__
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 412 in _acquire
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 418 in ds
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 356 in __init__
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 409 in open
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 646 in open_dataset
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/xarray/backends/api.py", line 573 in open_dataset
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/xarray/backends/api.py", line 749 in open_dataarray
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/pyaerocom/colocateddata.py", line 1287 in read_netcdf
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/pyaerocom/colocateddata.py", line 1361 in open
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/pyaerocom/colocateddata.py", line 135 in validate_data
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pydantic/main.py", line 175 in __init__
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/tests/fixtures/collocated_data.py", line 253 in <lambda>
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/tests/fixtures/collocated_data.py", line 265 in coldata
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 913 in call_fixture_func
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 1132 in pytest_fixture_setup
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 1079 in execute
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 647 in _compute_fixture_value
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 570 in _get_active_fixturedef
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 541 in getfixturevalue
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 687 in _fillfixtures
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/python.py", line 1775 in setup
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/runner.py", line 512 in setup
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/runner.py", line 158 in pytest_runtest_setup
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/runner.py", line 240 in <lambda>
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/runner.py", line 340 in from_call
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/runner.py", line 239 in call_and_report
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/runner.py", line 128 in runtestprotocol
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/runner.py", line 115 in pytest_runtest_protocol
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/main.py", line 339 in _main
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 174 in main
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 197 in console_main
  File "/lustre/storeB/users/thlun8736/python/pyaerocom/venv/bin/pytest", line 8 in <module>

This happens at this call to xr.open_dataarray().

Setting cache=False prevents the Segmentation Fault so I think it may be related to this open issue on the xarray repo, in particular this comment.

@thorbjoernl
Copy link
Collaborator

Another fix appears to be engine="h5netcdf" also appears to resolve it. This is consistent with the table here.

@thorbjoernl thorbjoernl mentioned this issue May 2, 2024
Closed
9 tasks
@thorbjoernl
Copy link
Collaborator

Fixed in #1150

This was referenced May 3, 2024
Merged
Closed
Closed
@heikoklein heikoklein modified the milestones: m2024-06, m2024-05 May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
dependencies Issues related to pyaerocom dependencies enhancement ✨ New feature or request
Projects
None yet
Milestone
m2024-05
Development

Successfully merging a pull request may close this issue.

Upgrade xarray dependency metno/pyaerocom
2 participants
@heikoklein @thorbjoernl