set_multiple_rc_from_isimip does not compute results for future events/files

[LucasWouters]

Hi,

I've been running the set_multiple_rc_from_isimip function to calculate the historical means for ISIMIP files according to the tutorial: Crop production risk based on ISIMIP and FAO data tutorial. My goal is to compare baseline and future crop yield values.

It works fine for historical files but does not compute results for future files. It seems to unable to read the files

Input data examples (masked to Australia to reduce run time):

• gepic_gfdl-esm2m_ewembi_historical_2005soc_co2_yield-mai-firr_aus_annual_1861_2005.nc
• gepic_gfdl-esm2m_ewembi_rcp60_2005soc_co2_yield-mai-firr_aus_annual_2006_2099.nc

I'm running a recent stable CLIMADA version.

data_path = CONFIG.local_data.save_dir.dir() / r"C:\Shared" # set path of working data directory
input_dir = data_path / "ISIMIP_crop" # set path where you place hazard input data
input_dir.mkdir(parents=True, exist_ok=True)


output_dir = data_path / "climada_results" # set output directory
path_hist_mean = output_dir / "Hist_mean" # set output directory for hist_mean
path_hist_mean.mkdir(parents=True, exist_ok=True)

filelist_haz, hazards_list = set_multiple_rc_from_isimip(input_dir=input_dir, output_dir=output_dir,
                                                      isimip_run='ISIMIP2b', return_data=True)

This only returns output for the historical file.

I've been digging into the functions (running calc_fut_haz_isimip separately), but to no avail. There seems to be issues with reading rasters, as Rasterio cannot find the future files:

haz = haz_fut.from_raster([str(Path(input_dir, filename))], band=id_bands,
                      geometry=list([shapely.geometry.box(lonmin, latmin, lonmax, latmax)]))
2023-05-03 09:28:09,216 - climada.util.coordinates - INFO - Reading C:\Shared\ISIMIP_crop\gepic_gfdl-esm2m_ewembi_rcp60_2005soc_co2_yield-mai-firr_aus_annual_2006_2099.nc
2023-05-03 09:28:09,216 - climada.util.coordinates - INFO - Reading C:\Shared\ISIMIP_crop\gepic_gfdl-esm2m_ewembi_rcp60_2005soc_co2_yield-mai-firr_aus_annual_2006_2099.nc
Traceback (most recent call last):

  File "rasterio\_base.pyx", line 261, in rasterio._base.DatasetBase.__init__

  File "rasterio\_shim.pyx", line 78, in rasterio._shim.open_dataset

  File "rasterio\_err.pyx", line 216, in rasterio._err.exc_wrap_pointer

CPLE_OpenFailedError: C:/Shared/ISIMIP_crop/gepic_gfdl-esm2m_ewembi_rcp60_2005soc_co2_yield-mai-firr_aus_annual_2006_2099.nc: No such file or directory


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "C:\Users\LucasWouters\AppData\Local\Temp\2\ipykernel_3076\818546461.py", line 1, in <module>
    haz = haz_fut.from_raster([str(Path(input_dir, filename))], band=id_bands,

  File "c:\users\lucaswouters\climada_workspace\climada_python\climada\hazard\base.py", line 357, in from_raster
    centroids = Centroids.from_raster_file(

  File "c:\users\lucaswouters\climada_workspace\climada_python\climada\hazard\centroids\centr.py", line 447, in from_raster_file
    meta, _ = u_coord.read_raster(

  File "c:\users\lucaswouters\climada_workspace\climada_python\climada\util\coordinates.py", line 1902, in read_raster
    with rasterio.open(_add_gdal_vsi_prefix(file_name), 'r') as src:

  File "C:\ProgramData\Anaconda3\envs\climada_env\lib\site-packages\rasterio\env.py", line 437, in wrapper
    return f(*args, **kwds)

  File "C:\ProgramData\Anaconda3\envs\climada_env\lib\site-packages\rasterio\__init__.py", line 220, in open
    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)

  File "rasterio\_base.pyx", line 263, in rasterio._base.DatasetBase.__init__

RasterioIOError: C:/Shared/ISIMIP_crop/gepic_gfdl-esm2m_ewembi_rcp60_2005soc_co2_yield-mai-firr_aus_annual_2006_2099.nc: No such file or directory

Is this because of the dependencies or am I missing an important step here?

[chahank]

It is because the file does not exist at C:/Shared/ISIMIP_crop/gepic_gfdl-esm2m_ewembi_rcp60_2005soc_co2_yield-mai-firr_aus_annual_2006_2099.nc. Please check whether the file is there.

[LucasWouters]

These are the exact paths in my input folder:

• "C:\Shared\ISIMIP_crop\gepic_gfdl-esm2m_ewembi_historical_2005soc_co2_yield-mai-firr_aus_annual_1861_2005.nc4"
• "C:\Shared\ISIMIP_crop\gepic_gfdl-esm2m_ewembi_rcp60_2005soc_co2_yield-mai-firr_aus_annual_2006_2099.nc4"
  The first one runs, the second does not. That is why I'm confused, since it's clear that the file is there.

[peanutfun]

The issue does not seem to be related to the Climada code base. @LucasWouters please make sure the files are present under these exact paths and your Python interpreter/environment can access them.

[LucasWouters]

@peanutfun , I've downloaded multiple files (historical and future) from the ISIMIP repository, saved them in the same folder and not altered the paths. In all cases, Climada finds the historical files, but not the future files.

[peanutfun]

From what I can see set_multiple_rc_from_isimip tries to determine the file paths from the input parameters. I guess the default settings do not match the scenario for the future files you have. Please have a look at the method docstring.

To check if there is an issue with reading a single file, please try the following:

import xarray as xr

with xr.open_dataset("C:\Shared\ISIMIP_crop\gepic_gfdl-esm2m_ewembi_rcp60_2005soc_co2_yield-mai-firr_aus_annual_2006_2099.nc4") as dset:
    print(dset)

[LucasWouters]

I'm able the read both files manually using xarray, so the problem does not seem to be with the input data itself.

The set_multiple_rc_from_isimip function should take the input parameters from the future filenames as well, as both the scenario ("rcp60") as yearrange ("2006_2099") parameters are listed in the ISIMIP input data specific global variables (default).

[peanutfun]

Alright, seems like it's related to the code base after all. Thanks for reporting back! I am afraid I cannot help with that issue. The code is several years old and the original author is not around anymore. If you happen to find a solution, a contribution would be most welcome!

[carmensteinmann]

@LucasWouters I helped out in writing this code back in the days. One issue with this module is that many of the functions are hard coded to use the filename to populate objects. I think in your case the problem results due to the variable "file_props" because you changed part of the file name from "global_annual" to "aus_annual", but our code uses the default FN_STR_VAR = "global_annual". Could you test whether it all runs smoothly if you use the original file name? (The cropping of the data shouldn't be the issue, just the naming.)

[LucasWouters]

Hi @carmensteinmann , thanks for getting back. I noticed this as well, so I tried 2 things:

• downloading the global ISIMIP files
• changing the name of the cropped files from "aus_annual" to "global_annual"
  However, both didn't work for the future files (but did for the historical)