Cannot generate composites on bigger extents using MTG FCI Level-1c files

[AlejandroGorgues]

Describe the bug
I am using the fci_l1c_nc reader to generate the composites.
When the extent used is big or the full extent of the scene is used, two things could happen:

• A Segmentation fault (core dumped) message
• A NetCDF error:

NetCDF: HDF error
Traceback (most recent call last):
  File "/usr/src/app/test/MTGSatpy.py", line 169, in <module>
    satpy.run_script()
  File "/usr/src/app/test/MTGSatpy.py", line 160, in run_script
    self.get_composites()
  File "/usr/src/app/test/MTGSatpy.py", line 115, in get_composites
    local_scene.save_datasets(writer='geotiff', filename=str(file_no_projected), fill_value=0)
  File "/usr/local/lib/python3.12/dist-packages/satpy/scene.py", line 1296, in save_datasets
    return writer.save_datasets(dataarrays, compute=compute, **save_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/satpy/writers/__init__.py", line 756, in save_datasets
    return compute_writer_results([results])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/satpy/writers/__init__.py", line 594, in compute_writer_results
    da.compute(delayeds)
  File "/usr/local/lib/python3.12/dist-packages/dask/base.py", line 660, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 5079, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 6051, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

To Reproduce

import hdf5plugin
from satpy.scene import Scene
from satpy import find_files_and_readers
# data_input corresponds to the path where the .nc resides
files = find_files_and_readers(base_dir=data_input, reader='fci_l1c_nc')
# output file path
file_no_projected = Path(data_output, f'NO_PROJECTED_{composite["COMPOSITE_NAME"]}.tif')
scn = Scene(filenames=files, reader='fci_l1c_nc')
# composite is a dictionary that contains the key SATPY_NAME associated to the name in the satpy library (ir_105, vis_06 or colorized_ir_clouds)
scn.load([composite["SATPY_NAME"]], upper_right_corner="NE")
""" 
model contains general data that includes the extend that surrounds the sea:
"EXTENT_EUROPE":{
        "XMAX": 20.0,
        "XMIN": -45.0,
        "YMAX": 55.0,
        "YMIN": 20.0
    },
"""
local_scene = scn.crop(ll_bbox=(model["EXTENT_EUROPE"]["XMIN"], model["EXTENT_EUROPE"]["YMIN"], \
                    model["EXTENT_EUROPE"]["XMAX"], model["EXTENT_EUROPE"]["YMAX"]))
local_scene.save_datasets(writer='geotiff', filename=str(file_no_projected))

Expected behavior
The geotiff file associated to the product requested

Actual results
Included above.
Also on both results mentioned previously, I executed the code with debug_on and debug_off, showing the same messages:

[DEBUG: 2025-01-07 15:18:19 : satpy.writers] Reading ['/usr/local/lib/python3.12/dist-packages/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] GDAL data found in package: path='/usr/local/lib/python3.12/dist-packages/rasterio/gdal_data'.
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] PROJ data found in package: path='/usr/local/lib/python3.12/dist-packages/rasterio/proj_data'.
[DEBUG: 2025-01-07 15:18:19 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.12/dist-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2025-01-07 15:18:19 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.12/dist-packages/satpy/etc/enhancements/fci.yaml
[DEBUG: 2025-01-07 15:18:19 : satpy.writers] Data for DataID(name='vis_06', wavelength=WavelengthRange(min=0.59, central=0.64, max=0.69, unit='µm'), resolution=500, calibration=<1>, modifiers=()) will be enhanced with options:
        [{'name': 'linear_stretch', 'method': <function stretch at 0x7f010b1f72e0>, 'kwargs': {'stretch': 'crude', 'min_stretch': 0.0, 'max_stretch': 100.0}}, {'name': 'gamma', 'method': <function gamma at 0x7f010b954040>, 'kwargs': {'gamma': 1.5}}]
[DEBUG: 2025-01-07 15:18:19 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': 0.0, 'max_stretch': 100.0}
[DEBUG: 2025-01-07 15:18:19 : trollimage.xrimage] Applying gamma 1.5
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f010b147c20>
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Starting outermost env
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] No GDAL environment exists
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f0107ab9330> created
[DEBUG: 2025-01-07 15:18:19 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2025-01-07 15:18:19 : rasterio._env] GDAL data found in package: path='/usr/local/lib/python3.12/dist-packages/rasterio/gdal_data'.
[DEBUG: 2025-01-07 15:18:19 : rasterio._env] PROJ data found in package: path='/usr/local/lib/python3.12/dist-packages/rasterio/proj_data'.
[DEBUG: 2025-01-07 15:18:19 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x7f0107ab9330>.
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f010b147c20>
[DEBUG: 2025-01-07 15:18:19 : rasterio._io] Skipped delete for overwrite, dataset does not exist: '/usr/src/app/test/satpy_tif_mtg/NO_PROJECTED_vis_06.tif'
[DEBUG: 2025-01-07 15:18:19 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2025-01-07 15:18:19 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f010b147c20>
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f0107ab9330> options
[DEBUG: 2025-01-07 15:18:19 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f0107ab9330>.
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Exiting outermost env
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f010b147c20>
[INFO: 2025-01-07 15:18:19 : satpy.writers] Computing and writing results...

Screenshots
N/A

Environment Info:

• OS: GNU/Linux
• Satpy Version: 0.53.0
• PyResample Version: 1.31.0
• Readers and writers dependencies (when relevant):

Readers
=======
abi_l1b:  ok
abi_l1b_scmi:  ok
abi_l2_nc:  ok
acspo:  ok
agri_fy4a_l1:  ok
agri_fy4b_l1:  ok
ahi_hrit:  ok
ahi_hsd:  ok
ahi_l1b_gridded_bin:  ok
ahi_l2_nc:  ok
ami_l1b:  ok
amsr2_l1b:  ok
amsr2_l2:  ok
amsr2_l2_gaasp:  ok
amsub_l1c_aapp:  ok
ascat_l2_soilmoisture_bufr:  cannot find module 'satpy.readers.ascat_l2_soilmoisture_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n           Error: ', ModuleNotFoundError("No module named 'eccodes'")))
atms_l1b_nc:  ok
atms_sdr_hdf5:  ok
avhrr_l1b_aapp:  ok
avhrr_l1b_eps:  ok
avhrr_l1b_gaclac:  cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt:  ok
avhrr_l1c_eum_gac_fdr_nc:  ok
caliop_l2_cloud:  cannot find module 'satpy.readers.caliop_l2_cloud' (cannot import name 'Dataset' from 'satpy.dataset' (/usr/local/lib/python3.12/dist-packages/satpy/dataset/__init__.py))
clavrx:  ok
cmsaf-claas2_l2_nc:  ok
electrol_hrit:  ok
epic_l1b_h5:  ok
fci_l1c_nc:  ok
fci_l2_bufr:  cannot find module 'satpy.readers.eum_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
fci_l2_grib:  cannot find module 'satpy.readers.eum_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
fci_l2_nc:  ok
fy3a_mersi1_l1b:  ok
fy3b_mersi1_l1b:  ok
fy3c_mersi1_l1b:  ok
generic_image:  cannot find module 'satpy.readers.generic_image' (No module named 'rioxarray')
geocat:  ok
gerb_l2_hr_h5:  ok
ghi_l1:  ok
ghrsst_l2:  ok
gld360_ualf2:  ok
glm_l2:  ok
gms5-vissr_l1b:  cannot find module 'satpy.readers.gms.gms5_vissr_l1b' (No module named 'numba')
goci2_l2_nc:  ok
goes-imager_hrit:  ok
goes-imager_nc:  ok
gpm_imerg:  ok
grib:  cannot find module 'satpy.readers.grib' (No module named 'pygrib')
hsaf_grib:  cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
hsaf_h5:  ok
hy2_scat_l2b_h5:  ok
iasi_l2:  ok
iasi_l2_cdr_nc:  ok
iasi_l2_so2_bufr:  cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n           Error: ', ModuleNotFoundError("No module named 'eccodes'")))
ici_l1b_nc:  ok
insat3d_img_l1b_h5:  ok
jami_hrit:  ok
li_l2_nc:  ok
maia:  ok
mcd12q1:  ok
meris_nc_sen3:  ok
mersi2_l1b:  ok
mersi3_l1b:  ok
mersi_ll_l1b:  ok
mersi_rm_l1b:  ok
mhs_l1c_aapp:  ok
mimicTPW2_comp:  ok
mirs:  ok
modis_l1b:  ok
modis_l2:  ok
modis_l3:  ok
msi_safe:  ok
msi_safe_l2a:  ok
msu_gsa_l1b:  ok
mtsat2-imager_hrit:  ok
mviri_l1b_fiduceo_nc:  ok
mwi_l1b_nc:  ok
mws_l1b_nc:  ok
nucaps:  ok
nwcsaf-geo:  ok
nwcsaf-msg2013-hdf5:  ok
nwcsaf-pps_nc:  ok
oceancolorcci_l3_nc:  ok
oci_l2_bgc:  ok
olci_l1b:  ok
olci_l2:  ok
oli_tirs_l1_tif:  ok
omps_edr:  ok
osisaf_nc:  ok
safe_sar_l2_ocn:  ok
sar-c_safe:  cannot find module 'satpy.readers.sar_c_safe' (No module named 'rioxarray')
satpy_cf_nc:  ok
scatsat1_l2b:  cannot find module 'satpy.readers.scatsat1_l2b' (cannot import name 'Dataset' from 'satpy.dataset' (/usr/local/lib/python3.12/dist-packages/satpy/dataset/__init__.py))
seadas_l2:  ok
seviri_l1b_hrit:  ok
seviri_l1b_icare:  ok
seviri_l1b_native:  ok
seviri_l1b_nc:  ok
seviri_l2_bufr:  cannot find module 'satpy.readers.eum_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
seviri_l2_grib:  cannot find module 'satpy.readers.eum_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
sgli_l1b:  ok
slstr_l1b:  ok
smos_l2_wind:  ok
tropomi_l2:  ok
vii_l1b_nc:  ok
vii_l2_nc:  ok
viirs_compact:  ok
viirs_edr:  ok
viirs_edr_active_fires:  ok
viirs_edr_flood:  ok
viirs_l1b:  ok
viirs_l2:  ok
viirs_sdr:  ok
viirs_vgac_l1c_nc:  ok
virr_l1b:  ok

Writers
=======
awips_tiled:  ok
cf:  ok
geotiff:  ok
mitiff:  ok
ninjogeotiff:  ok
ninjotiff:  cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
simple_image:  ok

Versions
======
platform: Linux-6.1.61-85.141.amzn2023.x86_64-x86_64-with-glibc2.39
python: 3.12.3

cartopy: not installed
dask: 2024.12.1
fsspec: 2024.12.0
gdal: 3.9.2
geoviews: not installed
h5netcdf: 1.4.1
h5py: 3.12.1
netcdf4: 1.7.2
numpy: 1.26.4
pyhdf: 0.11.6
pyproj: 3.7.0
rasterio: 1.4.2
xarray: 2025.1.0

Additional context

1. I tested this process in the past (around september) with older versions such as 0.51.0 and 0.52.0 and I made it work, however now even with older versions, I cannot generate the products.
2. I tested this process with smaller extents than the mentioned in the code and it worked.
3. I think that due to the volatility of the MTG base files, some configuration could be changed, but not updated on the satpy library

[gerritholl]

Not enough RAM? Try adapting some of the parameters suggested at https://satpy.readthedocs.io/en/stable/faq.html

[AlejandroGorgues]

Hello!
I have adjusted the parameters as suggested in the documentation and tried all the combinations between the different suggestions in the list below.

• Change GDAL_CACHEMAX to '15%' or '30%'
• Change num_workers for dask operations to 4 (this is the max num of cores that I have)
• Change OMP_NUM_THREADS to 2 (half the num_workers)
• Added num_threads=2 while saving the product as geotiff
• Added generate=False while loading the composite

Despite these changes, the two error messages mentioned earlier appears.

Also, if I resample the scene using a gradient_search or a nearest for example, to get the ir_105, the GeoTIFF is sometimes generated successfully. However, at other times, the same errors appears

I also have around 13Gb available memory (16Gb total)

[meteomara]

I had a similar issue and was able to solve it by downgrading the netCDF4 package to version 1.6.5.

[AlejandroGorgues]

Thanks for the answer!!

I tried downgrading it to 1.6.5 and some things happened:

• Could get ir_105 and ir_38 products.
• When I generate nir_22 sometimes I can and sometimes I can't.
• I can't generate vis_06 at all.

[DomRuk]

Hi, downgrading to netCDF4 1.6.5 solved the problem for me. Check if other library incompatibilities are the issue. h5py, hf5plugin, numpy....

[AlejandroGorgues]

Hi, downgrading to netCDF4 1.6.5 solved the problem for me. Check if other library incompatibilities are the issue. h5py, hf5plugin, numpy....

Ok, so I did that in the past and I could not generate any product, but now that I played more with different versions of other libraries like h5py or h5netcdf and I think I found out the solution by downgrading h5py version to 3.11.0. while keeping the latest versions for the rest of the libraries (including netCDF4 on 1.7.2 version).

Thank you all for your help!!. I will keep an eye on the libraries from now on, as for this issue, it blocked me a lot.

As I can generate all products right now, I close this issue.