Newly added opener argument and subsequent behavior

[digital-idiot]

I was trying out the newly added opener argument in fiona.open and observed the following behavior. I am trying to understand why different driver behaves differently.

@sgillies explained that opener feature was introduced in v1.10. However I am observing the exact same behavior with fiona 1.10.1 as well. To avoid spamming the issues, I asking it again here.

The Issue

When creating and writing to new file by passing a fsspec fs object as opener argument, depending upon the driver creation/write fails or successful. For local filesystem everything works as expected, this has been observed specifically with S3. Except GeoJSON other formats I tested fails. Am I doing something wrong?

Steps to reproduce the problem.

Below is the script to reproduce the problem. First the common part of the script followed by variations and subsequent outcomes.

Common Code Section

import fiona
import fsspec
from fiona.crs import from_epsg


fs = fsspec.filesystem(protocol="s3", key="KKKKKKKKKK", secret="XXXXXXXXXXXXX")

schema = {
    'geometry': 'Polygon',
    'properties': {
        'id': 'int',
        'name': 'str'
    }
}

# Create some sample polygons
polygons = [
    {
        'geometry': {
            'type': 'Polygon',
            'coordinates': [[(0, 0), (0, 1), (1, 1), (1, 0), (0, 0)]]
        },
        'properties': {
            'id': 1,
            'name': 'square'
        }
    },
    {
        'geometry': {
            'type': 'Polygon',
            'coordinates': [[(2, 2), (2, 3), (3, 2), (2, 2)]]
        },
        'properties': {
            'id': 2,
            'name': 'triangle'
        }
    }
]

Export to GeoJSON

with fiona.open(
    's3://bucket/path/vector_layer.geojson',
    'w',
    driver='GeoJSON',
    crs=from_epsg(4326),
    schema=schema,
    opener=fs
) as dst:
    for polygon in polygons:
        dst.write(polygon)

print("Write Completed Successfully!")

Output

Write Completed Successfully!

Export to GPKG

with fiona.open(
    's3://bucket/path/vector_layer.gpkg',
    'w',
    driver='GPKG',
    crs=from_epsg(4326),
    schema=schema,
    opener=fs
) as dst:
    for polygon in polygons:
        dst.write(polygon)

print("Write Completed Successfully!")

Output

---------------------------------------------------------------------------
CPLE_OpenFailedError                      Traceback (most recent call last)
File fiona/ogrext.pyx:175, in fiona.ogrext.gdal_create()

File fiona/_err.pyx:291, in fiona._err.exc_wrap_pointer()

CPLE_OpenFailedError: sqlite3_open(/vsis3/bucket/path/vector_layer.gpkg) failed: unable to open database file

During handling of the above exception, another exception occurred:

DriverError                               Traceback (most recent call last)
Cell In[45], line 1
----> 1 with fiona.open(
      2     's3://bucket/path/vector_layer.gpkg',
      3     'w',
      4     driver='GPKG',
      5     crs=from_epsg(4326),
      6     schema=schema,
      7     opener=fs
      8 ) as dst:
      9     for polygon in polygons:
     10         dst.write(polygon)

File ~/.micromamba/envs/PyDev/lib/python3.12/site-packages/fiona/env.py:457, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    454     session = DummySession()
    456 with env_ctor(session=session):
--> 457     return f(*args, **kwds)

File ~/.micromamba/envs/PyDev/lib/python3.12/site-packages/fiona/__init__.py:303, in open(fp, mode, driver, schema, crs, encoding, layer, vfs, enabled_drivers, crs_wkt, allow_unsupported_drivers, **kwargs)
    292     colxn = Collection(
    293         path,
    294         mode,
   (...)
    300         **kwargs
    301     )
    302 elif mode == "w":
--> 303     colxn = Collection(
    304         path,
    305         mode,
    306         crs=crs,
    307         driver=driver,
    308         schema=schema,
    309         encoding=encoding,
    310         layer=layer,
    311         enabled_drivers=enabled_drivers,
    312         crs_wkt=crs_wkt,
    313         allow_unsupported_drivers=allow_unsupported_drivers,
    314         **kwargs
    315     )
    316 else:
    317     raise ValueError("mode string must be one of {'r', 'w', 'a'}")

File ~/.micromamba/envs/PyDev/lib/python3.12/site-packages/fiona/collection.py:246, in Collection.__init__(self, path, mode, driver, schema, crs, encoding, layer, vsi, archive, enabled_drivers, crs_wkt, ignore_fields, ignore_geometry, include_fields, wkt_version, allow_unsupported_drivers, **kwargs)
    244     elif self.mode in ("a", "w"):
    245         self.session = WritingSession()
--> 246         self.session.start(self, **kwargs)
    247 except OSError:
    248     self.session = None

File fiona/ogrext.pyx:1110, in fiona.ogrext.WritingSession.start()

File fiona/ogrext.pyx:1111, in fiona.ogrext.WritingSession.start()

File fiona/ogrext.pyx:179, in fiona.ogrext.gdal_create()

DriverError: sqlite3_open(/vsis3/bucket/path/vector_layer.gpkg) failed: unable to open database file

Export to ESRI Shapefile

with fiona.open(
    's3://bucket/path/vector_layer.shp',
    'w',
    driver='ESRI Shapefile',
    crs=from_epsg(4326),
    schema=schema,
    opener=fs
) as dst:
    for polygon in polygons:
        dst.write(polygon)

print("Write Completed Successfully!")

Output

---------------------------------------------------------------------------
CPLE_AppDefinedError                      Traceback (most recent call last)
File fiona/ogrext.pyx:1239, in fiona.ogrext.WritingSession.start()

File fiona/ogrext.pyx:1240, in fiona.ogrext.WritingSession.start()

File fiona/_err.pyx:291, in fiona._err.exc_wrap_pointer()

CPLE_AppDefinedError: Failed to create file /vsis3/bucket/path/vector_layer.shp: Permission denied

During handling of the above exception, another exception occurred:

DriverIOError                             Traceback (most recent call last)
Cell In[46], line 1
----> 1 with fiona.open(
      2     's3://bucket/path/vector_layer.shp',
      3     'w',
      4     driver='ESRI Shapefile',
      5     crs=from_epsg(4326),
      6     schema=schema,
      7     opener=fs
      8 ) as dst:
      9     for polygon in polygons:
     10         dst.write(polygon)

File ~/.micromamba/envs/PyDev/lib/python3.12/site-packages/fiona/env.py:457, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    454     session = DummySession()
    456 with env_ctor(session=session):
--> 457     return f(*args, **kwds)

File ~/.micromamba/envs/PyDev/lib/python3.12/site-packages/fiona/__init__.py:303, in open(fp, mode, driver, schema, crs, encoding, layer, vfs, enabled_drivers, crs_wkt, allow_unsupported_drivers, **kwargs)
    292     colxn = Collection(
    293         path,
    294         mode,
   (...)
    300         **kwargs
    301     )
    302 elif mode == "w":
--> 303     colxn = Collection(
    304         path,
    305         mode,
    306         crs=crs,
    307         driver=driver,
    308         schema=schema,
    309         encoding=encoding,
    310         layer=layer,
    311         enabled_drivers=enabled_drivers,
    312         crs_wkt=crs_wkt,
    313         allow_unsupported_drivers=allow_unsupported_drivers,
    314         **kwargs
    315     )
    316 else:
    317     raise ValueError("mode string must be one of {'r', 'w', 'a'}")

File ~/.micromamba/envs/PyDev/lib/python3.12/site-packages/fiona/collection.py:246, in Collection.__init__(self, path, mode, driver, schema, crs, encoding, layer, vsi, archive, enabled_drivers, crs_wkt, ignore_fields, ignore_geometry, include_fields, wkt_version, allow_unsupported_drivers, **kwargs)
    244     elif self.mode in ("a", "w"):
    245         self.session = WritingSession()
--> 246         self.session.start(self, **kwargs)
    247 except OSError:
    248     self.session = None

File fiona/ogrext.pyx:1248, in fiona.ogrext.WritingSession.start()

DriverIOError: Failed to create file /vsis3/bucket/path/vector_layer.shp: Permission denied

Demonstrably, the creation of both GPKG and ESRI Shapefile fails but stating different errors. Since the creation and writing of GeoJSON is successful the underlying issue is unlikely to be an access issue or permission issue. So the question is are these expected behaviors? Is the requirement of creating auxiliary files (.dbf, .prj etc) in the case of ESRI Shapefile playing a role?

Operating system

MacOS 15.2 (24C101) [Kernel: Darwin 24.2.0]

Fiona and GDAL version and provenance

• Environment Manager: micromamba
• Channel: conda-forge
  • fiona=1.10.1
  • GDAL=3.9.2

[sgillies]

@digital-idiot in trying to reproduce this, I've run into the same issue you found at fsspec/s3fs#925.

GDAL's vector drivers are rather different in their writing implementations. For some, like Shapefile, a local temporary directory that is later uploaded to S3 may be required.

[digital-idiot]

I was also thinking the same, it is because of the aux files (dbf, prj etc). GPKG seem to be a single sqlite file (at least on the surface), what happens in that case? Does it try to crate shm and wal file and hence the issue?

[sgillies]

@digital-idiot it's not the auxiliary files themselves, but the incremental writes to them. We can copy data to S3 (using HTTP PUT), but we can't generally write or append.

There's a new exception that I haven't had the opportunity to try: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-express-one-zone-append-data-object/. I suspect GDAL/OGR, fsspec, etc are not ready to take advantage of that.

[digital-idiot]

Thanks @sgillies for patiently explaining this. I will stick to GeoJSON for now.