qgis_process doesn't seem to handle remote layers #54264

florisvdh · 2023-08-16T15:15:00Z

What is the bug or the crash?

When providing a remote layer as input argument – I tried both a WCS and a COG – qgis_process says it 'could not load source layer'. Below examples do work in the QGIS GUI though, as do the associated Python processing.run commands in QGIS.

Steps to reproduce the issue

Below qgis_process syntax was obtained by clicking 'Copy as qgis_process command' in the QGIS processing dialog.

$ wget -q https://github.com/r-spatial/sf/raw/main/inst/gpkg/b_pump.gpkg
$ 
$ qgis_process run native:rastersampling --distance_units=meters --area_units=m2 --ellipsoid=EPSG:7030 --INPUT='b_pump.gpkg|layername=b_pump' --RASTERCOPY='/vsicurl/https://s3.eu-central-1.wasabisys.com/openlandmap/dtm/dtm.bareearth_ensemble_p10_30m_s_2018_go_epsg4326_v20230221.tif' --COLUMN_PREFIX=SAMPLE_ --OUTPUT=TEMPORARY_OUTPUT

----------------
Inputs
----------------

COLUMN_PREFIX:	SAMPLE_
INPUT:	b_pump.gpkg|layername=b_pump
OUTPUT:	TEMPORARY_OUTPUT
RASTERCOPY:	/vsicurl/https://s3.eu-central-1.wasabisys.com/openlandmap/dtm/dtm.bareearth_ensemble_p10_30m_s_2018_go_epsg4326_v20230221.tif

Using ellipsoid:	EPSG:7030
Using distance unit:	meters
Using area unit:	square meters
ERROR:	An error was encountered while checking parameter values
	Could not load source layer for RASTERCOPY: /vsicurl/https://s3.eu-central-1.wasabisys.com/openlandmap/dtm/dtm.bareearth_ensemble_p10_30m_s_2018_go_epsg4326_v20230221.tif not found
$ 
$ qgis_process run native:rastersampling --distance_units=meters --area_units=m2 --ellipsoid=EPSG:7030 --INPUT='b_pump.gpkg|layername=b_pump' --RASTERCOPY='dpiMode=7&identifier=1&tilePixelRatio=0&url=https://image.discomap.eea.europa.eu/arcgis/services/Elevation/EUElev_DEM_V11/MapServer/WCSServer' --COLUMN_PREFIX=SAMPLE_ --OUTPUT=TEMPORARY_OUTPUT

----------------
Inputs
----------------

COLUMN_PREFIX:	SAMPLE_
INPUT:	b_pump.gpkg|layername=b_pump
OUTPUT:	TEMPORARY_OUTPUT
RASTERCOPY:	dpiMode=7&identifier=1&tilePixelRatio=0&url=https://image.discomap.eea.europa.eu/arcgis/services/Elevation/EUElev_DEM_V11/MapServer/WCSServer

Using ellipsoid:	EPSG:7030
Using distance unit:	meters
Using area unit:	square meters
ERROR:	An error was encountered while checking parameter values
	Could not load source layer for RASTERCOPY: dpiMode=7&identifier=1&tilePixelRatio=0&url=https://image.discomap.eea.europa.eu/arcgis/services/Elevation/EUElev_DEM_V11/MapServer/WCSServer not found

Versions

$ qgis_process --version
QGIS 3.32.1-Lima 'Lima' (4a078ecf63)
QGIS code revision 4a078ecf63
Qt version 5.15.3
Python version 3.10.6
Compiled against GDAL/OGR 3.6.2
Running against GDAL/OGR 3.6.4
PROJ version 9.1.1
EPSG Registry database version v10.076 (2022-08-31)
GEOS version 3.11.1-CAPI-1.17.1
SQLite version 3.37.2
OS Linux Mint 21.2

Supported QGIS version

I'm running a supported QGIS version according to the roadmap.

New profile

I tried with a new QGIS profile

Additional context

The COG file from OpenLandMap is documented at Zenodo. The WCS is documented by the Copernicus programme; see also GetCapabilities page.

The text was updated successfully, but these errors were encountered:

florisvdh · 2023-09-12T12:18:10Z

Adding an additional example here, now using an online geopackage vector layer:

$ ogrinfo https://github.com/r-spatial/qgisprocess/raw/main/inst/longlake/longlake.gpkg
INFO: Open of `https://github.com/r-spatial/qgisprocess/raw/main/inst/longlake/longlake.gpkg'
      using driver `GPKG' successful.
1: longlake (3D Polygon)
$ 
$ qgis_process run native:buffer --INPUT='https://github.com/r-spatial/qgisprocess/raw/main/inst/longlake/longlake.gpkg|layername=longlake' --OUTPUT=TEMPORARY_OUTPUT

----------------
Inputs
----------------

INPUT:	https://github.com/r-spatial/qgisprocess/raw/main/inst/longlake/longlake.gpkg|layername=longlake
OUTPUT:	TEMPORARY_OUTPUT

ERROR:	An error was encountered while checking parameter values
	Could not load source layer for INPUT: https://github.com/r-spatial/qgisprocess/raw/main/inst/longlake/longlake.gpkg|layername=longlake not found
$
$ qgis_process --version
QGIS 3.32.2-Lima 'Lima' (d3e85889c7)
QGIS code revision d3e85889c7
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.6.4
PROJ version 9.1.1
EPSG Registry database version v10.076 (2022-08-31)
GEOS version 3.11.1-CAPI-1.17.1
SQLite version 3.37.2
OS Linux Mint 21.2

nirvn · 2023-10-26T03:15:30Z

@florisvdh , you'll need to prefix your input string with /vsicurl/ to tell the gdal provider that you are trying to access a remote dataset.

nyalldawson · 2023-10-26T03:56:19Z

@nirvn -- actually there's more to it then that. There's a hardcoded assumption in processing when layer sources are tested that if the provider is gdal or ogr, then we check QFile::exists and fail if the source doesn't exist. There's a number of similar assumptions scattered around which need to be re-worked.

nirvn · 2023-10-26T04:22:01Z

@nyalldawson , good to know.

jdlom · 2023-12-13T15:25:51Z

Same issue with S3 and /vsis3/

…files Its better to defer this to that data provider, as the uri may not need to correspond to an actual on disk file for some providers. Eg ogr provider can load remote http sources Fixes qgis#54264

…files Its better to defer this to that data provider, as the uri may not need to correspond to an actual on disk file for some providers. Eg ogr provider can load remote http sources Fixes #54264

florisvdh · 2024-01-29T10:22:54Z

Confirmed for the COG 🎉 using QGIS 3.34.3.

$ wget -q https://github.com/r-spatial/sf/raw/main/inst/gpkg/b_pump.gpkg
$ 
$ qgis_process run native:rastersampling --distance_units=meters --area_units=m2 --ellipsoid=EPSG:7030 --INPUT='b_pump.gpkg|layername=b_pump' --RASTERCOPY='/vsicurl/https://s3.eu-central-1.wasabisys.com/openlandmap/dtm/dtm.bareearth_ensemble_p10_30m_s_2018_go_epsg4326_v20230221.tif' --COLUMN_PREFIX=SAMPLE_ --OUTPUT=TEMPORARY_OUTPUT

----------------
Inputs
----------------

COLUMN_PREFIX:	SAMPLE_
INPUT:	b_pump.gpkg|layername=b_pump
OUTPUT:	TEMPORARY_OUTPUT
RASTERCOPY:	/vsicurl/https://s3.eu-central-1.wasabisys.com/openlandmap/dtm/dtm.bareearth_ensemble_p10_30m_s_2018_go_epsg4326_v20230221.tif

Using ellipsoid:	EPSG:7030
Using distance unit:	meters
Using area unit:	square meters


----------------
Results
----------------

OUTPUT:	Sampled_57097dec_c7ca_433c_903d_12999a86fe92

florisvdh · 2024-01-29T10:33:00Z

However not working with a WCS example.

Regarding the /vsicurl/ prefix, remind that I just took the qgis_process CLI string from the processing dialog using the Advanced button. It appears this works OK for COG but not for WCS.

Played around with WCS, but with no success (contrary to the GUI operation). See below results. Now using another WCS than the one in the top post, since that one is dead.

Extracted qgis_process command does not have /vsicurl/:

qgis_process run native:rastersampling --distance_units=meters --area_units=m2 --ellipsoid=EPSG:7030 --INPUT='/home/floris/b_pump.gpkg|layername=b_pump' --RASTERCOPY='cache=PreferNetwork&crs=EPSG:4326&dpiMode=7&format=GEOTIFF_INT16&identifier=nitrogen_0-5cm_Q0.5&tilePixelRatio=0&url=https://maps.isric.org/mapserv?map%3D/map/nitrogen.map%26COVERAGEID%3Dnitrogen_5-15cm_Q0.5' --COLUMN_PREFIX=SAMPLE_ --OUTPUT=TEMPORARY_OUTPUT

It gives the same error as before (Could not load source layer for RASTERCOPY).

Also not working when changing to /vsicurl/ as below.

$ wget -q https://github.com/r-spatial/sf/raw/main/inst/gpkg/b_pump.gpkg
$ 
$ qgis_process run native:rastersampling --distance_units=meters --area_units=m2 --ellipsoid=EPSG:7030 --INPUT='b_pump.gpkg|layername=b_pump' --RASTERCOPY='/vsicurl/https://maps.isric.org/mapserv?map=/map/nitrogen.map&COVERAGEID=nitrogen_5-15cm_Q0.5' --COLUMN_PREFIX=SAMPLE_ --OUTPUT=TEMPORARY_OUTPUT

----------------
Inputs
----------------

COLUMN_PREFIX:	SAMPLE_
INPUT:	b_pump.gpkg|layername=b_pump
OUTPUT:	TEMPORARY_OUTPUT
RASTERCOPY:	/vsicurl/https://maps.isric.org/mapserv?map=/map/nitrogen.map&COVERAGEID=nitrogen_5-15cm_Q0.5

Using ellipsoid:	EPSG:7030
Using distance unit:	meters
Using area unit:	square meters
ERROR:	An error was encountered while checking parameter values
	Could not load source layer for RASTERCOPY: /vsicurl/https://maps.isric.org/mapserv?map=/map/nitrogen.map&COVERAGEID=nitrogen_5-15cm_Q0.5 not found
$ 
$ qgis_process --version | head -1
QGIS 3.34.3-Prizren 'Prizren' (47373234acd)

florisvdh · 2024-01-29T14:17:12Z

Same observations observed for:

$ ~/git_repositories2/QGIS/build-0aed505/output/bin/qgis_process --version
QGIS 3.35.0-Master 'Master' (0aed5058)
QGIS code revision 0aed5058
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.6.4
PROJ version 9.1.1
EPSG Registry database version v10.076 (2022-08-31)
GEOS version 3.11.1-CAPI-1.17.1
SQLite version 3.37.2
OS Linux Mint 21.3

Use the safer method of explicitly including the provider type when storing layer sources to string in more places, which eg fixes running the copied qgis_process string when a non gdal layer source is used for a raster layer. Also fixes the same situation when restoring an algorithm from the history log. Fixes qgis#54264

Use the safer method of explicitly including the provider type when storing layer sources to string in more places, which eg fixes running the copied qgis_process string when a non gdal layer source is used for a raster layer. Also fixes the same situation when restoring an algorithm from the history log. Fixes #54264

florisvdh · 2024-02-06T11:41:51Z

WCS appears to work now 🎉 Thanks!! 🙏

$ ~/git_repositories2/QGIS/build-8dc3558/output/bin/qgis_process run native:rastersampling --distance_units=meters --area_units=m2 --ellipsoid=EPSG:7030 --INPUT='/home/floris/b_pump.gpkg|layername=b_pump' --RASTERCOPY='wcs://cache=PreferNetwork&crs=EPSG:4326&dpiMode=7&format=GEOTIFF_INT16&identifier=nitrogen_0-5cm_Q0.5&tilePixelRatio=0&url=https://maps.isric.org/mapserv?map%3D/map/nitrogen.map%26COVERAGEID%3Dnitrogen_5-15cm_Q0.5' --COLUMN_PREFIX=SAMPLE_ --OUTPUT=TEMPORARY_OUTPUT

----------------
Inputs
----------------

COLUMN_PREFIX:	SAMPLE_
INPUT:	/home/floris/b_pump.gpkg|layername=b_pump
OUTPUT:	TEMPORARY_OUTPUT
RASTERCOPY:	wcs://cache=PreferNetwork&crs=EPSG:4326&dpiMode=7&format=GEOTIFF_INT16&identifier=nitrogen_0-5cm_Q0.5&tilePixelRatio=0&url=https://maps.isric.org/mapserv?map%3D/map/nitrogen.map%26COVERAGEID%3Dnitrogen_5-15cm_Q0.5

Using ellipsoid:	EPSG:7030
Using distance unit:	meters
Using area unit:	square meters
proj_create_from_database: crs not found
proj_create_from_database: crs not found
proj_create_from_database: crs not found

Using non-preferred coordinate operation between  and EPSG:4326. Using +proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.999601 +x_0=400000 +y_0=-100000 +ellps=airy +step +proj=push +v_3 +step +proj=cart +ellps=airy +step +proj=helmert +x=446.448 +y=-125.157 +z=542.06 +rx=0.15 +ry=0.247 +rz=0.842 +s=-20.489 +convention=position_vector +step +inv +proj=cart +ellps=WGS84 +step +proj=pop +v_3 +step +proj=unitconvert +xy_in=rad +xy_out=deg, preferred +proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.999601 +x_0=400000 +y_0=-100000 +ellps=airy +step +proj=hgridshift +grids=uk_os_OSTN15_NTv2_OSGBtoETRS.tif +step +proj=unitconvert +xy_in=rad +xy_out=deg.

----------------
Results
----------------

OUTPUT:	Sampled_2206186b_f5db_4b45_8ab5_f4b6ba03ef03
$ 
$ ~/git_repositories2/QGIS/build-8dc3558/output/bin/qgis_process --version
QGIS 3.35.0-Master 'Master' (8dc35588)
QGIS code revision 8dc35588
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.8.3
PROJ version 9.3.1
EPSG Registry database version v10.098 (2023-11-24)
GEOS version 3.12.1-CAPI-1.18.1
SQLite version 3.37.2
OS Linux Mint 21.3

Use the safer method of explicitly including the provider type when storing layer sources to string in more places, which eg fixes running the copied qgis_process string when a non gdal layer source is used for a raster layer. Also fixes the same situation when restoring an algorithm from the history log. Fixes qgis#54264 (cherry picked from commit c3fa3af)

Use the safer method of explicitly including the provider type when storing layer sources to string in more places, which eg fixes running the copied qgis_process string when a non gdal layer source is used for a raster layer. Also fixes the same situation when restoring an algorithm from the history log. Fixes #54264 (cherry picked from commit c3fa3af)

p-sims · 2024-06-18T21:16:05Z

Same issue here with:

qgis_process.bin -v
QGIS 3.28.11-Firenze 'Firenze' (exported)
QGIS code branchRelease 3.28
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.6.4
PROJ version 9.1.1
EPSG Registry database version v10.076 (2022-08-31)
GEOS version 3.11.1-CAPI-1.17.1
SQLite version 3.37.2
OS Pop!_OS 22.04 LTS

trying to run the following which I copied out of QGIS GUI using advanced menu:

qgis_process.bin run native:intersection --distance_units=meters --area_units=m2 --ellipsoid=EPSG:7019 --INPUT='dbname='\''qgis'\'' host=/var/run/postgresql port=5432 user='\''peter'\'' sslmode=disable key='\''fid'\'' srid=3035 type=MultiPolygon checkPrimaryKeyUnicity='\''1'\'' table="public"."clc18_by_la" (geom)' --OVERLAY='dbname='\''qgis'\'' host=/var/run/postgresql port=5432 user='\''peter'\'' sslmode=disable key='\''objectid'\'' srid=3035 type=MultiPolygon checkPrimaryKeyUnicity='\''1'\'' table="public"."agricultural_land_classification_provisional_england" (shape)' --OVERLAY_FIELDS_PREFIX= --OUTPUT='ogr:dbname='\''/home/peter/cloud/Greenhouse/_2024_Drax-Climate-Jobs/Data/Shapefiles+Maps/QGIS/CLC18-by-LA-by-AgLandGrades.gpkg'\'' table="CLC2018_v2020u1+AgLandGrade_jun24+LAD" (geom)'

Note: I had to add the .bin as other wise I get 'Command --noversioncheck not known!
' error....

Result is:

----------------
Inputs
----------------

INPUT:	dbname='qgis' host=/var/run/postgresql port=5432 user='peter' sslmode=disable key='fid' srid=3035 type=MultiPolygon checkPrimaryKeyUnicity='1' table="public"."clc18_by_la" (geom)
OUTPUT:	ogr:dbname='/home/peter/cloud/Greenhouse/_2024_Drax-Climate-Jobs/Data/Shapefiles+Maps/QGIS/CLC18-by-LA-by-AgLandGrades.gpkg' table="CLC2018_v2020u1+AgLandGrade_jun24+LAD" (geom)
OVERLAY:	dbname='qgis' host=/var/run/postgresql port=5432 user='peter' sslmode=disable key='objectid' srid=3035 type=MultiPolygon checkPrimaryKeyUnicity='1' table="public"."agricultural_land_classification_provisional_england" (shape)
OVERLAY_FIELDS_PREFIX:	

Using ellipsoid:	EPSG:7019
Using distance unit:	meters
Using area unit:	square meters
ERROR:	An error was encountered while checking parameter values
	Could not load source layer for INPUT: dbname='qgis' host=/var/run/postgresql port=5432 user='peter' sslmode=disable key='fid' srid=3035 type=MultiPolygon checkPrimaryKeyUnicity='1' table="public"."clc18_by_la" (geom) not found

I will try QGIS 3.35 at somepoint.

florisvdh added the Bug Either a bug report, or a bug fix. Let's hope for the latter! label Aug 16, 2023

agiudiceandrea added the Processing Relating to QGIS Processing framework or individual Processing algorithms label Aug 24, 2023

nirvn added the Feedback Waiting on the submitter for answers label Oct 26, 2023

nyalldawson removed the Feedback Waiting on the submitter for answers label Oct 26, 2023

florisvdh mentioned this issue Dec 20, 2023

Problem extracting output after using native:zonalstatistics algorithm r-spatial/qgisprocess#193

Closed

2 tasks

nyalldawson mentioned this issue Jan 10, 2024

Processing: Loosen check for uris by not checking whether they actually exist as files #55804

Merged

nyalldawson closed this as completed in #55804 Jan 13, 2024

nyalldawson reopened this Feb 4, 2024

nyalldawson mentioned this issue Feb 4, 2024

[processing] Use safer provider://source syntax in more places #56175

Merged

nyalldawson closed this as completed in #56175 Feb 5, 2024

nyalldawson mentioned this issue Feb 7, 2024

[processing] Use safer provider://source syntax in more places (3.34 backport) #56220

Merged

florisvdh mentioned this issue Feb 15, 2024

qgis_process dev+release+LTR abort on Ubuntu with ubuntugis-unstable PPA #56285

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

qgis_process doesn't seem to handle remote layers #54264

qgis_process doesn't seem to handle remote layers #54264

florisvdh commented Aug 16, 2023 •

edited

Loading

florisvdh commented Sep 12, 2023

nirvn commented Oct 26, 2023

nyalldawson commented Oct 26, 2023

nirvn commented Oct 26, 2023

jdlom commented Dec 13, 2023

florisvdh commented Jan 29, 2024

florisvdh commented Jan 29, 2024

florisvdh commented Jan 29, 2024

florisvdh commented Feb 6, 2024

p-sims commented Jun 18, 2024 •

edited

Loading

qgis_process doesn't seem to handle remote layers #54264

qgis_process doesn't seem to handle remote layers #54264

Comments

florisvdh commented Aug 16, 2023 • edited Loading

What is the bug or the crash?

Steps to reproduce the issue

Versions

Supported QGIS version

New profile

Additional context

florisvdh commented Sep 12, 2023

nirvn commented Oct 26, 2023

nyalldawson commented Oct 26, 2023

nirvn commented Oct 26, 2023

jdlom commented Dec 13, 2023

florisvdh commented Jan 29, 2024

florisvdh commented Jan 29, 2024

florisvdh commented Jan 29, 2024

florisvdh commented Feb 6, 2024

p-sims commented Jun 18, 2024 • edited Loading

florisvdh commented Aug 16, 2023 •

edited

Loading

p-sims commented Jun 18, 2024 •

edited

Loading