Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] master from OSGeo:master #108

Merged
merged 2 commits into from
Dec 5, 2024
Merged

[pull] master from OSGeo:master #108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
01f5361
VRTProcessedDataset: allow to define a OutputBands element
rouault Dec 3, 2024
98d0625
Merge pull request #11431 from rouault/fix_11430
rouault Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_FROM_LAST_STEP.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<OutputBands count="FROM_LAST_STEP" dataType="FROM_LAST_STEP"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
<Argument name="coefficients_2">20,1</Argument>
</Step>
</ProcessingSteps>
</VRTDataset>
31 changes: 31 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_FROM_LAST_STEP_with_stats.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<OutputBands count="FROM_LAST_STEP" dataType="FROM_LAST_STEP"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
<Argument name="coefficients_2">20,1</Argument>
</Step>
</ProcessingSteps>
<VRTRasterBand dataType="Float64" band="1" subClass="VRTProcessedRasterBand">
<Metadata>
<MDI key="STATISTICS_MINIMUM">84</MDI>
<MDI key="STATISTICS_MAXIMUM">265</MDI>
<MDI key="STATISTICS_MEAN">136.765</MDI>
<MDI key="STATISTICS_STDDEV">22.928470838676</MDI>
<MDI key="STATISTICS_VALID_PERCENT">100</MDI>
</Metadata>
</VRTRasterBand>
<VRTRasterBand dataType="Float64" band="2" subClass="VRTProcessedRasterBand">
<Metadata>
<MDI key="STATISTICS_MINIMUM">94</MDI>
<MDI key="STATISTICS_MAXIMUM">275</MDI>
<MDI key="STATISTICS_MEAN">146.765</MDI>
<MDI key="STATISTICS_STDDEV">22.928470838676</MDI>
<MDI key="STATISTICS_VALID_PERCENT">100</MDI>
</Metadata>
</VRTRasterBand>
</VRTDataset>
31 changes: 31 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_FROM_LAST_STEP_with_stats_error.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<OutputBands count="FROM_LAST_STEP" dataType="FROM_LAST_STEP"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
<Argument name="coefficients_3">20,1</Argument>
</Step>
</ProcessingSteps>
<VRTRasterBand dataType="Float64" band="1" subClass="VRTProcessedRasterBand">
<Metadata>
<MDI key="STATISTICS_MINIMUM">84</MDI>
<MDI key="STATISTICS_MAXIMUM">265</MDI>
<MDI key="STATISTICS_MEAN">136.765</MDI>
<MDI key="STATISTICS_STDDEV">22.928470838676</MDI>
<MDI key="STATISTICS_VALID_PERCENT">100</MDI>
</Metadata>
</VRTRasterBand>
<VRTRasterBand dataType="Float64" band="2" subClass="VRTProcessedRasterBand">
<Metadata>
<MDI key="STATISTICS_MINIMUM">94</MDI>
<MDI key="STATISTICS_MAXIMUM">275</MDI>
<MDI key="STATISTICS_MEAN">146.765</MDI>
<MDI key="STATISTICS_STDDEV">22.928470838676</MDI>
<MDI key="STATISTICS_VALID_PERCENT">100</MDI>
</Metadata>
</VRTRasterBand>
</VRTDataset>
23 changes: 23 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_FROM_LAST_STEP_with_stats_missing_band.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<OutputBands count="FROM_LAST_STEP" dataType="FROM_LAST_STEP"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
<Argument name="coefficients_2">20,1</Argument>
</Step>
</ProcessingSteps>
<VRTRasterBand dataType="Float64" band="1" subClass="VRTProcessedRasterBand">
<Metadata>
<MDI key="STATISTICS_MINIMUM">84</MDI>
<MDI key="STATISTICS_MAXIMUM">265</MDI>
<MDI key="STATISTICS_MEAN">136.765</MDI>
<MDI key="STATISTICS_STDDEV">22.928470838676</MDI>
<MDI key="STATISTICS_VALID_PERCENT">100</MDI>
</Metadata>
</VRTRasterBand>
<!-- missing 2nd VRTRasterBand -->
</VRTDataset>
13 changes: 13 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_FROM_SOURCE.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<!-- default -->
<OutputBands count="FROM_SOURCE" dataType="FROM_SOURCE"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
</Step>
</ProcessingSteps>
</VRTDataset>
14 changes: 14 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_FROM_SOURCE_wrong_band_count.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<!-- default -->
<OutputBands count="FROM_SOURCE" dataType="FROM_SOURCE"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
<Argument name="coefficients_2">20,1</Argument>
</Step>
</ProcessingSteps>
</VRTDataset>
12 changes: 12 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_USER_PROVIDED.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<OutputBands count="1" dataType="Float32"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
</Step>
</ProcessingSteps>
</VRTDataset>
12 changes: 12 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_USER_PROVIDED_invalid_type.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<OutputBands count="1" dataType="invalid"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
</Step>
</ProcessingSteps>
</VRTDataset>
12 changes: 12 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_USER_PROVIDED_non_numeric_count.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<OutputBands count="invalid" dataType="Float32"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
</Step>
</ProcessingSteps>
</VRTDataset>
12 changes: 12 additions & 0 deletions autotest/gdrivers/data/vrt/processed_OutputBands_USER_PROVIDED_too_large_count.vrt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">../byte.tif</SourceFilename>
</Input>
<OutputBands count="100000000" dataType="Float32"/>
<ProcessingSteps>
<Step name="Affine combination of band values">
<Algorithm>BandAffineCombination</Algorithm>
<Argument name="coefficients_1">10,1</Argument>
</Step>
</ProcessingSteps>
</VRTDataset>
84 changes: 84 additions & 0 deletions autotest/gdrivers/vrtprocesseddataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1239,3 +1239,87 @@ def test_vrtprocesseddataset_serialize(tmp_vsimem):
11.5,
0.5,
]


###############################################################################
# Test OutputBands


def test_vrtprocesseddataset_OutputBands():

with gdal.Open("data/vrt/processed_OutputBands_FROM_LAST_STEP.vrt") as ds:
assert ds.RasterCount == 2
assert (ds.GetRasterBand(1).GetMinimum(), ds.GetRasterBand(1).GetMaximum()) == (
None,
None,
)
assert (ds.GetRasterBand(2).GetMinimum(), ds.GetRasterBand(2).GetMaximum()) == (
None,
None,
)
assert ds.GetRasterBand(1).ComputeRasterMinMax() == (84, 265)
assert ds.GetRasterBand(2).ComputeRasterMinMax() == (94, 275)

with gdal.Open(
"data/vrt/processed_OutputBands_FROM_LAST_STEP_with_stats.vrt"
) as ds:
assert ds.RasterCount == 2
assert (ds.GetRasterBand(1).GetMinimum(), ds.GetRasterBand(1).GetMaximum()) == (
84,
265,
)
assert (ds.GetRasterBand(2).GetMinimum(), ds.GetRasterBand(2).GetMaximum()) == (
94,
275,
)

with gdal.Open(
"data/vrt/processed_OutputBands_FROM_LAST_STEP_with_stats_missing_band.vrt"
) as ds:
assert ds.RasterCount == 2
assert (ds.GetRasterBand(1).GetMinimum(), ds.GetRasterBand(1).GetMaximum()) == (
None,
None,
)
assert (ds.GetRasterBand(2).GetMinimum(), ds.GetRasterBand(2).GetMaximum()) == (
None,
None,
)
assert ds.GetRasterBand(1).ComputeRasterMinMax() == (84, 265)
assert ds.GetRasterBand(2).ComputeRasterMinMax() == (94, 275)

with pytest.raises(Exception, match="Argument coefficients_2 is missing"):
gdal.Open("data/vrt/processed_OutputBands_FROM_LAST_STEP_with_stats_error.vrt")

with gdal.Open("data/vrt/processed_OutputBands_FROM_SOURCE.vrt") as ds:
assert ds.RasterCount == 1
assert ds.GetRasterBand(1).ComputeRasterMinMax() == (84, 255)

with pytest.raises(
Exception,
match="Final step expect 1 bands, but only 2 coefficient_XX are provided",
):
gdal.Open("data/vrt/processed_OutputBands_FROM_SOURCE_wrong_band_count.vrt")

with gdal.Open("data/vrt/processed_OutputBands_USER_PROVIDED.vrt") as ds:
assert ds.RasterCount == 1
assert ds.GetRasterBand(1).DataType == gdal.GDT_Float32
assert ds.GetRasterBand(1).ComputeRasterMinMax() == (84, 265)

with pytest.raises(
Exception,
match="Invalid band count",
):
gdal.Open("data/vrt/processed_OutputBands_USER_PROVIDED_too_large_count.vrt")

with pytest.raises(
Exception,
match="Invalid value for OutputBands.count",
):
gdal.Open("data/vrt/processed_OutputBands_USER_PROVIDED_non_numeric_count.vrt")

with pytest.raises(
Exception,
match="Invalid value for OutputBands.dataType",
):
gdal.Open("data/vrt/processed_OutputBands_USER_PROVIDED_invalid_type.vrt")
2 changes: 1 addition & 1 deletion doc/source/drivers/raster/vrt.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ VRTRasterBand
The attributes for VRTRasterBand are:

- **dataType** (optional): type of the pixel data associated with this band (use
names Byte, UInt16, Int16, UInt32, Int32, Float32, Float64, CInt16, CInt32, CFloat32 or CFloat64).
names Byte, Int8, UInt16, Int16, UInt32, Int32, UInt64, Int64, Float32, Float64, CInt16, CInt32, CFloat32 or CFloat64).
If not specified, defaults to 1

- **band** (optional): band number this element represents (1 based).
Expand Down
46 changes: 43 additions & 3 deletions doc/source/drivers/raster/vrt_processed_dataset.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,7 @@ The ``VRTDataset`` root element must have a ``subClass="VRTProcessedDataset"`` a

The following child elements of ``VRTDataset`` may be defined: ``SRS``, ``GeoTransform``, ``Metadata``. If they are not explicitly set, they are inferred from the input dataset.

``VRTRasterBand`` elements may be explicitly defined, in particular if the data type of the virtual dataset after all processing steps is different from the input one, or if the number of output bands is different from the number of input bands. If there is no explicit ``VRTRasterBand`` element, the number and data types of input bands are used implicitly. When explicitly defined, ``VRTRasterBand`` elements must have a ``subClass="VRTProcessedRasterBand"`` attribute.
`
It must also have the 2 following child elements:
The ``VRTDataset`` root element must also have the 2 following child elements:

- ``Input``, which must have one and only one of the following ``SourceFilename`` or ``VRTDataset`` as child elements, to define the input dataset to which to apply the processing steps.

Expand All @@ -132,6 +130,48 @@ The value of ``Algorithm`` must be a registered VRTProcessedDataset function. At

A ``Step`` will generally have one or several ``Argument`` child elements, some of them being required, others optional. Consult the documentation of each algorithm.

Starting with GDAL 3.11, a ``OutputBands`` element can be
defined as a child element of ``VRTDataset``, with the following 2 attributes:

* ``count`` whose value can be ``FROM_SOURCE`` to indicate that the output band
count must be the same as the number of bands of the input dataset,
``FROM_LAST_STEP`` to indicate that it must be the number of output bands
returned by the initialization function of the last step, or an integer value.

* ``dataType`` whose value can be ``FROM_SOURCE`` to indicate that the output band
data type must be the same as one of the input dataset,
``FROM_LAST_STEP`` to indicate that it must be the one returned by the
initialization function of the last step, or a value among
Byte, Int8, UInt16, Int16, UInt32, Int32, UInt64, Int64, Float32, Float64, CInt16, CInt32, CFloat32 or CFloat64

Example:

.. code-block:: xml

<VRTDataset subClass="VRTProcessedDataset">
<Input>
<SourceFilename relativeToVRT="1">source.tif</SourceFilename>
</Input>
<OutputBands count="FROM_LAST_STEP" dataType="FROM_LAST_STEP"/>
<ProcessingSteps>...</ProcessingSteps>
</VRTDataset>


If ``OutputBands`` is omitted,

* if there are explicit ``VRTRasterBand`` elements, they must have a
``subClass="VRTProcessedRasterBand"`` attribute

* it there are no explicit ``VRTRasterBand`` elements, the number and data types
of input bands are used implicitly.

Both ``OutputBands`` and ``VRTRasterBand`` elements may be defined. The information
specified by ``OutputBands`` will be used in priority, and ``VRTRasterBand`` elements
will be used only if they are compatible with the band count and data type specified
through ``OutputBands``. A situation where ``OutputBands`` and ``VRTRasterBand`` elements
are both found is for example when computing statistics on a .vrt file with only
``OutputBands`` initially set.

LocalScaleOffset algorithm
--------------------------

Expand Down
25 changes: 25 additions & 0 deletions frmts/vrt/data/gdalvrt.xsd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@
<xs:documentation>Allowed only if subClass="VRTProcessedDataset"</xs:documentation>
</xs:annotation>
</xs:element>
<xs:element name="OutputBands" type="OutputBandsType">
<xs:annotation>
<xs:documentation>Allowed only if subClass="VRTProcessedDataset"</xs:documentation>
</xs:annotation>
</xs:element>
<xs:element name="ProcessingSteps" type="ProcessingStepsType">
<xs:annotation>
<xs:documentation>Allowed only if subClass="VRTProcessedDataset"</xs:documentation>
Expand Down Expand Up @@ -195,6 +200,26 @@
</xs:sequence>
</xs:complexType>

<xs:complexType name="OutputBandsType">
<xs:sequence/>
<xs:attribute name="count" type="xs:string">
<xs:annotation>
<xs:documentation>FROM_SOURCE, FROM_LAST_STEP or positive integer value.
If not specified, FROM_SOURCE is the default if there are no VRTRasterBand elements.
Otherwise, if there are VRTRasterBand elements, they are used.
</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="dataType" type="xs:string">
<xs:annotation>
<xs:documentation>FROM_SOURCE, FROM_LAST_STEP or one of the allowed values of DataTypeType.
If not specified, FROM_SOURCE is the default if there are no VRTRasterBand elements.
Otherwise, if there are VRTRasterBand elements, they are used.
</xs:documentation>
</xs:annotation>
</xs:attribute>
</xs:complexType>

<xs:complexType name="ProcessingStepsType">
<xs:sequence minOccurs="1" maxOccurs="unbounded">
<xs:element name="Step" type="ProcessingStepType"/>
Expand Down
Loading
Loading