Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add zstd compression #451

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

feat: add zstd compression #451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6e65842
feat: add zstd compression
Rdataflow Jan 8, 2025
d497708
ci: add tests for zstd
Rdataflow Jan 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/compression/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,11 @@ addDecoder(34887, () => import('./lerc.js')
})
.then((m) => m.default),
);
addDecoder(50000, () => import('./zstd.js')
.then(async (m) => {
await m.zstd.init();
return m;
})
.then((m) => m.default),
);
addDecoder(50001, () => import('./webimage.js').then((m) => m.default));
10 changes: 10 additions & 0 deletions src/compression/zstd.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import { ZSTDDecoder } from 'zstddec';
import BaseDecoder from './basedecoder.js';

export const zstd = new ZSTDDecoder();

export default class ZstdDecoder extends BaseDecoder {
decodeBlock(buffer) {
return zstd.decode(new Uint8Array(buffer), 1000_000_000).buffer;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second parameter is uncompressedSize, which is fixed here to 1 million. Is this significant? can we actually know it beforehand? What if the size is actually smaller/larger?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the source: if uninitialized (i.e: 0) the size should automatically be found out. Wouldn't that be a better solution?

I'm worried, that if the actual size is larger than the 1 million bytes buffer (which can easily be reached by 1024x1024 tile sizes) we might run into troubles.

Copy link
Author

@Rdataflow Rdataflow Jan 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@constantinius this value of 1 billion has been approximated using trial and error.
AFAICS 0 or omitting the param fails in the CI tests for some unknown reason.
however huge numbers make it pass w/o issues.
I couldn't observe any negative impact neither with 100 billion. (i.e. no mem limits hit)
for the full journey see this branch https://github.com/Rdataflow/geotiff.js/commits/ci-test/ and it's CI tests failing or passing
that's what drove us here... but maybe there exist even better ideas?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Rdataflow

I investigated this. There is a function in zstd to determine the uncompressed size of a chunk. This is a property set in the chunk itself which is optional. When it is optional, zstd says to read the chunk in streaming mode. Otherwise proper decompression is not guaranteed. And this is something I'd like to avoid, since it may work on our test files, but not on random files used by people. So I'm against abusing the zstd library that way. Unfortunately, the zstd library we are currently using (and no other one I investigated) supports streaming decompression as it does not wrap the necessary functions of the wasm library.

I'm now trying to implement this streaming decoding into the zstd library. I'm thinking of incorporating that into the source code of geotiff.js directly, as it may be easier to handle.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @constantinius

Great you'll an even better way to implement zstd decompression 😃

just in case that becomes too heavy, I'll share my thoughts on alternative ways (up to you to decide and implement properly):

  • precalculate the uncompressed buffer size using the geotiff properties (tile size, bit depth, etc.)

}
}
2 changes: 2 additions & 0 deletions test/data/setup_data.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,12 @@ gdal_translate -of GTiff -co COMPRESS=LERC -co MAX_Z_ERROR=1000 stripped.tiff le
gdal_translate -of GTiff -co COMPRESS=LERC -co MAX_Z_ERROR=1000 -co INTERLEAVE=BAND stripped.tiff lerc_interleave.tiff
gdal_translate -of GTiff -co COMPRESS=LERC_DEFLATE -co MAX_Z_ERROR=1000 stripped.tiff lerc_deflate.tiff
gdal_translate -of GTiff -co COMPRESS=LERC_ZSTD -co MAX_Z_ERROR=1000 stripped.tiff lerc_zstd.tiff
gdal_translate -of GTiff -co COMPRESS=ZSTD stripped.tiff zstd.tiff
gdal_translate -of GTiff -ot Float32 -co COMPRESS=LERC -co MAX_Z_ERROR=1000 stripped.tiff float32lerc.tiff
gdal_translate -of GTiff -ot Float32 -co COMPRESS=LERC -co MAX_Z_ERROR=1000 -co INTERLEAVE=BAND stripped.tiff float32lerc_interleave.tiff
gdal_translate -of GTiff -ot Float32 -co COMPRESS=LERC_DEFLATE -co MAX_Z_ERROR=1000 stripped.tiff float32lerc_deflate.tiff
gdal_translate -of GTiff -ot Float32 -co COMPRESS=LERC_ZSTD -co MAX_Z_ERROR=1000 stripped.tiff float32lerc_zstd.tiff
gdal_translate -of GTiff -ot Float32 -co COMPRESS=ZSTD stripped.tiff float32zstd.tiff

gdal_translate -of COG initial.tiff cog.tiff

Expand Down
10 changes: 10 additions & 0 deletions test/geotiff.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,11 @@ describe('GeoTIFF', () => {
await performTiffTests(tiff, 539, 448, 15, Uint16Array);
});

it('should work on Zstandard compressed tiffs', async () => {
const tiff = await GeoTIFF.fromSource(createSource('zstd.tiff'));
await performTiffTests(tiff, 539, 448, 15, Uint16Array);
});

it('should work on Float32 and LERC compressed tiffs', async () => {
const tiff = await GeoTIFF.fromSource(createSource('float32lerc.tiff'));
await performTiffTests(tiff, 539, 448, 15, Float32Array);
Expand All @@ -268,6 +273,11 @@ describe('GeoTIFF', () => {
await performTiffTests(tiff, 539, 448, 15, Float32Array);
});

it('should work on Float32 and Zstandard compressed tiffs', async () => {
const tiff = await GeoTIFF.fromSource(createSource('float32zstd.tiff'));
await performTiffTests(tiff, 539, 448, 15, Float32Array);
});

it('should work with worker pool', async () => {
const testPool = new Pool();
const tiff = await GeoTIFF.fromSource(createSource('nasa_raster.tiff'));
Expand Down
Loading