Out-of-Core Processing

[mullinsms]

Out-of-core processing has started to arrive upstream and I'm wondering when it might start to show up in the wasm version of duckdb? My understanding is that FileSystemSyncAccessHandle was made fully synchronous to support just these kinds of use cases. We're particularly interested in being able to spill query results that exceed the wasm memory limits.

[tc-cole]

I have been looking into using SQLITE Wasm with OPFS with Duckdb to circumnavigate this issue.

[CMCDragonkai]

Would this make the browser a native platform for analytics on HUGE data that can only exist on disk and requires Mapped IO to be practical? If so, awesome!

[carlopi]

Update: capabilities for Out-Of-Core processing can be turned on setting a temporary directory (like: `SET temp_directory = 'tmp'), but then currently most use cases will experience problems connected to the filesystem (tracked here: #1576).
Once those are fixed, out-of-core processing for duckdb-wasm should work.

I will update when those problem are sorted out.

[CMCDragonkai]

Why is temp_directory relevant? Are there other directories?

[attila-kun]

@CMCDragonkai I'm guessing that's where DuckDB spills those results which would go above the memory limit.