XST can not download large document/collection sets

[dizzzz]

What happened?

When downloading a collection with a lot of resources (documents, collections recursively), xst stops after 3000+ files.

eg

Γ£ÿ AA.xml could not be created! Reason: connect ENOBUFS 1.2.3.4:8080 - Local (undefined:undefined)
Γ£ÿ BB.xml could not be created! Reason: connect ENOBUFS 1.2.3.4:8080 - Local (undefined:undefined)
Γ£ÿ CC.xml could not be created! Reason: connect ENOBUFS 1.2.3.4:8080 - Local (undefined:undefined)

the process stops with

Γ£ÿ c5b327080fd4d751bbc4d19160dc8a49.xsd could not be created! Reason: aborted
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

Error: aborted
    at Socket.socketCloseListener (node:_http_client:460:19)
    at Socket.emit (node:events:531:35)
    at TCP.<anonymous> (node:net:338:12) {
  code: 'ECONNRESET'
}

Node.js v20.14.0

System Information

Build: eXist-6.2.0 (c8fa4958b6d4a50bd0cba7f3e76a150226414187)
Java: 1.8.0_275 (Red Hat, Inc.)
OS: Linux 2.6.32-754.50.1.el6.x86_64 (amd64)

Relevant shell output

Γ£ÿ c5b327080fd4d751bbc4d19160dc8a49.xsd could not be created! Reason: aborted
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

Error: aborted
    at Socket.socketCloseListener (node:_http_client:460:19)
    at Socket.emit (node:events:531:35)
    at TCP.<anonymous> (node:net:338:12) {
  code: 'ECONNRESET'

[line-o]

Thanks for bringing this up @dizzzz . While xst upload already features a queue to not overload a connected database with jobs, this is not yet the case for the download command.

So I suspect that exist-db flat out refuses any more connections after several thousand are already opened. This is what happened on uploads, too.