Reading data from cloud storage

[St3V0Bay]

Dear MONAI Community,

a question regarding the input for ImageReader (and more specifically WSIReader):

is possible to also use paths from some sort of cloud storage (e.g. S3 or GCS)?

Given the size of WSIs it would be so nice to not copy the data back and forth anymore. In case it's not supported, do you see a chance that this is technically feasible without running into some I/O bottleneck?

Curious about your thoughts.

Thanks in advance.

[bhashemian]

Hi @St3V0Bay,

Sorry for the long delay in responding to this question. I just saw this. For the future, please use MONAI repo to ask your questions about monai features and to submit a feature request.

I agree that reading from a network/cloud storage is more convenient and sometimes for efficient (e.g. for inference); however, its implementation is not trivial as each cloud vendor has their own API. Also it may create a bottleneck for training.

Here is nice work on comparing reading common data formats from local, http and s3 storage. You can see that for TIFF file for instance there is an order of magnitude slow down when loading from S3 instead of local.
https://www.nature.com/articles/s41592-021-01326-w