How do we want to handle test data?

[mathematicalmichael]

• minio instance (how will github actions interact with it? secrets in github?)
• public server (apache, nginx) with files in it (zip file) kept somewhere...

workflow can be "manage data" with minio and "access data" through public server OR s3.

this way credentialed users can modify data but anyone can access it.

• question: what permissions will minio assign the files? ownership? do we need to run a cronjob to add chmod +r to files uploaded with minio?

[mathematicalmichael]

minio + apache deployment together, put in services folder in root of this repo, validate functionality

• test everything below with docker volumes AND local volumes
• create file on disk
• upload file w awscli
• access it from apache and have file be visible

[mathematicalmichael]

once files are available... how will the code itself go download them? tflow, torch, etc all do some version of this.

[mathematicalmichael]

right now we just pull these in with the makefile from a storage server.

update: now using cloudflare to proxy that server, need to verify things still work, as well as test s3/minio access.