are aws s3 resources downloadable by all?

[kelly-sovacool]

aws s3 cp s3://nciccbr/Resources/RNA-seq/multiqc_config.yaml .
fatal error: Unable to locate credentials

[kopardev]

Using curl

curl -I https://nciccbr.s3.us-east-1.amazonaws.com/Resources/RNA-seq/multiqc_config.yaml
HTTP/1.1 200 OK
x-amz-id-2: yHjNua1ycT/6dOJ8dMNpbzV9cV4cBdVm7d1ntkIPpiT+2IlKPI5/XmELb4GXH5pKdt9UpHM3IIk=
x-amz-request-id: A20RSWRT4JN5ZTGZ
Date: Wed, 04 Dec 2024 02:20:15 GMT
Last-Modified: Fri, 26 Mar 2021 22:29:52 GMT
ETag: "a45a57e549eb5cce74d1e6e27fbff730"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 1885
Server: AmazonS3

Using wget

wget https://nciccbr.s3.us-east-1.amazonaws.com/Resources/RNA-seq/multiqc_config.yaml
--2024-12-04 02:20:43--  https://nciccbr.s3.us-east-1.amazonaws.com/Resources/RNA-seq/multiqc_config.yaml
Resolving nciccbr.s3.us-east-1.amazonaws.com (nciccbr.s3.us-east-1.amazonaws.com)... 52.216.50.10, 52.217.80.104, 52.217.232.2, ...
Connecting to nciccbr.s3.us-east-1.amazonaws.com (nciccbr.s3.us-east-1.amazonaws.com)|52.216.50.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1885 (1.8K) [binary/octet-stream]
Saving to: ‘multiqc_config.yaml’

multiqc_config.yaml                100%[=============================================================>]   1.84K  --.-KB/s    in 0s      

2024-12-04 02:20:43 (61.4 MB/s) - ‘multiqc_config.yaml’ saved [1885/1885]

suggests that these files are freely accessible from the internet.

[kopardev]

Tried uploading a new file with specific acl options to make it public readable

aws s3 cp test_public_file.txt s3://nciccbr/Resources/RNA-seq/test_public_file.txt --acl public-read

The acl looks like this

aws s3api get-object-acl --bucket nciccbr --key Resources/RNA-seq/test_public_file.txt
{
    "Owner": {
        "DisplayName": "margaret.cam",
        "ID": "aa52aea548c28ebef8cfc7d00912b682d9bf3b44539f571c9a7694fe34d43e96"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "margaret.cam",
                "ID": "aa52aea548c28ebef8cfc7d00912b682d9bf3b44539f571c9a7694fe34d43e96",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        },
        {
            "Grantee": {
                "Type": "Group",
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
        }
    ]
}

while that of an older file looks like this

aws s3api get-object-acl --bucket nciccbr --key Resources/RNA-seq/multiqc_config.yaml
{
    "Owner": {
        "DisplayName": "margaret.cam",
        "ID": "aa52aea548c28ebef8cfc7d00912b682d9bf3b44539f571c9a7694fe34d43e96"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "margaret.cam",
                "ID": "aa52aea548c28ebef8cfc7d00912b682d9bf3b44539f571c9a7694fe34d43e96",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]
}

But both files give the same error

aws s3 cp s3://nciccbr/Resources/RNA-seq/multiqc_config.yaml .
fatal error: Unable to locate credentials

aws s3 cp s3://nciccbr/Resources/RNA-seq/test_public_file.txt .
fatal error: Unable to locate credentials

[kopardev]

The entire buckets policy looks like this:

aws s3api get-bucket-policy --bucket nciccbr | jq -r '.Policy' | jq .
{
  "Version": "2012-10-17",
  "Id": "Policy1604978077308",
  "Statement": [
    {
      "Sid": "Stmt1604978061256",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::nciccbr/Resources/*"
    }
  ]
}

• "Principal": "*": Allows access to anyone.
• "Action": "s3:GetObject": Grants permission to download files.
• "Resource": "arn:aws:s3:::nciccbr/Resources/*": Applies the policy to all objects in the Resources folder of the nciccbr bucket.
• So all files in Resources should be accessible to all.

[kopardev]

@kelly-sovacool I was able to download a file with --no-sign-request argument:

aws s3 cp s3://nciccbr/Resources/RNA-seq/arriba/blacklist_hg38_GRCh38_v2.0.0.tsv.gz . --no-sign-request
download: s3://nciccbr/Resources/RNA-seq/arriba/blacklist_hg38_GRCh38_v2.0.0.tsv.gz to ./blacklist_hg38_GRCh38_v2.0.0.tsv.gz

Can you try?

[kelly-sovacool]

I was able to download a file with --no-sign-request argument:

aws s3 cp s3://nciccbr/Resources/RNA-seq/arriba/blacklist_hg38_GRCh38_v2.0.0.tsv.gz . --no-sign-request
download: s3://nciccbr/Resources/RNA-seq/arriba/blacklist_hg38_GRCh38_v2.0.0.tsv.gz to ./blacklist_hg38_GRCh38_v2.0.0.tsv.gz

Can you try?

This worked for me!

[kelly-sovacool]

@kopardev I think we can close this issue confident that our resources are accessible, since when using the aws s3 CLI we can use --no-sign-request and within snakemake workflows it works regardless.

[kopardev]

aws s3api get-object-acl --bucket nciccbr --key

acl read can be added like this

aws s3api put-object-acl --bucket nciccbr --key Resources/CHARLIE/fasta_gtf.tar --acl public-read