Trying to migrate from aspera ats download to an ascli command

[darcyrailip]

Per IBM's request I am trying to get off of an older version of Aspera and onto the ascli version offered here.

Note: we have a primary server that generates Aspera access keys and secrets for our server-initiated transfers (CentOS 7 on its way to AlmaLinux), and generates transfer specs for our Desktop-initiated Aspera Connect transfers (typically Mac or Windows users).

In Linux, we can run a command similar to:

"/opt/.aspera/cli/bin/aspera" ats download -H http://ats-aws-us-east-1.aspera.io -u someusertoken -p somepasswordtoken -s "AFileOrDirectoryInAmazonS3" -d /mnt/library/aspera-downloads

I'm trying to find the equivalent with ascli.

Even better, I'd love to find out how to use the transfer_spec with downloading multiple files from S3 to local directories, and building out the different mapping from remote-to-local:

{
  "paths": [
      {
          "source": "projects\/5b468336ac9f2510ad311c39\/source-files\/63922f5777488eb4ff0f0b96\/master\/63922f5777488eb4ff0f0b96.mov",
          "destination": "\/"
      }
  ],
  "source_root": "",
  "destination_root": "",
  "token": "ATB2_SOME_TOKEN",
  "direction": "receive",
  "target_rate_cap_kbps": 3000000,
  "cipher": "aes-128",
  "tags": {
    "aspera": {
          "node": {
              "access_key": "some-access-key"
          }
      },
  },
  "rate_policy_allowed": "fair",
  "rate_policy": "fair",
  "target_rate_kbps": 700000,
  "min_rate_cap_kbps": 0,
  "min_rate_kbps": 0,
  "lock_min_rate": true,
  "remote_host": "ats-aws-us-east-1.aspera.io",
  "remote_user": "xfer",
  "sshfp": null,
  "ssh_port": 33001,
  "fasp_port": 33001,
  "http_fallback": false
}

I feel like all the information must be somewhere in the README but I'll be honest I'm having trouble narrowing down all the different commands.

[laurent-martin]

For ATS use the node plugin

ascli node download --url=http://ats-aws-us-east-1.aspera.io --username=someusertoken --password=somepasswordtoken "AFileOrDirectoryInAmazonS3" --to-folder=/mnt/library/aspera-downloads

For transfers , make sure that the transfer sdk is installed: ascli conf ascp install

If you do several commands , it can be interesting to create a "preset" configuration:

ascli conf preset update myatsnode --url=http://ats-aws-us-east-1.aspera.io --username=someusertoken --password=somepasswordtoken

Then you can recall that config:

ascli node -Pmyatsnode download "AFileOrDirectoryInAmazonS3" --to-folder=/mnt/library/aspera-downloads

or you can even make this the default:

ascli conf preset set default node myatsnode

and then just do:

ascli node download "AFileOrDirectoryInAmazonS3" --to-folder=/mnt/library/aspera-downloads

note that the node plugin also provides browse and other operations.

yes, internally, ascli will build a "transfer spec", and use this for transfers: you can modify the transfer spec using option --ts= during a transfer operation, like download.

[darcyrailip]

Hi Laurent, thanks. I'm having a bit of trouble still, here is a command that works with aspera ats download:

/opt/.aspera/cli/bin/aspera ats download -H ats-aws-us-east-1.aspera.io -u usernametoken -p passwordtoken -s BigBuckBunny_SHR_F_EN-XX_CA_51_2K_20170711_BIT_IOP_OV -d /mnt/library/aspera-downloads --target-rate 500000
Setting up REST parameters... done
Building JSON... done
Serializing JSON... done
POSTting https://ats-aws-us-east-1.aspera.io:443/files/download_setup... done
Parsing response as JSON... done
Checking for errors in JSON... done
Setting up FASP parameters... done
Performing package download
^C% [>                                     ] 108MB / 9.3GB @ 453Mb/s    06:45 ETA

Notice it starts downloading and spitting out transfer status / ETA.

Here I have tried replicating with ascli node downloads:

ascli node download --url=http://ats-aws-us-east-1.aspera.io --username=usernametoken --password=passwordtoken "BigBuckBunny_SHR_F_EN-XX_CA_51_2K_20170711_BIT_IOP_OV" --to-folder=/mnt/library/aspera-downloads --log-level=debug
...
D, [2024-11-04T14:05:30.020065 #24253] DEBUG -- : paths=[{"source"=>"BigBuckBunny_SHR_F_EN-XX_CA_51_2K_20170711_BIT_IOP_OV"}]
D, [2024-11-04T14:05:30.020108 #24253] DEBUG -- : POST [files/download_setup]
D, [2024-11-04T14:05:30.020129 #24253] DEBUG -- : using Basic auth
D, [2024-11-04T14:05:30.020189 #24253] DEBUG -- : URI=http://ats-aws-us-east-1.aspera.io/files/download_setup
D, [2024-11-04T14:05:30.020300 #24253] DEBUG -- : req_body (json)=
"{\"transfer_requests\":[{\"transfer_request\":{\"paths\":[{\"source\":\"BigBuckBunny_SHR_F_EN-XX_CA_51_2K_20170711_BIT_IOP_OV\"}]}}]}"
D, [2024-11-04T14:05:30.020350 #24253] DEBUG -- : send request (retries=0)
ERROR: Other(Net::OpenTimeout): execution expired
Traceback (most recent call last):
	17: from /root/.rbenv/versions/2.6.0/bin/ascli:23:in `<main>'
	16: from /root/.rbenv/versions/2.6.0/bin/ascli:23:in `load'
	15: from /root/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/aspera-cli-4.19.0/bin/ascli:24:in `<top (required)>'
	14: from /root/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/aspera-cli-4.19.0/lib/aspera/cli/main.rb:300:in `process_command_line'
	13: from /root/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/aspera-cli-4.19.0/lib/aspera/cli/plugins/node.rb:735:in `execute_action'
	12: from /root/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/aspera-cli-4.19.0/lib/aspera/cli/plugins/node.rb:359:in `execute_simple_common'
	11: from /root/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/aspera-cli-4.19.0/lib/aspera/cli/plugins/node.rb:336:in `execute_command_gen3'
	10: from /root/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/aspera-cli-4.19.0/lib/aspera/rest.rb:427:in `create'
	 9: from /root/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/aspera-cli-4.19.0/lib/aspera/rest.rb:334:in `call'
	 8: from /root/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/aspera-cli-4.19.0/lib/aspera/rest.rb:175:in `http_session'
	 7: from /root/.rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/aspera-cli-4.19.0/lib/aspera/rest.rb:113:in `start_http_session'
	 6: from /root/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:925:in `start'
	 5: from /root/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
	 4: from /root/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:945:in `connect'
	 3: from /root/.rbenv/versions/2.6.0/lib/ruby/2.6.0/timeout.rb:103:in `timeout'
	 2: from /root/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:947:in `block in connect'
	 1: from /root/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:947:in `open'
/root/.rbenv/versions/2.6.0/lib/ruby/2.6.0/net/http.rb:947:in `initialize': execution expired (Net::OpenTimeout)

Note, I have removed a few DEBUG logs from in between the command and the error.

Also to be clear, the username and password that exist above are created against the Aspera Access Keys URL: https://ats.aspera.io/pub/v1/access_keys

They are created with a path parameter that locks the access to that path in S3 specifically. I'm wondering if ascli node download is expecting different credentials?

[darcyrailip]

I'm also trying to simply use an existing transfer spec that works with Aspera Connect (Desktop):

ascli node download --ts=@json:'{"paths": [{"source": "projects\/5b468336ac9f2510ad311c39\/source-files\/63922f5777488eb4ff0f0b96\/master\/63922f5777488eb4ff0f0b96.mov", "destination": "\/"}],"source_root": "","destination_root": "","token": "ATB2_AEAlAD0i60sSO5NyeO7eKvo_C_FgTdhLa39pPg5mKUgqlEh6OAPefziL4_ACjCsfCuhAAFJtW_2a6GL4Xba90HZOs8u_2BTA","direction": "receive","target_rate_cap_kbps": 3000000,"cipher": "aes-128","rate_policy_allowed": "fair","rate_policy": "fair","target_rate_kbps": 700000,"min_rate_cap_kbps": 0,"min_rate_kbps": 0,"lock_min_rate": true,"remote_host": "ats-aws-us-east-1.aspera.io","remote_user": "xfer","sshfp": null,"ssh_port": 33001,"fasp_port": 33001,"http_fallback": false}'

However, apparently this requires a password:

ERROR: Argument: Missing mandatory option: password

We do not want to expose root username or password to the clients that need to perform these operations. Generating these one-time tokens is ideal.

[laurent-martin]

First, in the command you have given, the url scheme is http : this won't work without the s because ATS is only on port 443 and not 80, so use the url https://ats-aws-us-east-1.aspera.io : else this explains the "connection failed".

The username and password options provided to the node plugin can be the accesskey / secret .

Using the node plugin with url / username / password will basically call the node api upload/download_setup which generates a transfer spec, and then start the transfer using that transfer spec.

Now, if you want to use the token directly and provide this to ascli, then you can probably use the server plugin this will directly start a transfer, at minimum provide the url, transfer username and password or key (that creates the base transfer spec), and then provide additional transfer spec info, such as token.

Here an example:

ascli -N server upload --url=ssh://aspera.example.com:33001 --username=xfer 'faux:///test1?1k' --to-folder=/Upload --ts=@json:'{"token":"ATB3_AEA5bVarQ65KbyqjU1Nsln7oFpwGhRZLBvGUN-UnDfAsCeNIhGdqpwL_CT_1AxqEtlUAAF8OJmC-Yn7imJKD7pGNTqf_3BTA"}'

Another example with all in transfer spec:

ascli -N server upload --url=ssh://asperahsts.example.com:33001 --sources=@ts --ts=@json:'{"paths":[{"source":"faux:///test2?1k","destination":"/Upload/file1"}],"token":"ATB3_AEA5bVarQ65KbyqjU1Nsln7oFpwGhRZLBvGUN-UnDfAsCeNIhGdqpwL_CT_1AxqEtlUAAF8OJmC-Yn7imJKD7pGNTqf_3BTA"}'

in ascli, every transfer ends up with a transfer spec, that the use can modify with --ts.
The difference between plugins is how the transfer spec is generated.
For server, it's basically the url and user/pass which end in the transfer spec
for node : it's a call to the setup API.
fo faspex, it first creates a package,
etc...

Another tool is also provided: asession : it takes a value that directly includes a transfer spec like this:

asession @json:'{"file_list_folder":".","spec":{_put_your_transfer_spec_here_}}'

[darcyrailip]

Hi Laurent,

Thanks for this. I'm successfully triggering downloads now using ascli node download, and will try out overriding the transfer spec to trigger multiple files at once.

The output from the command seems different than with aspera ats download, and I'm not sure how to capture it when using NodeJS -- child_process.exec(). Do you have any ideas on that?

I suppose I may need to handle downloads with one process and track download progress with another?

Using this code:

import { exec } from "child_process"

const child = exec(`ascli node download ${params}`) // vs aspera ats download
child.stdout.on("data", data => {
	console.log(data) // data appears here for aspera ats download but not ascli node download
})

[laurent-martin]

Hi,

In fact, as specified here:
https://github.com/IBM/aspera-cli?tab=readme-ov-file#when-to-use-and-when-not-to-use

ascli is perfect to use in shell scripts or interactive shell, but to execute an aspera transfer in any programming language than shell, then it's much more flexible to directly use the Aspera Transfer SDK.

Really, the best to do, if you want to trigger a transfer in nodejs, is to use the SDK instead of the CLI.
As described in above paragraph in ascli manual, you can find nodejs examples here:

https://github.com/laurent-martin/aspera-api-examples/blob/main/app/js/src/examples/node_v2.js
and
https://github.com/laurent-martin/aspera-api-examples/blob/main/app/js/src/examples/node.js

The Aspera Transfer SDK available here:

https://developer.ibm.com/apis/catalog/aspera--aspera-transfer-sdk/Introduction

Note:
When you have a working scenario with ascli it's relatively easy to go direct to APIs:

add option --log-level=debug or --log-level=trace2 and you'll see actual API calls made by ascli, which you can then reproduce with regular REST calls (HTTPS) and transfer SDK.