From 7084b7cf0741fec3f34bc669502fd9a3eeff0cdb Mon Sep 17 00:00:00 2001 From: Matt Pritchard Date: Thu, 6 Feb 2025 19:07:50 +0000 Subject: [PATCH] gridftp, hpxfer, nx, sci updates --- .../docs/data-transfer/data-transfer-tools.md | 1 - content/docs/data-transfer/ftp-and-lftp.md | 2 +- .../globus-transfers-with-jasmin.md | 27 +- .../data-transfer/gridftp-cert-based-auth.md | 255 ------------------ .../docs/data-transfer/gridftp-ssh-auth.md | 5 +- .../docs/data-transfer/hpxfer-access-role.md | 44 --- .../data-transfer/transfers-from-archer2.md | 193 +++---------- .../jasmin-training-accounts.md | 3 +- .../centos7-sci-login-xfer-servers.md | 85 ------ ...graphical-linux-desktop-access-using-nx.md | 6 +- .../interactive-computing/login-problems.md | 8 +- .../interactive-computing/nx-update-nov24.md | 81 ------ .../rocky9-migration-2024.md | 33 ++- data/docs.yml | 3 - 14 files changed, 82 insertions(+), 664 deletions(-) delete mode 100644 content/docs/data-transfer/gridftp-cert-based-auth.md delete mode 100644 content/docs/data-transfer/hpxfer-access-role.md delete mode 100644 content/docs/interactive-computing/centos7-sci-login-xfer-servers.md delete mode 100644 content/docs/interactive-computing/nx-update-nov24.md diff --git a/content/docs/data-transfer/data-transfer-tools.md b/content/docs/data-transfer/data-transfer-tools.md index a31a951ef..64c3420ed 100644 --- a/content/docs/data-transfer/data-transfer-tools.md +++ b/content/docs/data-transfer/data-transfer-tools.md @@ -21,7 +21,6 @@ Tool | Info {{}}sftp{{}} | SSH FTP - works over SSH. {{}}bbcp{{}} | A command-line tool that allows the user to specify parallel transfer over multiple streams, using SSH authentication. {{}}GridFTP (over SSH){{}} | An old but comprehensive data transfer tool. Highly configurable and able to transfer over multiple parallel streams. Used over SSH in this case. Superseded by {{}}Globus{{}} -{{}}GridFTP (certificate-based){{}} | Legacy Gridftp using certificate-based authentication instead of SSH credentials. Efficient for moving large volumes and/or numbers of files, especially over long distances. Superseded by {{}}Globus{{}} {{}}FTP{{}} | File Transfer Protocol. An aged transfer protocol suitable for small file transfers but limited. {{}}LFTP{{}} | Parallel-capable FTP client. wget, curl | Download tools for accessing resources over HTTP primarily. (see 3rd party documentation) diff --git a/content/docs/data-transfer/ftp-and-lftp.md b/content/docs/data-transfer/ftp-and-lftp.md index af68961d0..f4119c339 100644 --- a/content/docs/data-transfer/ftp-and-lftp.md +++ b/content/docs/data-transfer/ftp-and-lftp.md @@ -27,7 +27,7 @@ FTP can only be used as a client on JASMIN, to pull data from external FTP servers to local storage on JASMIN, for example a Group Workspace or your home directory. **There is no FTP server within JASMIN providing the ability to upload files to these locations.** Please use an alternative, more secure -method instead. See other [Data Transfer Tools]({{% ref "data-transfer-tools" %}}) such as [scp/rsync/sftp]({{% ref "rsync-scp-sftp" %}}), [bbcp]({{% ref "bbcp" %}}) or GridFTP ([over SSH]({{% ref "gridftp-ssh-auth" %}}), [certificate-based]({{% ref "gridftp-cert-based-auth" %}}) or using [Globus Online]({{% ref "globus-transfers-with-jasmin"%}})) +method instead. See other [Data Transfer Tools]({{% ref "data-transfer-tools" %}}) such as [scp/rsync/sftp]({{% ref "rsync-scp-sftp" %}}), [bbcp]({{% ref "bbcp" %}}) or GridFTP ([over SSH]({{% ref "gridftp-ssh-auth" %}}) or using [Globus]({{% ref "globus-transfers-with-jasmin"%}})) On the [transfer servers]({{% ref "transfer-servers" %}}), you can use one of the installed FTP clients to download data from elsewhere. These are: diff --git a/content/docs/data-transfer/globus-transfers-with-jasmin.md b/content/docs/data-transfer/globus-transfers-with-jasmin.md index 9da9b4c80..ec0b105c6 100644 --- a/content/docs/data-transfer/globus-transfers-with-jasmin.md +++ b/content/docs/data-transfer/globus-transfers-with-jasmin.md @@ -2,38 +2,37 @@ aliases: - /article/5106-globus-transfers-with-jasmin - /article/5008-data-transfer-tools-using-the-globus-web-interface +- gridftp-cert-based-auth slug: globus-transfers-with-jasmin collection: jasmin-documentation description: Globus transfers with JASMIN title: Globus transfers with JASMIN --- -This article describes how to do data transfers using JASMIN's **new** Globus -endpoint (now called a **collection** ), based on the most recent version of -[Globus Connect Server](https://www.globus.org/globus-connect-server). +This article describes how to do data transfers to and from JASMIN using {{}}Globus{{}}, +an online data transfer service designed specifically for moving large datasets +between research institutions. -JASMIN's old Globus endpoint, based on -the previous version of the Globus service, ceased operating on 18 -December 2023 as support was discontinued by Globus. We have implemented a -new endpoint, based on Globus Connect Server v5.4, with equivalent (but better!) -functionality. +{{}} +**Globus** now replaces the previous certificate-based **gridftp** service. -The new collection can be used as a drop-in replacement for the previous -endpoint, aside from a few differences in terminology, and a change to the -authentication process. +Although gridftp transfers are currently still possible (using the perhaps confusingly-named `globus-url-copy` client tool +still available on the +transfer servers), this now only works with [ssh authentication]({{% ref "gridftp-ssh-auth" %}}). +{{}} ## Main differences -There are some differences to how the new (v5) version of Globus works on JASMIN compared to previously: +JASMIN moved to a newer version of Globus in 2023, resulting in a few changes: - Users now interact with a **collection** - **Most users**: please use ["JASMIN Default Collection"](https://app.globus.org/file-manager/collections/a2f53b7f-1b4e-4dce-9b7c-349ae760fee0/overview) with ID `a2f53b7f-1b4e-4dce-9b7c-349ae760fee0` - For **STFC users only** where the other collection (either {{}}GCP{{}} or {{}}GCS{{}}) is within STFC, an additional collection is provided ["JASMIN STFC Internal Collection"](https://app.globus.org/file-manager/collections/9efc947f-5212-4b5f-8c9d-47b93ae676b7/overview) and has ID `9efc947f-5212-4b5f-8c9d-47b93ae676b7`. -- You now use the JASMIN Accounts Portal to authenticate (using your JASMIN account credentials) via OpenID Connect (OIDC). +- You now use the JASMIN Accounts Portal to authenticate (using your JASMIN account credentials) via OpenID Connect (OIDC). - During the authentication process, you are redirected to the JASMIN Accounts Portal to link your Globus identity with your JASMIN account. - Consent needs to be granted at a number of points in the process to allow the Globus service to carry out actions on your behalf. - The default lifetime of the authentication granted to your JASMIN account is now **30 days**. After this, you may need to refresh the consent for your "session". -- This service is now available to **all** users of JASMIN: you no longer need to hold the `hpxfer` access role. +- This service is now available to **all** users of JASMIN: you no longer need the `hpxfer` access role (now removed). The following examples show you how to authenticate with the new JASMIN Default Collection and list the contents of your home directory. As before, diff --git a/content/docs/data-transfer/gridftp-cert-based-auth.md b/content/docs/data-transfer/gridftp-cert-based-auth.md deleted file mode 100644 index c79cf322b..000000000 --- a/content/docs/data-transfer/gridftp-cert-based-auth.md +++ /dev/null @@ -1,255 +0,0 @@ ---- -aliases: /article/3808-data-transfer-tools-gridftp-cert-based-auth -description: 'Data Transfer Tool: GridFTP (certificate-based authentication)' -slug: gridftp-cert-based-auth -title: 'GridFTP (certificate-based authentication)' ---- - -{{}} -Deprecated: please use Globus instead for transfers which previously used this method. -{{}} - -This article describes how to transfer data using gridftp with certificate- -based authentication. - -{{}}The `globus-url-copy` command used here should not be confused with the Globus online data transfer service. They used to be associated, but no longer. If you are starting out and looking for a reliable, high-performance transfer method, the recommendation now is to learn about [Globus Transfers with JASMIN]({{% ref "globus-transfers-with-jasmin" %}}) (using the Globus online data transfer service) instead of command-line gridftp as described in this document.{{}} - -## Basics of certificate-based authentication - -Gridftp servers commonly use a network of "trust" based on electronic -certificates. In order to make use of a gridftp server at one end of your -proposed transfer, you will need to use a certificate which identifies you as -the user, and which is issued by an identity provider which is "trusted" by -the servers at both ends. The trust between the servers is maintained by the -administrators of the service who will ensure that the necessary certificates -are in place. - -The presentation of a valid credential which is trusted by the server at the -other end is merely the authentication step (proving who you are). -Authorisation also needs to follow: you, as a user (identified by the -credential you present) need to be authorised to use the resource at the other -end. You should check with the operator of the other gridftp server to see -what additional steps are required before you can actually perform a transfer. - -## Getting a short-term credential - -In order to access the JASMIN gridftp server, you can now use your JASMIN -portal account to gain a short-term credential which the server will recognise -to authenticate you. This is the same username and password you would use to -log in to to administer your JASMIN account. -**IT IS NOT YOUR SSH PASSPHRASE.** - -Here's what to do: - - 1. Download tools to interact with JASMIN's Online Certificate Authority (OnlineCA). You can use these to interact with other OnlineCAs too (not just JASMIN's. These replace the "myproxy-logon" tool previously mentioned here) - 2. Use these tools to: - 1. "Bootstrap trust" i.e. to setup your local certificate store with those needed to interact with the JASMIN server [First time use only] - 2. Obtain a short-term credential using your JASMIN account details [First time, and to renew your short-term credendial as needed] - 3. Use this short-term credential to authenticate with a remote gridftp server which trusts this credential (for example, the JASMIN gridftp server) - -### Download OnlineCA tools - -On the machine you intend to use as the transfer client, e.g. -`xfer1.jasmin.ac.uk`, in your JASMIN home directory, download 2 shell scripts -which will interact with the Online CA for you. Make them executable: - -{{}} -wget https://raw.githubusercontent.com/cedadev/online_ca_client/master/contrail/security/onlineca/client/sh/onlineca-get-cert-wget.sh -wget https://raw.githubusercontent.com/cedadev/online_ca_client/master/contrail/security/onlineca/client/sh/onlineca-get-trustroots-wget.sh -chmod u+x onlineca-get-*.sh -{{}} - -View help information for the shell scripts: - -{{}} -./onlineca-get-trustroots-wget.sh -h -./onlineca-get-cert-wget.sh -h -{{}} - -Bootstrap trust between your own machine and the JASMIN gridftp server: (First time only) - -{{}} -./onlineca-get-trustroots-wget.sh -U https://slcs.jasmin.ac.uk/trustroots/ -b -(out)Bootstrapping Short-Lived Credential Service root of trust. -(out)Trust roots have been installed in /home/users/USERNAME/.globus/certificates. -{{}} - -Obtain a credential, to be written to an output file `credfile` using your -JASMIN Accounts Portal username USERNAME: - -{{}} -./onlineca-get-cert-wget.sh -U https://slcs.jasmin.ac.uk/certificate/ -l USERNAME -o ./cred.jasmin -{{}} - -When prompted, enter the password associated with your **JASMIN** account -**(NOT your SSH passphrase)** - -Change the permissions on your newly-created `cred.jasmin` file so that it's -only readable by you (client software may insist on this): - -{{}} -chmod 600 ./cred.jasmin -{{}} - -This credential obtained by this method is valid by default for 720 hours (30 -days), as you can see by inspecting the certificate using the following -command: - -{{}} -openssl x509 -in cred.jasmin -noout -startdate -enddate -(out) notBefore=Mar 11 17:32:59 2022 GMT -(out) notAfter=Apr 10 17:32:59 2022 GMT -{{}} - -After the `notAfter` date, it will no longer be valid, but you can -repeat this process at any time (e.g. before it expires) to update it. - -## Example Gridftp usage - -(General case, or with a JASMIN host as gridftp client) - -Once you have obtained a valid short-term credential on the client transfer -server, and assuming that the gridftp server at the remote end of the -transfer recognises and is able to authorize you via this credential, then -you should be able to transfer data between the remote server and local -client with commands such as shown below: - -Please consult the documentation for the `globus-url-copy` command for the -full range of options and arguments. - -Please note that the examples below use a fictitious client `gridftp-client.localsite.ac.uk` and server `gridftp-server.remotesite.ac.uk` which need to be replaced in your commands with the hostname of the actual gridftp server and client you are actually using. - -Check help documentation for the globus-url-copy command: - -{{}} -globus-url-copy -help -{{}} - -**NOTE:** On some systems, you have to load a relevant module to get access to the globus-url-copy command, however not on the JASMIN \[hp\]xfer servers. - -It is recommended to try things out using the regular xfer servers xfer-vm-0[12] but to perform "real" transfers using hpxfer[34] for better performance. - -1\. Remote directory listing issued by client on `gridftp- -client.localsite.ac.uk` to server `gridftp-server.remotesite.ac.uk` where you -have a home directory `/home/users/USERNAME`: - -{{}} -globus-url-copy -cred cred.jasmin -vb -list gsiftp://gridftp-server.remotesite.ac.uk/home/users/USERNAME/ -{{}} - -2\. Download a file from remote directory `/home/users/USERNAME` to -destination on the client machine: - -{{}} -globus-url-copy -cred cred.jasmin -vb gsiftp://gridftp-server.remotesite.ac.uk/home/users/USERNAME/myfile file:///path/to/localdir/myfile -{{}} - -The `-p N` and `-fast` options can additionally be used in combination to -enable `N` parallel streams at once, as shown below. You can experiment with N -in the range 4 to 16 to obtain the best performance, but please be aware that -many parallel transfers can draw heavily on shared resources and degrade -performance for other users: - -{{}} -globus-url-copy -cred cred.jasmin -vb -p 16 -fast gsiftp://gridftp-server.remotesite.ac.uk/home/users/USERNAME/myfile file:///path/to/localdir/myfile -{{}} - -3\. Recursively download the contents of a directory on a remote location to a -local destination. - -{{}} -globus-url-copy -cred cred.jasmin -vb -p 4 -fast -cc 4 -cd -r gsiftp://gridftp-server.remotesite.ac.uk/home/users/USERNAME/mydir/ file:///path/to/localdir/mydir/ -{{}} - -Where: - - - `-cc N` requests `N` concurrent transfers (in this case, each with `p=4` parallel streams) - - `-cd` requests creation of the destination directory if this does not already exist - - `-r` denotes recursive transfer of directories - * `-sync` and `-sync-level` options can be used to synchronise data between the two locations, where destination files do not exist or differ - y criteria that can be selected) from corresponding source files. See `-help` option for details. - - the `file:///` URI is used to specify the destination on the local file system. - -## Uploading data - -The above commands can also be adapted to invoke transfers from a local source -to a remote destination, i.e. uploading data, since the commands all take the -following general form: - -{{}} -globus-url-copy [OPTIONS] source-uri desination-uri -{{}} - -You can use the above examples by replacing the local machine `gridftp-client.localsite.ac.uk` with one of the jasmin `xfer` or `hpxfer` servers as a client, To do this, you first need to be logged in via SSH to one of these hosts and can initiate a transfer by invoking `globus-url-copy` in one of the ways above. - -- For high-performance transfer (large volumes and/or longer distances), use [Globus]({{% ref "globus-transfers-with-jasmin" %}}) or the [hpxfer servers]({{% ref "transfer-servers/#hpxfer-servers" %}}) -- For remote hosts using JASMIN's dedicated network link (Met Office only) use `xfer-vm-0[123].jasmin.ac.uk` as the client (These are virtual machines so have limited performance, but your transfer will be over a dedicated network connection) - -## Connecting to the JASMIN GridFTP server - -In order to do a transfer using a JASMIN host as the gridftp server (rather than -client), you would need to interact with the JASMIN GridFTP server -`gridftp1.jasmin.ac.uk`. You cannot log in to this server directly via SSH: -you only initiate GridFTP transfers to and from it from another client. - -In the following example, a client is initiated on a fictitious remote host -`client.remotesite.ac.uk` and tests the connection by transferring from -/dev/zero on the local machine (at `remotesite` ) to /dev/null on the JASMIN -gridftp server. Note that you can use the SLCS server at JASMIN to obtain the -short-term credential required ( **but the first time, you will need to -download and use the OnlineCA tools as described above** ). You can renew your -credential and perform the test transfer as follows: - -{{}} -./onlineca-get-cert-wget.sh -U https://slcs.jasmin.ac.uk/certificate/ -l USERNAME -o ./cred.jasmin -globus-url-copy -cred cred.jasmin -vb -p 8 -fast /dev/zero gsiftp://gridftp1.jasmin.ac.uk/dev/null -(out) Source: file:///dev/ -(out) Dest: gsiftp://gridftp1.jasmin.ac.uk/dev/ -(out) zero -> null -(out) -(out) 4153409536 bytes 792.20 MB/sec avg 792.20 MB/sec inst -{{}} - -This server is also used as the JASMIN GridFTP Server globus endpoint, see -[GridFTP transfers using Globus Online]({{% ref "globus-command-line-interface" %}}) (however you can only currently use your CEDA -SLCSs credential with Globus Online. The JASMIN team is working on a solution -for this). - -Please note that the servers `xfer-vm-0[123].jasmin.ac.uk` and -`hpxfer[34].ceda.ac.uk` are not gridftp **servers**. They have the `globus-url-copy` client installed, so can be used as clients to connect to remote -gridftp servers, and also support [gridftp over SSH]({{% ref "gridftp-ssh-auth" %}}) (both incoming and outgoing), but do not act as -servers for certificate-based gridftp as shown in these examples. The JASMIN -gridftp server for read-write access to home directories and group workspaces -is `gridftp1.jasmin.ac.uk`. Access to this requires the [hpxfer access role]({{% ref "hpxfer-access-role" %}}). See -also [Transfer Servers]({{% ref "transfer-servers" %}}). - -## Third-party transfers - -It should be possible, with the correct configuration at each site, to -initiate on host `A` a transfer of data between two other gridftp servers `B` and -`C` (a third party transfer). Both URIs would use `gsiftp:` as the protocol: - -{{}} -globus-url-copy -vb -p 4 gsiftp://B/source gsiftp://C/destination -{{}} - -Further information can be found in the documentation for globus-url-copy. - -This is the basis of the [Globus Online](https://www.globus.org/app/transfer) -managed service to orchestrate and monitor transfers between gridftp endpoints -in a more user-friendly way. It has evolved considerably since diverging from -the "traditional" gridftp setup described in this article and is recommended as -it provides a much easier user experience and better reliability. - -See {{}}. - -## Future plans - -As [support for the open-source Globus Toolkit (including globus-url-copy) has -now been withdrawn by Globus](https://www.globus.org/blog/support-open-source- -globus-toolkit-ends-january-2018), the future of direct gridftp transfers is -uncertain. It is currently maintained by the Grid Community Forum. - -**We advise users to spend some time understanding and testing -transfer workflows with the {{}}Globus Online{{}} transfer service, -including the command-Line, web interfaces and (for advanced users) a Python SDK, as these -are likely to replace direct gridftp on JASMIN in due course.** diff --git a/content/docs/data-transfer/gridftp-ssh-auth.md b/content/docs/data-transfer/gridftp-ssh-auth.md index d18df222e..c5d01d6f3 100644 --- a/content/docs/data-transfer/gridftp-ssh-auth.md +++ b/content/docs/data-transfer/gridftp-ssh-auth.md @@ -169,7 +169,6 @@ Push data to JASMIN from a remote server: globus-url-copy -vb -p 8 -fast mydir/myfile sshftp://username@hpxfer3.jasmin.ac.uk/group_workspaces/jasmin/myworkspace/mydir/ {{}} -Note that for this to work, you need to be able to authenticate over SSH to the JASMIN host. This should be possible if you can log in interactively, but will NOT work if you are using the command in a cron job or other situation where your ssh-agent (on the host remote to JASMIN) is not running and/or does not have access to your private key. For those situations, consider using either +Note that for this to work, you need to be able to authenticate over SSH to the JASMIN host. This should be possible if you can log in interactively, but will NOT work if you are using the command in a cron job or other situation where your ssh-agent (on the host remote to JASMIN) is not running and/or does not have access to your private key. -- [Globus (recommended)]({{% ref "globus-transfers-with-jasmin" %}}), or -- [Gridftp using certificate-based authentication]({{% ref "gridftp-cert-based-auth" %}}) +Instead, for those situations, use [Globus]({{% ref "globus-transfers-with-jasmin" %}}) diff --git a/content/docs/data-transfer/hpxfer-access-role.md b/content/docs/data-transfer/hpxfer-access-role.md deleted file mode 100644 index 0eae86a28..000000000 --- a/content/docs/data-transfer/hpxfer-access-role.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -aliases: /article/4414-data-transfer-hpxfer -description: 'Access to certain high-performance data transfer methods' -title: 'hpxfer access role' ---- - -{{}} -This article is deprecated, since the `hpxfer` is no longer needed for -the new Rocky 9 services introduced Autumn 2024. - -However, until the older machines (`hpxfer[12]` & `gridftp1`) are taken out of service, -the role is still required for access to those. -{{}} - -This article explains about access to high-performance data transfer services. - -## Applying for access - -Some data transfer services are hosted in the JASMIN Data Transfer Zone (a special area of JASMIN's network, located optimally for connections to the outside world via {{}}JANET{{}}) for increased performance. However, to -maintain security in this zone, access to some services is controlled via an additional access -role `hpxfer`. If you have a login account already, you can apply here for -this additional role: - -{{< button href="https://accounts.jasmin.ac.uk/account/login/?next=/services/additional_services/hpxfer/" >}}Apply for hpxfer{{< /button >}} - -## Additional information required - -If you will be connecting from your home institution directly via ssh or ssh- -based gridftp to one of the servers in the JASMIN Data Transfer Zone, the -specific IP address of your machine will need to be added to an allow-list. -Please supply this as part of the application process above. - -However, this should not be necessary if you are accessing the server from a -remote server which is already allowed. - -{{}}If your **only** reason for applying for `hpxfer` is in order to use the **Globus** service on JASMIN, this is no longer required. You only need this role for accessing the hpxfer servers or for using certificate-based gridftp with the `globus-url-copy` command. -{{}} - -**Inward** pulls of data (to JASMIN) are possible by logging in to -`hpxfer[1,2].jasmin.ac.uk` **via the login servers** and pulling data from -external data sources. So if you're not sure what IP address to use when -applying for access above, it is OK to quote the IP address of -`login1.jasmin.ac.uk`, i.e. `130.246.130.28` as a "dummy" value if you will -only be accessing them via this route, and not directly from outside. diff --git a/content/docs/data-transfer/transfers-from-archer2.md b/content/docs/data-transfer/transfers-from-archer2.md index 4a4ff67a5..311b7a8dd 100644 --- a/content/docs/data-transfer/transfers-from-archer2.md +++ b/content/docs/data-transfer/transfers-from-archer2.md @@ -1,6 +1,6 @@ --- aliases: /article/4997-transfers-from-archer2 -description: Transfers from ARCHER2 +description: Transferring data from ARCHER2 to JASMIN, efficiently slug: transfers-from-archer2 title: Transfers from ARCHER2 --- @@ -8,7 +8,7 @@ title: Transfers from ARCHER2 ## Choice of available Tools/Routes See [Data Transfer Tools]({{% ref "data-transfer-tools" %}}) for general -details. +information. Users transferring data between ARCHER2 and JASMIN are often transferring relatively large sets of data, so it is important to choose the most @@ -30,48 +30,11 @@ Please note: ## Available transfer methods -### Basic SSH transfer +1. [Globus](#1st-choide-method-globus) (recommended) +2. [Basic SSH transfer](#basic-ssh-transfer) (slow but convenient) +3. [Gridftp using SSH authentication](#gridftp-using-certificate-auth) (efficient, currently still available but now superceded in convenience/reliability by Globus) -[**scp/rsync/sftp**]({{% ref "rsync-scp-sftp" %}}): Simple transfers using easy method, pushing data to general purpose xfer nodes. Convenient, but limited performance. - -_source_ | _dest_ | _notes_ ---- | --- | --- -`login.archer2.ac.uk` | `xfer-vm-0[123].jasmin.ac.uk` | to virtual machine at JASMIN end -`login.archer2.ac.uk` | `hpxfer[34].jasmin.ac.uk` | to high-performance physical machine at JASMIN end -{.table .table-striped} - -### GridFTP over SSH - -[GridFTP over SSH]({{% ref "gridftp-ssh-auth" %}}): GridFTP performance with convenience of SSH. Requires persistent ssh agent -on local machine where you have your JASMIN key. **2nd choice method** - -_source_ | _dest_ | _notes_ ---- | --- | --- -`login.archer2.ac.uk` | `hpxfer[34].jasmin.ac.uk` | -{.table .table-striped} - -### GridFTP using certificate auth - -[GridFTP using certificate auth]({{% ref "gridftp-cert-based-auth" %}}): Fully-featured GridFTP. Suitable for person-not-present transfers & long- -running ARCHER2 workflows. **3rd choice method: legacy technology which will be discontinued on JASMIN in 2025** - -Additional requirement: - -- you need to have registered the subject of your JASMIN-issued short-term credential with ARCHER2 support. - -_source_ | _dest_ | _notes_ ---- | --- | --- -`login.archer2.ac.uk` | `gridftp1.jasmin.ac.uk` | over 10G JANET.
Dedicated GridFTP server.
**No need for persistent SSH agent at ARCHER2 end** -{.table .table-striped} - -Notes: - -- We are currently struggling to get the legacy components working on our new operating system, Rocky 9, so the current service -continues on the old (CentOS7) server `gridftp1` for now, but may need to be withdrawn at short notice. -- Even if/when we succeed in redeploying the service on Rocky 9, we plan to discontinue this service now that a better alternative -is available with Globus. - -## 1st choide method: Globus +## 1st choice method: Globus This is now the recommended method, because: @@ -108,7 +71,7 @@ export jdc=a2f53b7f-1b4e-4dce-9b7c-349ae760fee0 3\. Check access to these collections -These collecitons are restricted-access rather than public, so your access to them is via a series of authentication/authorisation/consent steps which the following actions will guide you through: +These collections are restricted-access rather than public, so your access to them is via a series of authentication/authorisation/consent steps which the following actions will guide you through: {{}} globus ls $a2c:/~/ @@ -172,7 +135,7 @@ globus task wait aa0597a4-80a7-11ef-b36b-a1206a7ee65f will now return control immediately, since the task has completed. -Globus transfer tasks are aysychronous, submitted to **your own** mini-queue, +Globus transfer tasks are asynchronous, submitted to **your own** mini-queue, where you can have as many queued tasks as you like but only 3 in progress at any one time. This ensures good performance for all users, but your tasks do not linger in long multi-user queues. The best way to reassure yourself of this is to try it out. @@ -185,7 +148,7 @@ Relevant examples: - [sync with wait](https://github.com/mjpritchard/my-globus-examples/blob/main/sync_wait_simple.sh) using the CLI. - [Repeatable transfer](https://github.com/mjpritchard/my-globus-examples/blob/main/repeatableTransferWithRefreshTokenStorage.py) using the PythonSDK (more advanced) -Note that Globus transfers (and other actions) can be managed & monitoried by: +Note that Globus transfers (and other actions) can be managed & monitored by: - a web interface - the command-line interface, and @@ -196,10 +159,28 @@ all of which interact with the same underlying service. NCAS-CMS users should note that work is currently underway to adopt Globus as a drop-in replacement for certificate-based gridftp in Rose suites currently in use for automating processing and transferring to JASMIN. -## 2nd choice method: gridftp over SSH +## 2nd choice method: Basic SSH transfer + +[**scp/rsync/sftp**]({{% ref "rsync-scp-sftp" %}}): Simple transfers using easy method, pushing data to general purpose xfer nodes. Convenient, but limited performance. -The next-best method for transfers between ARCHER2 and JASMIN is using globus- -url-copy with SSH authentication, as described below: +_source_ | _dest_ | _notes_ +--- | --- | --- +`login.archer2.ac.uk` | `xfer-vm-0[123].jasmin.ac.uk` | to virtual machine at JASMIN end +`login.archer2.ac.uk` | `hpxfer[34].jasmin.ac.uk` | to high-performance physical machine at JASMIN end +{.table .table-striped} + +## 3rd choice method: gridftp over SSH + +[GridFTP over SSH]({{% ref "gridftp-ssh-auth" %}}): GridFTP performance with convenience of SSH. Requires persistent ssh agent +on local machine where you have your JASMIN key. + +_source_ | _dest_ +--- | --- | +`login.archer2.ac.uk` | `hpxfer[34].jasmin.ac.uk` +{.table .table-striped} + +The next-best method for transfers between ARCHER2 and JASMIN is using the `globus-url-copy` client tool with SSH authentication, as described below: +(**This is not [Globus](#1st-choice-method-globus), however, despite the tool name!**) 1\. Load your SSH keys for both JASMIN and ARCHER2 on your local machine, then log in to ARCHER2. @@ -238,119 +219,26 @@ which globus-url-copy 3\. Transfer a single file to your home directory on JASMIN (limited space, but to check things work) - {{}} -globus-url-copy -vb sshftp://@hpxfer1.jasmin.ac.uk/~/ +globus-url-copy -vb sshftp://@hpxfer3.jasmin.ac.uk/~/ {{}} Obviously, replace `` with the path to the file you want to transfer. -From here on, the commands are the same as described above in the "1st choice -method" but simply replace - -```bash --cred cred.jasmin gsiftp://gridftp1.jasmin.ac.uk -``` - -with - -```bash -sshftp://@hpxfer1.jasmin.ac.uk -``` - - - -## 3rd choice method: certificate-based gridftp - -{{}}The use of certificate-based gridtp for transfers to JASMIN has now been replaced by Globus. -Server `gridftp1` will be closed on Friday 13th December 2024 at 16:00 -{{}} - -This method for transfers between ARCHER2 and JASMIN uses -globus-url-copy with the concurrency option, as described below, but using -certificate-based authentication rather than SSH. This will work for person- -not-present transfers, so is suitable for long-running processes on ARCHER2 -which need to spawn a transfer to JASMIN at intervals up to a month from -initiation. - -1\. Load your SSH key for ARCHER2 on your local machine, then log in to -ARCHER2. - -This method **does not** require you to use your JASMIN SSH key. It involves: - -- obtaining tools to communicate with JASMIN's short-lived credentials service -- using the service to obtain a credential (it should last for 30 days, but a new one can be obtained at any time) -- using the credential to initiate a transfer (this what you would need to repeat for each transfer) - -A fuller explanation of the process is given in this document: - -- [Data Transfer Tools: GridFTP (certificate-based authentication)]({{% ref "gridftp-cert-based-auth" %}}) - -Once you have done these steps, you should be able to obtain a short-term -credential as follows (do this command at the ARCHER2 end, after having -downloaded the onlineca script as described in the document mentioned above): - -{{}} -./onlineca-get-cert-wget.sh -U https://slcs.jasmin.ac.uk/certificate/ -l USERNAME -o ./cred.jasmin -chmod 600 cred.jasmin -{{}} - -Note that the path `./` is used for the script `onlineca-get-cert-wget.sh`, -but you should use the path to wherever you saved it. Alternatively, if you -make yourself a `bin` directory and add that to your `PATH`, then you don't -need to specify the path. - -2\. Load the `gct` module (to make the current `globus-url-copy` command -available in your path on ARCHER2). - -Once loaded, check with `which` to see that you have the `globus-url-copy` command available to you. - -{{}} -module load gct -which globus-url-copy -(out)/work/y07/shared/gct/v6.2.20201212/bin/globus-url-copy -{{}} - -3\. Transfer a single file to your home directory on JASMIN (limited space, -but to check things work) - -{{}} -globus-url-copy -vb -cred cred.jasmin SRC/FILE gsiftp://gridftp1.jasmin.ac.uk/DEST/FILE -{{}} - -Note that we specify the credentials file `cred.jasmin` and use the protocol -`gsiftp://` with no need to specify the username in the connection string -(we've used the path `/~/` to signify "my home directory" as the destination -path). Note also that the hostname in this case, `gridftp1.jasmin.ac.uk` is a -host that you can ONLY connect to directly using gsiftp: it does not permit -SSH connections. - -In all other aspects, the transfer is the same as for the SSH method (see "2nd -choice method" below), so the commands below are very similar: we're just -using the gsiftp method instead of sshftp (both are ways of using the gridftp -transfer protocol) - 4\. Recursively transfer a directory of files, using the concurrency option for multiple parallel transfers {{}} -globus-url-copy -vb -cd -r -cc 4 -cred cred.jasmin SRC/DATA/ gsiftp://gridftp1.jasmin.ac.uk/DEST/DATA/ +globus-url-copy -vb -cd -r -cc 4 SRC/DATA/ sshftp://@hpxfer3.jasmin.ac.uk/DEST/DATA/ {{}} -**NOTE:** The `-cc` option initiates the parallel transfer of several files at +**NOTE:** - The `-cc` option initiates the parallel transfer of several files at a time, which achieves good overall transfer rates for recursive directory transfers. This is different from using the `-p N -fast` options which use parallel network streams to parallelism the transfer of each file. - -The `-p N -fast` options (for parallel-streamed transfers) are not currently -working to all JASMIN storage locations, so use at your own risk until further -notice. The transfer should work OK out of ARCHER2 (check by writing a single -file to `/dev/null` at the JASMIN end) but currently will not work properly -when writing to the SOF storage (`/gws/nopw/j04` or `/gws/nopw/j07`, or -`/work/xfc/vol[1-3]`, though other paths should work OK). This is a known -issue at the JASMIN end, thought to be related to network configuration, which -is still under investigation. Single-stream transfers (omitting the `-p N --fast` options) should work fine. +A sensible value for `-cc` is 2 or 4, whereas a sensible value for `-p` is between +2 and 16. In both cases, try first and avoid numbers at the higher end, which can +increase resource usage without further performance gains. Here, the options used are (see `man globus-url-copy` for full details): @@ -366,14 +254,11 @@ Here, the options used are (see `man globus-url-copy` for full details): Copy files in subdirectories ``` -Experiment with different concurrency options (4 is a good start, more than 16 -would start to "hog" resources so please consider - 5\. Use the sync option to synchronise 2 directories between source and target file systems: {{}} -globus-url-copy -vb -cd -r -cc 4 -sync -cred cred.jasmin SRC/DATA/ gsiftp://gridftp1.jasmin.ac.uk/DEST/DATA/ +globus-url-copy -vb -cd -r -cc 4 -sync SRC/DATA/ sshftp://@hpxfer3.jasmin.ac.uk/DEST/DATA/ {{}} where `SRC/DATA/` and `/DEST/DATA/` are source and destination paths, @@ -408,5 +293,5 @@ cost.** So a full sync including comparison of checksums would be: {{}} -globus-url-copy -vb -cd -r -cc 4 -sync -sync-level 3 -cred cred.jasmin src/data/ gsiftp://gridftp1.jasmin.ac.uk/path/dest/data/ -{{}} \ No newline at end of file +globus-url-copy -vb -cd -r -cc 4 -sync -sync-level 3 SRC/DATA/ sshftp://@hpxfer3.jasmin.ac.uk/DEST/DATA/ +{{}} diff --git a/content/docs/getting-started/jasmin-training-accounts.md b/content/docs/getting-started/jasmin-training-accounts.md index 49965d79d..9c00a88a7 100644 --- a/content/docs/getting-started/jasmin-training-accounts.md +++ b/content/docs/getting-started/jasmin-training-accounts.md @@ -43,9 +43,8 @@ helpdesk with the following information: - Any special requests for accessing resources - By default, the training accounts have access to the following services. - - Any request for other services beyond these would need to be considered by the JASMIN team: + - Any request for other services beyond these would need to be considered by the JASMIN team: - Login, nx-login, sci and xfer servers - - `hpxfer` service - `workshop` group workspace (`/gws/pw/j07/workshop`) - use of LOTUS via the `workshop` Slurm queue - Jupyter Notebooks service (requires users to set password) diff --git a/content/docs/interactive-computing/centos7-sci-login-xfer-servers.md b/content/docs/interactive-computing/centos7-sci-login-xfer-servers.md deleted file mode 100644 index 1ebab77ad..000000000 --- a/content/docs/interactive-computing/centos7-sci-login-xfer-servers.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -aliases: /article/4859-centos7-sci-login-xfer-servers -description: CentOS7 Login, Sci and Xfer servers -title: CentOS7 Login, Sci and Xfer servers -weight: 110 -draft: true ---- - -{{}} -This article was used to describe the new servers introduced as part of the RHEL -> CentOS7 migration in 2020/2021. It is now superseded. -{{}} - -This article describes the CentOS7 versions of the 3 familiar -types of JASMIN server. These are: - -- CentOS7 Login servers -- CentOS7 Scientific analysis (sci) servers -- CentOS7 Data transfer (xfer) servers - -## Login servers - -The new CentOS7 login (bastion or gateway) servers are available to access -resources within JASMIN. Users with the `jasmin-login` access role -can access the following servers using SSH: - -**Table 1** : Centos7 Login servers - -Login server name -|---| -`login1.jasmin.ac.uk` | -`login2.jasmin.ac.uk` | -`login3.jasmin.ac.uk` | -`login4.jasmin.ac.uk` | - -See also NX login servers which are part of the [graphical linux desktop]({{% ref "graphical-linux-desktop-access-using-nx" %}}) service. - -You should use these servers as your default route into JASMIN. -Additional servers in this series will follow in due course. -These login servers provide a table displayed at login showing the -list of available sci servers and their current load and number of logged-in -users. Please make use of these to select the most appropriate sci server. - -[Further details of the login servers]({{% ref "login-servers" %}}) - -## Scientific analysis servers - -CentOS7 Scientific analysis servers are now available (see **Table 2** ). -These can be used by users with the `jasmin-login` access role to test -workflows/tasks that: - -1. Use the {{< link "https://drive.google.com/file/d/1gD9C0TZyNITibgDhlv3pRzgd4JjzVfBW/view" >}}new software environments{{}} -2. Do not make use of the software `/apps/contrib` (which has only been tested for RHEL6 operating systems) -3. Do not make use of the software available under the `module` environment (which has only been tested for RHEL6 operating systems) except the modules `jaspy` and `jasmin-sci` -4. A job submitted to the new batch scheduler Slurm will run on a CentOS7 node in the LOTUS cluster. - -**Table 2:** List of CentOS7 Scientific analysis servers - -Server name ---- | -`sci1.jasmin.ac.uk` | -`sci2.jasmin.ac.uk` | -`sci3.jasmin.ac.uk` | -`sci4.jasmin.ac.uk` | -`sci5.jasmin.ac.uk` | -`sci6.jasmin.ac.uk` | -`sci8.jasmin.ac.uk` | - -[Further details of the sci servers]({{% ref "sci-servers" %}}). - -## Transfer servers - -CentOS7 transfer (xfer) servers are now available (see Table 3). These can -be used by users with the `jasmin-login` access role and are functionally the -same as their predecessors. - -**Table 3:** Centos7 xfer servers - -Server name | Details ---- | --- | -`xfer1.jasmin.ac.uk` | -`xfer2.jasmin.ac.uk` | -`xfer3.jasmin.ac.uk` | (special access rules similar to `login2`, but requires additional access role, apply [here](https://accounts.jasmin.ac.uk/services/additional_services/xfer-sp)) -`hpxfer[12].jasmin.ac.uk` | (physical, high-performance transfer servers,require `hpxfer` access role) | - -[Further details of the xfer servers]({{% ref "transfer-servers" %}}) diff --git a/content/docs/interactive-computing/graphical-linux-desktop-access-using-nx.md b/content/docs/interactive-computing/graphical-linux-desktop-access-using-nx.md index 134b23f7f..990e94709 100644 --- a/content/docs/interactive-computing/graphical-linux-desktop-access-using-nx.md +++ b/content/docs/interactive-computing/graphical-linux-desktop-access-using-nx.md @@ -1,5 +1,7 @@ --- -aliases: /article/4810-graphical-linux-desktop-access-using-nx +aliases: +- /article/4810-graphical-linux-desktop-access-using-nx +- nx-update-nov24 description: Graphical linux desktop using NoMachine NX tags: - nx @@ -39,7 +41,7 @@ name | notes `nx1.jasmin.ac.uk` | `nx2.jasmin.ac.uk` | `nx3.jasmin.ac.uk` | -`nx4.jasmin.ac.uk` | (not yet converted to Rocky 9) +`nx4.jasmin.ac.uk` | (new server now available) {.table .table-striped .w-auto} ### Notes diff --git a/content/docs/interactive-computing/login-problems.md b/content/docs/interactive-computing/login-problems.md index 2c0589846..5c992629b 100644 --- a/content/docs/interactive-computing/login-problems.md +++ b/content/docs/interactive-computing/login-problems.md @@ -158,11 +158,9 @@ users.** The [`sci` servers]({{% ref "sci-servers"%}}) and [`xfer` servers]({{% ref "transfer-servers" %}}) should be available to all with `jasmin-login` access -(see above). However, some other machines are restricted to particular project -participants and require special permission to use. For example, **old** the high- -performance transfer servers `hpxfer[12].jasmin.ac.uk` require the -{{}}hpxfer access role{{}}, which can be applied for at the JASMIN accounts portal, -as can most roles currently in use. +(see above). Where you need special access to a particular service, this will be indicated +in the relevant documentation on this site. Normally you would apply for access for the relevant +access role via the {{}}JASMIN accounts portal{{}}. **3) There is a problem with the host you are trying to connect to.** diff --git a/content/docs/interactive-computing/nx-update-nov24.md b/content/docs/interactive-computing/nx-update-nov24.md deleted file mode 100644 index 78fae242a..000000000 --- a/content/docs/interactive-computing/nx-update-nov24.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -description: Update on connecting to NoMachine NX service November 2024 -tags: -- nx -- nomachine -- desktop -- X11 -title: NX update November 2024 -weight: 55 ---- - -## Good news! Connection problems resolved - -We now have a solution to the issues some users had with connecting to the NoMachine NX (graphical linux desktop) service, especially from Windows. - -- This involves updating your SSH key to a new algorithm, ECDSA (previously RSA). -- We have updated our accounts system to support these new keys, which can now be used throughout JASMIN. -- So [updating your JASMIN SSH key](#updating-your-key) is the first step, but you then have a choice of connection options. - -In these notes, `~/` means "your home directory". On Windows this is also represented by `%USERPROFILE%`. - -We will update the full documentation and videos in due course, at which point this page will be removed and you will be redirected to that page from here. - -## Key presentation options - -The configuration you need depends on how you choose to present your key: - -- **File-based method**: specify the location of your key file: no admin permission required. - -- **Agent-based methods**: load your key into an ssh-agent which persists and can be used for subsequent connections across multiple applications. - - Compatible agents: - - Windows 10 or 11: - - Windows Native OpenSSH client (optional feature, needs admin permissions to enable and start the service for the first time). - - Pageant (part of the {{}}PuTTY suite of SSH tools{{}}). - - Mac & Linux - - the built-in ssh-agent should work. - -Notes: - - 1. MobaXterm's own "MobAgent" is NOT compatible for use as an agent, but MobaXterm itself can be configured to use Pageant as an external agent. Your environment may also depend on the agent for other applications. - 2. PuTTY users will either need to create their new key in the PuTTYgen utility, taking care to select the equivalent options to the command below, or convert their new key into a PuTTY-format `*.ppk` file, for use with Pageant. - -## NX Configuration - -You will need to check that the settings in `~/.nx/config/player.cfg` match your choice of method. If you modify the file, make sure you do it with the NoMachine application **closed**, otherwise the file will get overwritten as the application closes. - -- using the **file-based method** on all platforms requires the default settings, which are: - ```xml -