From 0ecd2bb3a80b29d264287efdc6f33a0c93b903dc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Thu, 15 Feb 2024 17:15:15 +0100 Subject: [PATCH 01/26] update page for setting up a private stratum 1 --- docs/filesystem_layer/stratum1.md | 135 ++++++++++++------------------ 1 file changed, 54 insertions(+), 81 deletions(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 7c7551f8d..77779f7d5 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -1,44 +1,55 @@ # Setting up a Stratum 1 -Setting up a Stratum 1 involves the following steps: - -- set up the Stratum 1, preferably by running the Ansible playbook that we provide; -- request a Stratum 0 firewall exception for your Stratum 1 server; -- request a `.stratum1.cvmfs.eessi-infra.org` DNS entry; -- open a pull request to include the URL to your Stratum 1 in the EESSI configuration. - -The last two steps can be skipped if you want to host a "private" Stratum 1 for your site. - +The EESSI project provides a number of geographically distributed public Stratum 1 servers that you can use to make EESSI available on your machine(s). +If you want to be better protected against network outages and increase the bandwith between your cluster nodes and the Stratum 1 servers, +you could consider setting up a local (private) Stratum 1 server that replicates the EESSI CVMFS repository. +This guarantees that you always have a full and up-to-date copy of the entire stack available in your local network. ## Requirements for a Stratum 1 The main requirements for a Stratum 1 server are a good network connection to the clients it is going to serve, -and sufficient disk space. For the EESSI repository, a few hundred gigabytes should suffice, but for production -environments at least 1 TB would be recommended. +and sufficient disk space. As the EESSI repository is constantly growing, make sure that the disk space can easily be extended if necessary. +Currently, we recommend to have at least 1 TB available. In terms of cores and memory, a machine with just a few (~4) cores and 4-8 GB of memory should suffice. -Various Linux distributions are supported, but we recommend one based on RHEL 7 or 8. +Various Linux distributions are supported, but we recommend one based on RHEL 8 or 9. + +Finally, make sure that ports 80 and 8000 are open to clients. -Finally, make sure that ports 80 (for the Apache web server) and 8000 are open. +## Configure the Stratum 1 -## Step 1: set up the Stratum 1 +Stratum 1 servers usually replicate from the Stratum 0 server. +In order to ensure the stability and security of the EESSI Stratum 0 server, it has a strict firewall, and only the EESSI-maintained public Stratum 1 servers are allowed to replicate from it. +However, EESSI provides a synchronisation server that can be used for setting up private Stratum 1 replica servers, and this is available at `http://aws-eu-west-s1-sync.eessi.science`. + +### Manual configuration + +In order to set up a Stratum 1 manually, you can make use of the instructions in the [Private Stratum 1 replica server](https://multixscale.github.io/cvmfs-tutorial-hpc-best-practices/access/stratum1/) +section of the MultiXscale tutorial ["Best Practices for CernVM-FS in HPC"](https://multixscale.github.io/cvmfs-tutorial-hpc-best-practices/). + +### Configuration using Ansible The recommended way for setting up an EESSI Stratum 1 is by running the Ansible playbook `stratum1.yml` from the [filesystem-layer repository on GitHub](https://github.com/EESSI/filesystem-layer). +For the commands in this section, we are assuming that you cloned this repository, and your working directory is `filesystem-layer`. -Installing a Stratum 1 requires a GEO API license key, which will be used to find the (geographically) closest Stratum 1 server for your client and proxies. -More information on how to (freely) obtain this key is available in the CVMFS documentation: https://cvmfs.readthedocs.io/en/stable/cpt-replica.html#geo-api-setup. - -You can put your license key in the local configuration file `inventory/local_site_specific_vars.yml`. +!!! note GEO API + Installing a Stratum 1 usually requires a GEO API license key, which will be used to find the (geographically) closest Stratum 1 server for your client and proxies. + However, for a private Stratum 1 this can be skipped, as clients should just connect to your local Stratum 1 by default. + + If you do want to set up the GEO API, you can find more information on how to (freely) obtain this key in the CVMFS documentation: https://cvmfs.readthedocs.io/en/stable/cpt-replica.html#geo-api-setup. + + You can put your license key in the local configuration file `inventory/local_site_specific_vars.yml`. -Furthermore, the Stratum 1 runs a Squid server. The template configuration file can be found at `templates/eessi_stratum1_squid.conf.j2`. -If you want to customize it, for instance for limiting the access to the Stratum 1, you can make your own version of this template file -and point to it by setting `local_stratum1_cvmfs_squid_conf_src` in `inventory/local_site_specific_vars.yml`. -See the comments in the example file for more details. +!!! note Squid reverse proxy + The Stratum 1 playbooks also installs and configures a Squid reverse proxy on the server. The template configuration file for Squid can be found at `templates/eessi_stratum1_squid.conf.j2`. + If you want to customize it, for instance for limiting the access to the Stratum 1, you can make your own version of this template file + and point to it by setting `local_stratum1_cvmfs_squid_conf_src` in `inventory/local_site_specific_vars.yml`. + See the comments in the example file for more details. -Start by installing Ansible: +Start by installing Ansible, e.g.: ```bash sudo yum install -y ansible @@ -47,58 +58,33 @@ sudo yum install -y ansible Then install Ansible roles for EESSI: ```bash -ansible-galaxy role install -r requirements.yml -p ./roles --force +ansible-galaxy role install -r ./requirements.yml -p ./roles --force ``` -Make sure you have enough space in `/srv` (on the Stratum 1) since the snapshot of the Stratum 0 -will end up there by default. To alter the directory where the snapshot gets copied to you can add -this variable in `inventory/host_vars/`: +Make sure you have enough space in `/srv` on the Stratum 1, since the snapshot of the repositories +will end up there by default. To alter the directory where the snapshots get stored you can add +the following variable in `inventory/host_vars/`: ```bash -cvmfs_srv_mount: /srv +cvmfs_srv_mount: /lots/of/space ``` -Make sure that you have added the hostname or IP address of your server to the -`inventory/hosts` file. Finally, install the Stratum 1 using one of the two following options. +Also make sure that you have added the hostname or IP address of your server to the +`inventory/hosts` file. -Option 1: +Finally, install the Stratum 1 using: ``` bash # -b to run as root, optionally use -K if a sudo password is required ansible-playbook -b [-K] -e @inventory/local_site_specific_vars.yml stratum1.yml ``` - -Option2: - -Create a ssh key pair and make sure the `ansible-host-keys.pub` is in the -`$HOME/.ssh/authorized_keys` file on your Stratum 1 server. - -```bash -ssh-keygen -b 2048 -t rsa -f ~/.ssh/ansible-host-keys -q -N "" -``` - -Then run the playbook: - -```bash -ansible-playbook -b --private-key ~/.ssh/ansible-host-keys -e @inventory/local_site_specific_vars.yml stratum1.yml -``` - Running the playbook will automatically make replicas of all the repositories defined in `group_vars/all.yml`. -## Step 2: request a firewall exception - -(This step is not implemented yet and can be skipped) - -You can request a firewall exception rule to be added for your Stratum 1 server by -[opening an issue on the GitHub page of the filesystem layer repository](https://github.com/EESSI/filesystem-layer/issues/new). - -Make sure to include the IP address of your server. - -## Step 3: Verification of the Stratum 1 +## Verification of the Stratum 1 -When the playbook has finished your Stratum 1 should be ready. In order to test your Stratum 1, even -without a client installed, you can use `curl`. +When the playbook has finished, your Stratum 1 should be ready. In order to test your Stratum 1, +even without a client installed, you can use `curl`: ```bash curl --head http:///cvmfs/software.eessi.io/.cvmfspublished @@ -115,25 +101,28 @@ The second time you run it, you should get a cache hit: ```bash X-Cache: HIT from - ``` -Example with the Norwegian Stratum 1: +Example with the EESSI Stratum 1 running in AWS: ```bash -curl --head http://bgo-no.stratum1.cvmfs.eessi-infra.org/cvmfs/software.eessi.io/.cvmfspublished +curl --head http://aws-eu-central-s1.eessi.science/cvmfs/software.eessi.io/.cvmfspublished ``` You can also test access to your Stratum 1 from a client, for which you will have to install the CVMFS [client](https://github.com/EESSI/filesystem-layer#clients). -Then run the following command to add your newly created Stratum 1 to the existing list of EESSI Stratum 1 servers by creating a local CVMFS configuration file: +Then run the following command to prepend your newly created Stratum 1 to the existing list of EESSI Stratum 1 servers by creating a local CVMFS configuration file: ```bash echo 'CVMFS_SERVER_URL="http:///cvmfs/@fqrn@;$CVMFS_SERVER_URL"' | sudo tee -a /etc/cvmfs/domain.d/eessi-hpc.org.local ``` -If this is the first time you set up the client you now run: +!!! note + By prepending your new Stratum 1 to the list of existing Stratum 1 servers, your clients should by default use the private Stratum 1. + In case of downtime of your private Stratum 1, they will also still be able to make use of the public EESSI Stratum 1 servers. + +If this is the first time you set up the client, you now run: ```bash sudo cvmfs_config setup @@ -151,24 +140,8 @@ Finally, verify that the client connects to your new Stratum 1 by running: cvmfs_config stat -v software.eessi.io ``` -Assuming that your new Stratum 1 is the geographically closest one to your client, this should return: +Assuming that your new Stratum 1 is working properly, this should return something like: ```bash Connection: http:///cvmfs/software.eessi.io through proxy DIRECT (online) ``` - - -## Step 4: request an EESSI DNS name - -In order to keep the configuration clean and easy, all the EESSI Stratum 1 servers have a DNS name -`.stratum1.cvmfs.eessi-infra.org`, where `` is often a short name or -abbreviation followed by the country code (e.g. `rug-nl` or `bgo-no`). You can request this for -your Stratum 1 by mentioning this in the issue that you created in Step 2, or by opening another -issue. - -## Step 5: include your Stratum 1 in the EESSI configuration - -If you want to include your Stratum 1 in the EESSI configuration, i.e. allow any (nearby) client to be able to use it, -you can open a pull request with updated configuration files. You will only have to add the URL to your Stratum 1 to the -`urls` list of the `eessi_cvmfs_server_urls` variable in the -[`all.yml` file](https://github.com/EESSI/filesystem-layer/blob/main/inventory/group_vars/all.yml). From d2cbf8425250c938de3ff4a0767bf72b1061d024 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Thu, 15 Feb 2024 17:18:21 +0100 Subject: [PATCH 02/26] fix typo in bandwidth --- docs/filesystem_layer/stratum1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 77779f7d5..edcb23973 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -1,7 +1,7 @@ # Setting up a Stratum 1 The EESSI project provides a number of geographically distributed public Stratum 1 servers that you can use to make EESSI available on your machine(s). -If you want to be better protected against network outages and increase the bandwith between your cluster nodes and the Stratum 1 servers, +If you want to be better protected against network outages and increase the bandwidth between your cluster nodes and the Stratum 1 servers, you could consider setting up a local (private) Stratum 1 server that replicates the EESSI CVMFS repository. This guarantees that you always have a full and up-to-date copy of the entire stack available in your local network. From 12900ff433f790cdfa8bca04bee664e00be0f118 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Thu, 15 Feb 2024 17:21:37 +0100 Subject: [PATCH 03/26] add sentence about SSH keys and sudo --- docs/filesystem_layer/stratum1.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index edcb23973..a1b177cb5 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -70,7 +70,8 @@ cvmfs_srv_mount: /lots/of/space ``` Also make sure that you have added the hostname or IP address of your server to the -`inventory/hosts` file. +`inventory/hosts` file, that you are able to log in to the server from the machine that is going to run the playbook +(preferably using an SSH key), and that you can use `sudo`. Finally, install the Stratum 1 using: From 6bbc5a0fa6a7cf1da4e33ed103d5394cc3a36541 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 16 Feb 2024 10:13:03 +0100 Subject: [PATCH 04/26] make site-specific vars file optional --- docs/filesystem_layer/stratum1.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index a1b177cb5..81c35bc29 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -76,8 +76,8 @@ Also make sure that you have added the hostname or IP address of your server to Finally, install the Stratum 1 using: ``` bash -# -b to run as root, optionally use -K if a sudo password is required -ansible-playbook -b [-K] -e @inventory/local_site_specific_vars.yml stratum1.yml +# -b to run as root, optionally use -K if a sudo password is required, and optionally include your site-specific variables +ansible-playbook -b [-K] [-e @inventory/local_site_specific_vars.yml] stratum1.yml ``` Running the playbook will automatically make replicas of all the repositories defined in `group_vars/all.yml`. From 0ff336a351c027e42994f4f1406ab28d783953b5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 16 Feb 2024 16:30:16 +0100 Subject: [PATCH 05/26] add warning about IPS --- docs/filesystem_layer/stratum1.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 81c35bc29..e888bb35b 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -24,6 +24,19 @@ Stratum 1 servers usually replicate from the Stratum 0 server. In order to ensure the stability and security of the EESSI Stratum 0 server, it has a strict firewall, and only the EESSI-maintained public Stratum 1 servers are allowed to replicate from it. However, EESSI provides a synchronisation server that can be used for setting up private Stratum 1 replica servers, and this is available at `http://aws-eu-west-s1-sync.eessi.science`. +!!! warn Potential issues with intrusion prevention systems + In the past we have seen a few occurrences of data transfer issues when files were being pulled in by or from a Stratum 1 server. + In such cases the `cvmfs_server snapshot` command, used for synchronizing the Stratum 1, may break with errors like `failed to download `. + Trying to manually download the mentioned file with `curl` will also not work, and result in errors like: + ``` + curl: (56) Recv failure: Connection reset by peer + ``` + In all cases this was due to an intrusion prevention system scanning the associated network, and hence scanning all files going in or out of the Stratum 1. + Though it was a false-positive in all cases, this breaks the synchronization procedure of your Stratum 1. + If this is the case, you can try switching to HTTPS by using `https://aws-eu-west-s1-sync.eessi.science` for synchronizing your Stratum 1. + Even though there is no advantage for CVMFS itself in using HTTPS (it has built-in mechasnims for ensuring the integrity of the data), + this will prevent the described issues, as the intrusion prevention system will not be able to inspect the encrypted data. + ### Manual configuration In order to set up a Stratum 1 manually, you can make use of the instructions in the [Private Stratum 1 replica server](https://multixscale.github.io/cvmfs-tutorial-hpc-best-practices/access/stratum1/) From f830adb81c0ffecaaf216dec9d62bdb9f86d669e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 16 Feb 2024 17:41:41 +0100 Subject: [PATCH 06/26] added sentence about downside of https --- docs/filesystem_layer/stratum1.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index e888bb35b..7da90ad68 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -36,6 +36,7 @@ However, EESSI provides a synchronisation server that can be used for setting up If this is the case, you can try switching to HTTPS by using `https://aws-eu-west-s1-sync.eessi.science` for synchronizing your Stratum 1. Even though there is no advantage for CVMFS itself in using HTTPS (it has built-in mechasnims for ensuring the integrity of the data), this will prevent the described issues, as the intrusion prevention system will not be able to inspect the encrypted data. + As HTTPS does introduce some overhead due to the encryption/decryption, it is still recommended to use HTTP as default. ### Manual configuration From b018fb4a4ccdf781064658e3b9df9f1beed75d1b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 16 Feb 2024 17:49:12 +0100 Subject: [PATCH 07/26] change headers of subsections --- docs/filesystem_layer/stratum1.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 7da90ad68..e70b434c5 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -96,7 +96,7 @@ ansible-playbook -b [-K] [-e @inventory/local_site_specific_vars.yml] stratum1.y Running the playbook will automatically make replicas of all the repositories defined in `group_vars/all.yml`. -## Verification of the Stratum 1 +### Verification of the Stratum 1 using `curl` When the playbook has finished, your Stratum 1 should be ready. In order to test your Stratum 1, even without a client installed, you can use `curl`: @@ -124,6 +124,8 @@ Example with the EESSI Stratum 1 running in AWS: curl --head http://aws-eu-central-s1.eessi.science/cvmfs/software.eessi.io/.cvmfspublished ``` +### Verification of the Stratum 1 using a CVMFS client + You can also test access to your Stratum 1 from a client, for which you will have to install the CVMFS [client](https://github.com/EESSI/filesystem-layer#clients). From a4851b048192da5cc7e711769a5c41607ab71e4d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 12 Apr 2024 14:20:59 +0200 Subject: [PATCH 08/26] add recommendation for having squid proxies --- docs/filesystem_layer/stratum1.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index e70b434c5..6418dce19 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -1,8 +1,9 @@ # Setting up a Stratum 1 The EESSI project provides a number of geographically distributed public Stratum 1 servers that you can use to make EESSI available on your machine(s). -If you want to be better protected against network outages and increase the bandwidth between your cluster nodes and the Stratum 1 servers, -you could consider setting up a local (private) Stratum 1 server that replicates the EESSI CVMFS repository. +It is always recommended to have a local caching layer consisting of a few Squid proxies. +If you want to be even better protected against network outages and increase the bandwidth between your cluster nodes and the Stratum 1 servers, +you could also consider setting up a local (private) Stratum 1 server that replicates the EESSI CVMFS repository. This guarantees that you always have a full and up-to-date copy of the entire stack available in your local network. ## Requirements for a Stratum 1 From d555b4bfd7b3756a57f250b7cd62ef919f5338ad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 12 Apr 2024 14:21:53 +0200 Subject: [PATCH 09/26] fix typo in mechanisms --- docs/filesystem_layer/stratum1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 6418dce19..dc83b6456 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -35,7 +35,7 @@ However, EESSI provides a synchronisation server that can be used for setting up In all cases this was due to an intrusion prevention system scanning the associated network, and hence scanning all files going in or out of the Stratum 1. Though it was a false-positive in all cases, this breaks the synchronization procedure of your Stratum 1. If this is the case, you can try switching to HTTPS by using `https://aws-eu-west-s1-sync.eessi.science` for synchronizing your Stratum 1. - Even though there is no advantage for CVMFS itself in using HTTPS (it has built-in mechasnims for ensuring the integrity of the data), + Even though there is no advantage for CVMFS itself in using HTTPS (it has built-in mechanisms for ensuring the integrity of the data), this will prevent the described issues, as the intrusion prevention system will not be able to inspect the encrypted data. As HTTPS does introduce some overhead due to the encryption/decryption, it is still recommended to use HTTP as default. From 2b32fa913c96592307fe89062740dc6f6f793ed0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 12 Apr 2024 14:24:18 +0200 Subject: [PATCH 10/26] reword sentence about replicating from stratum 0 a bit --- docs/filesystem_layer/stratum1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index dc83b6456..65874b250 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -21,7 +21,7 @@ Finally, make sure that ports 80 and 8000 are open to clients. ## Configure the Stratum 1 -Stratum 1 servers usually replicate from the Stratum 0 server. +Stratum 1 servers have to synchronize the contents of their CVMFS repositories regularly, and usually they replicate from a CVMFS Stratum 0 server. In order to ensure the stability and security of the EESSI Stratum 0 server, it has a strict firewall, and only the EESSI-maintained public Stratum 1 servers are allowed to replicate from it. However, EESSI provides a synchronisation server that can be used for setting up private Stratum 1 replica servers, and this is available at `http://aws-eu-west-s1-sync.eessi.science`. From ba514972104e7d80415a48fd246295b7276aa4a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 12 Apr 2024 14:27:50 +0200 Subject: [PATCH 11/26] discourage https --- docs/filesystem_layer/stratum1.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 65874b250..c35902a00 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -37,7 +37,8 @@ However, EESSI provides a synchronisation server that can be used for setting up If this is the case, you can try switching to HTTPS by using `https://aws-eu-west-s1-sync.eessi.science` for synchronizing your Stratum 1. Even though there is no advantage for CVMFS itself in using HTTPS (it has built-in mechanisms for ensuring the integrity of the data), this will prevent the described issues, as the intrusion prevention system will not be able to inspect the encrypted data. - As HTTPS does introduce some overhead due to the encryption/decryption, it is still recommended to use HTTP as default. + Howver, not only does HTTPS introduce some overhead due to the encryption/decryption, it also makes caching in forward proxies impossible. + Therefore, it is strongly discouraged to use HTTPS as default. ### Manual configuration From 40668b1d66756c9513b193edff95b1b4fca88cfc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 12 Apr 2024 14:30:54 +0200 Subject: [PATCH 12/26] extend paragraph about geo api, instructions for disabling it on the clients --- docs/filesystem_layer/stratum1.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index c35902a00..b739d72c4 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -53,7 +53,8 @@ For the commands in this section, we are assuming that you cloned this repositor !!! note GEO API Installing a Stratum 1 usually requires a GEO API license key, which will be used to find the (geographically) closest Stratum 1 server for your client and proxies. - However, for a private Stratum 1 this can be skipped, as clients should just connect to your local Stratum 1 by default. + However, for a private Stratum 1 this can be skipped, and you can disable the use of the GEO API in the configuration of your clients by setting `CVMFS_USE_GEOAPI=no`. + In this case, they will just connect to your local Stratum 1 by default. If you do want to set up the GEO API, you can find more information on how to (freely) obtain this key in the CVMFS documentation: https://cvmfs.readthedocs.io/en/stable/cpt-replica.html#geo-api-setup. From 7c6742ce8c88621c297a1adc565b9c72b111336d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 12 Apr 2024 14:31:32 +0200 Subject: [PATCH 13/26] remove note about Squid proxy on Stratum 1, as it's now disabled by default in the playbook --- docs/filesystem_layer/stratum1.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index b739d72c4..11933b2d0 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -60,12 +60,6 @@ For the commands in this section, we are assuming that you cloned this repositor You can put your license key in the local configuration file `inventory/local_site_specific_vars.yml`. -!!! note Squid reverse proxy - The Stratum 1 playbooks also installs and configures a Squid reverse proxy on the server. The template configuration file for Squid can be found at `templates/eessi_stratum1_squid.conf.j2`. - If you want to customize it, for instance for limiting the access to the Stratum 1, you can make your own version of this template file - and point to it by setting `local_stratum1_cvmfs_squid_conf_src` in `inventory/local_site_specific_vars.yml`. - See the comments in the example file for more details. - Start by installing Ansible, e.g.: ```bash From 0fac17c7a0f79bd95c2854fc54344474074176a7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 12 Apr 2024 14:33:15 +0200 Subject: [PATCH 14/26] remove cache hit example --- docs/filesystem_layer/stratum1.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 11933b2d0..f197f06d8 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -101,18 +101,12 @@ even without a client installed, you can use `curl`: ```bash curl --head http:///cvmfs/software.eessi.io/.cvmfspublished ``` -This should return: +This should return something like: ```bash HTTP/1.1 200 OK ... -X-Cache: MISS from -``` - -The second time you run it, you should get a cache hit: - -```bash -X-Cache: HIT from +Content-Type: application/x-cvmfs ``` Example with the EESSI Stratum 1 running in AWS: From 93b1c34c001027f9d20666b5a16f28f8624b8b83 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 12 Apr 2024 14:33:44 +0200 Subject: [PATCH 15/26] use eessi.io instead of eessi-hpc.org --- docs/filesystem_layer/stratum1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index f197f06d8..02cbb80f5 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -123,7 +123,7 @@ You can also test access to your Stratum 1 from a client, for which you will hav Then run the following command to prepend your newly created Stratum 1 to the existing list of EESSI Stratum 1 servers by creating a local CVMFS configuration file: ```bash -echo 'CVMFS_SERVER_URL="http:///cvmfs/@fqrn@;$CVMFS_SERVER_URL"' | sudo tee -a /etc/cvmfs/domain.d/eessi-hpc.org.local +echo 'CVMFS_SERVER_URL="http:///cvmfs/@fqrn@;$CVMFS_SERVER_URL"' | sudo tee -a /etc/cvmfs/domain.d/eessi.io.local ``` !!! note From 46a2f2ff4e5f194348be7de3b67e38f00c9bb2ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Fri, 12 Apr 2024 14:46:22 +0200 Subject: [PATCH 16/26] fix typo in however --- docs/filesystem_layer/stratum1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 02cbb80f5..93922e33e 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -37,7 +37,7 @@ However, EESSI provides a synchronisation server that can be used for setting up If this is the case, you can try switching to HTTPS by using `https://aws-eu-west-s1-sync.eessi.science` for synchronizing your Stratum 1. Even though there is no advantage for CVMFS itself in using HTTPS (it has built-in mechanisms for ensuring the integrity of the data), this will prevent the described issues, as the intrusion prevention system will not be able to inspect the encrypted data. - Howver, not only does HTTPS introduce some overhead due to the encryption/decryption, it also makes caching in forward proxies impossible. + However, not only does HTTPS introduce some overhead due to the encryption/decryption, it also makes caching in forward proxies impossible. Therefore, it is strongly discouraged to use HTTPS as default. ### Manual configuration From 59e2f8d9d07948a57c5d93a6d7df869ff4c08efa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 14:19:06 +0200 Subject: [PATCH 17/26] remove -p ./roles in ansible-galaxy command --- docs/filesystem_layer/stratum1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 93922e33e..9564dfa64 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -69,7 +69,7 @@ sudo yum install -y ansible Then install Ansible roles for EESSI: ```bash -ansible-galaxy role install -r ./requirements.yml -p ./roles --force +ansible-galaxy role install -r ./requirements.yml --force ``` Make sure you have enough space in `/srv` on the Stratum 1, since the snapshot of the repositories From 98253419a4e07761f52b60dea3e6e22079cbd838 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 14:44:43 +0200 Subject: [PATCH 18/26] add link to stratum 1 page --- docs/getting_access/native_installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_access/native_installation.md b/docs/getting_access/native_installation.md index b5cd197f0..0d6c31904 100644 --- a/docs/getting_access/native_installation.md +++ b/docs/getting_access/native_installation.md @@ -69,7 +69,7 @@ The good news is that all of this only requires a handful commands :astonished: This is good enough for an individual client, or for testing purposes, but for a production-quality setup you should also set up a Squid proxy cache. - For large-scale systems, like an HPC cluster, you should also consider setting up your own CernVM-FS Stratum-1 mirror server. + For large-scale systems, like an HPC cluster, you should also consider [setting up your own CernVM-FS Stratum-1 mirror server](filesystem_layer/stratum1.md). For more details on this, please refer to the [*Stratum 1 and proxies section* of the CernVM-FS tutorial](https://cvmfs-contrib.github.io/cvmfs-tutorial-2021/03_stratum1_proxies/). From c6adda4b5403f19cd264cebd5d08504da4b1c6e8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 14:58:13 +0200 Subject: [PATCH 19/26] add section about proxy configuration --- docs/getting_access/native_installation.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/getting_access/native_installation.md b/docs/getting_access/native_installation.md index 0d6c31904..71863cba8 100644 --- a/docs/getting_access/native_installation.md +++ b/docs/getting_access/native_installation.md @@ -73,3 +73,17 @@ The good news is that all of this only requires a handful commands :astonished: For more details on this, please refer to the [*Stratum 1 and proxies section* of the CernVM-FS tutorial](https://cvmfs-contrib.github.io/cvmfs-tutorial-2021/03_stratum1_proxies/). + +## Configuring your client to use a Squid proxy + +If you have set up one or more Squid proxies, you will have to add them to your CernVM-FS client configuration. +This can be done by removing `CVMFS_CLIENT_PROFILE="single"` from `/etc/cvmfs/default.local`, and add the following line: + +``` +CVMFS_HTTP_PROXY="http://ip-of-your-1st-proxy:port|http://ip-of-your-2nd-proxy:port" +``` + +In this case, both proxies are equally preferable. +More advanced use cases can be found in [the CernVM-FS documentation](https://cvmfs.readthedocs.io/en/stable/cpt-configure.html#proxy-list-examples). + + From 5ac0a4872a54d2a6beb39f2ddbf0fb69cb8eb8f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 15:13:10 +0200 Subject: [PATCH 20/26] add section about configuring an additional stratum 1 --- docs/getting_access/native_installation.md | 30 +++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/docs/getting_access/native_installation.md b/docs/getting_access/native_installation.md index 71863cba8..59548ec54 100644 --- a/docs/getting_access/native_installation.md +++ b/docs/getting_access/native_installation.md @@ -69,7 +69,7 @@ The good news is that all of this only requires a handful commands :astonished: This is good enough for an individual client, or for testing purposes, but for a production-quality setup you should also set up a Squid proxy cache. - For large-scale systems, like an HPC cluster, you should also consider [setting up your own CernVM-FS Stratum-1 mirror server](filesystem_layer/stratum1.md). + For large-scale systems, like an HPC cluster, you should also consider [setting up your own CernVM-FS Stratum 1 mirror server](filesystem_layer/stratum1.md). For more details on this, please refer to the [*Stratum 1 and proxies section* of the CernVM-FS tutorial](https://cvmfs-contrib.github.io/cvmfs-tutorial-2021/03_stratum1_proxies/). @@ -86,4 +86,32 @@ CVMFS_HTTP_PROXY="http://ip-of-your-1st-proxy:port|http://ip-of-your-2nd-proxy:p In this case, both proxies are equally preferable. More advanced use cases can be found in [the CernVM-FS documentation](https://cvmfs.readthedocs.io/en/stable/cpt-configure.html#proxy-list-examples). +## Configuring your client to use a private Stratum 1 mirror server + +If you have set up your own Stratum 1 mirror server that replicates the EESSI CernVM-FS repositories, +you can instruct your CernVM-FS client(s) to use it by prepending your newly created Stratum 1 to the existing list of EESSI Stratum 1 servers by creating a local CVMFS configuration file for the EESSI domain: + +```bash +echo 'CVMFS_SERVER_URL="http:///cvmfs/@fqrn@;$CVMFS_SERVER_URL"' | sudo tee -a /etc/cvmfs/domain.d/eessi.io.local +``` + +!!! note + By prepending your new Stratum 1 to the list of existing Stratum 1 servers, your clients should by default use the private Stratum 1. + In case of downtime of your private Stratum 1, they will also still be able to make use of the public EESSI Stratum 1 servers. + + +## Applying changes in the CernVM-FS client configuration files + +After you have made any changes to the CernVM-FS client configuration, you will have to apply them. +If this is the first time you set up the client, you can simply run: + +```bash +sudo cvmfs_config setup +``` + +If you already had configured the client before, you can reload the configuration for the EESSI repository (or, similarly, for any other repository) using: + +```bash +sudo cvmfs_config reload -c software.eessi.io +``` From 52bbfd7da71e3cb802e37d0c6d0e2a725bebec16 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 15:13:25 +0200 Subject: [PATCH 21/26] move client config part to native installation page --- docs/filesystem_layer/stratum1.md | 29 ++++------------------------- 1 file changed, 4 insertions(+), 25 deletions(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 9564dfa64..3aa3d4a86 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -117,32 +117,11 @@ curl --head http://aws-eu-central-s1.eessi.science/cvmfs/software.eessi.io/.cvmf ### Verification of the Stratum 1 using a CVMFS client -You can also test access to your Stratum 1 from a client, for which you will have to install the CVMFS -[client](https://github.com/EESSI/filesystem-layer#clients). +You can, of course, also test access to your Stratum 1 from a client. +This requires you to install a CernVM-FS client and add the Stratum 1 to the client configuration; +this is explained in more detail on the [native installation page](getting_access/native_installation.md). -Then run the following command to prepend your newly created Stratum 1 to the existing list of EESSI Stratum 1 servers by creating a local CVMFS configuration file: - -```bash -echo 'CVMFS_SERVER_URL="http:///cvmfs/@fqrn@;$CVMFS_SERVER_URL"' | sudo tee -a /etc/cvmfs/domain.d/eessi.io.local -``` - -!!! note - By prepending your new Stratum 1 to the list of existing Stratum 1 servers, your clients should by default use the private Stratum 1. - In case of downtime of your private Stratum 1, they will also still be able to make use of the public EESSI Stratum 1 servers. - -If this is the first time you set up the client, you now run: - -```bash -sudo cvmfs_config setup -``` - -If you already had configured the client before, you can simply reload the config: - -```bash -sudo cvmfs_config reload -c software.eessi.io -``` - -Finally, verify that the client connects to your new Stratum 1 by running: +Then verify that the client connects to your new Stratum 1 by running: ```bash cvmfs_config stat -v software.eessi.io From d8bb91e88dcebb74de05ad698f61853facbdc2c0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 15:17:02 +0200 Subject: [PATCH 22/26] fix link --- docs/filesystem_layer/stratum1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index 3aa3d4a86..dfb2e8b61 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -119,7 +119,7 @@ curl --head http://aws-eu-central-s1.eessi.science/cvmfs/software.eessi.io/.cvmf You can, of course, also test access to your Stratum 1 from a client. This requires you to install a CernVM-FS client and add the Stratum 1 to the client configuration; -this is explained in more detail on the [native installation page](getting_access/native_installation.md). +this is explained in more detail on the [native installation page](../getting_access/native_installation.md). Then verify that the client connects to your new Stratum 1 by running: From 47e3824a230c307e3061b63db566fc8f3aa907ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 15:17:21 +0200 Subject: [PATCH 23/26] fix link --- docs/getting_access/native_installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_access/native_installation.md b/docs/getting_access/native_installation.md index 59548ec54..cb173480d 100644 --- a/docs/getting_access/native_installation.md +++ b/docs/getting_access/native_installation.md @@ -69,7 +69,7 @@ The good news is that all of this only requires a handful commands :astonished: This is good enough for an individual client, or for testing purposes, but for a production-quality setup you should also set up a Squid proxy cache. - For large-scale systems, like an HPC cluster, you should also consider [setting up your own CernVM-FS Stratum 1 mirror server](filesystem_layer/stratum1.md). + For large-scale systems, like an HPC cluster, you should also consider [setting up your own CernVM-FS Stratum 1 mirror server](../filesystem_layer/stratum1.md). For more details on this, please refer to the [*Stratum 1 and proxies section* of the CernVM-FS tutorial](https://cvmfs-contrib.github.io/cvmfs-tutorial-2021/03_stratum1_proxies/). From 8c8ce18ae3f8e111209cd2da84eb9011dc072467 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 16:26:31 +0200 Subject: [PATCH 24/26] correct paragraph about /srv --- docs/filesystem_layer/stratum1.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index dfb2e8b61..ebbe5b047 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -72,12 +72,20 @@ Then install Ansible roles for EESSI: ansible-galaxy role install -r ./requirements.yml --force ``` -Make sure you have enough space in `/srv` on the Stratum 1, since the snapshot of the repositories -will end up there by default. To alter the directory where the snapshots get stored you can add -the following variable in `inventory/host_vars/`: +Make sure you have enough space in `/srv` on the Stratum 1, since the snapshots of the repositories +will end up there by default. To alter the directory where the snapshots get stored you can manually +create a symlink before running the playbook: +```bash +sudo ln -s /lots/of/space/cvmfs /srv/cvmfs +``` + +Alternatively, you can instruct the Ansible role to create a file system (e.g. ext4) on a given storage device +and mount it at `/srv` by adding the following variable +in `inventory/host_vars/`: ```bash -cvmfs_srv_mount: /lots/of/space +cvmfs_srv_device: /dev/somedevice +cvmfs_srv_fstype: ext4 ``` Also make sure that you have added the hostname or IP address of your server to the From 020139c4e414ee9e0c208879082ad6dde305520e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 16:33:16 +0200 Subject: [PATCH 25/26] remove instructions for mounting an additional file system --- docs/filesystem_layer/stratum1.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docs/filesystem_layer/stratum1.md b/docs/filesystem_layer/stratum1.md index ebbe5b047..e00662525 100644 --- a/docs/filesystem_layer/stratum1.md +++ b/docs/filesystem_layer/stratum1.md @@ -79,15 +79,6 @@ create a symlink before running the playbook: sudo ln -s /lots/of/space/cvmfs /srv/cvmfs ``` -Alternatively, you can instruct the Ansible role to create a file system (e.g. ext4) on a given storage device -and mount it at `/srv` by adding the following variable -in `inventory/host_vars/`: - -```bash -cvmfs_srv_device: /dev/somedevice -cvmfs_srv_fstype: ext4 -``` - Also make sure that you have added the hostname or IP address of your server to the `inventory/hosts` file, that you are able to log in to the server from the machine that is going to run the playbook (preferably using an SSH key), and that you can use `sudo`. From 9e97d4fc36dc185e75ef9dc0f90f7460cb536f20 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bob=20Dr=C3=B6ge?= Date: Tue, 4 Jun 2024 17:00:47 +0200 Subject: [PATCH 26/26] remove note, rearrange the sections, add section for larger systems --- docs/getting_access/native_installation.md | 28 ++++++++++++---------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/getting_access/native_installation.md b/docs/getting_access/native_installation.md index cb173480d..e35b3df41 100644 --- a/docs/getting_access/native_installation.md +++ b/docs/getting_access/native_installation.md @@ -1,5 +1,7 @@ # Native installation +## Installation for single clients + Setting up native access to EESSI, that is a system-wide deployment that does not require workarounds like [using a container](eessi_container.md), requires the installation and configuration of [CernVM-FS](https://cernvm.cern.ch/fs). @@ -62,19 +64,21 @@ The good news is that all of this only requires a handful commands :astonished: sudo cvmfs_config setup ``` -!!! note - - :point_up: The commands above only cover the basic installation of EESSI. - - This is good enough for an individual client, or for testing purposes, - but for a production-quality setup you should also set up a Squid proxy cache. +## Installation for larger systems (e.g. clusters) - For large-scale systems, like an HPC cluster, you should also consider [setting up your own CernVM-FS Stratum 1 mirror server](../filesystem_layer/stratum1.md). +When using CernVM-FS on a larger number of local clients, e.g. on a HPC cluster or set of workstations, +it is very strongly recommended to at least set up some Squid proxies close to your clients. +These Squid proxies will be used to cache content that was recently accessed by your clients, +which reduces the load on the Stratum 1 servers and reduces the latency for your clients. +As a rule of thumb, you should use about one proxy per 500 clients, and have a minimum of two. +Instructions for setting up a Squid proxy can be found in the [CernVM-FS documentation](https://cvmfs.readthedocs.io/en/stable/cpt-squid.html) and +in the [CernVM-FS tutorial](https://cvmfs-contrib.github.io/cvmfs-tutorial-2021/03_stratum1_proxies/#32-setting-up-a-proxy). - For more details on this, please refer to the - [*Stratum 1 and proxies section* of the CernVM-FS tutorial](https://cvmfs-contrib.github.io/cvmfs-tutorial-2021/03_stratum1_proxies/). +Additionally, setting up a private Stratum 1, which will make a full copy of the repository, + can be beneficial to improve the latency and bandwidth even further, and to be better protected against network outages. +Instructions for setting up your own EESSI Stratum 1 can be found in [setting up your own CernVM-FS Stratum 1 mirror server](../filesystem_layer/stratum1.md). -## Configuring your client to use a Squid proxy +### Configuring your client to use a Squid proxy If you have set up one or more Squid proxies, you will have to add them to your CernVM-FS client configuration. This can be done by removing `CVMFS_CLIENT_PROFILE="single"` from `/etc/cvmfs/default.local`, and add the following line: @@ -86,7 +90,7 @@ CVMFS_HTTP_PROXY="http://ip-of-your-1st-proxy:port|http://ip-of-your-2nd-proxy:p In this case, both proxies are equally preferable. More advanced use cases can be found in [the CernVM-FS documentation](https://cvmfs.readthedocs.io/en/stable/cpt-configure.html#proxy-list-examples). -## Configuring your client to use a private Stratum 1 mirror server +### Configuring your client to use a private Stratum 1 mirror server If you have set up your own Stratum 1 mirror server that replicates the EESSI CernVM-FS repositories, you can instruct your CernVM-FS client(s) to use it by prepending your newly created Stratum 1 to the existing list of EESSI Stratum 1 servers by creating a local CVMFS configuration file for the EESSI domain: @@ -100,7 +104,7 @@ echo 'CVMFS_SERVER_URL="http:///cvmfs/@fqrn@;$CVMFS_ In case of downtime of your private Stratum 1, they will also still be able to make use of the public EESSI Stratum 1 servers. -## Applying changes in the CernVM-FS client configuration files +### Applying changes in the CernVM-FS client configuration files After you have made any changes to the CernVM-FS client configuration, you will have to apply them. If this is the first time you set up the client, you can simply run: