From faf00c6f0f05848db4d4fe02befa496afd61f5d9 Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Wed, 2 Aug 2023 16:07:40 -0700 Subject: [PATCH 01/10] docs: Note on the "upgrade Nextstrain CLI" page how to update the runtime I'd like to make this distinction even clearer in the future, but it'll have to wait for another doc push. For now, at least provide a pointer to anyone coming to this page wanting to update the runtime. --- doc/upgrading.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/upgrading.rst b/doc/upgrading.rst index 76439a9a..327ac1bb 100644 --- a/doc/upgrading.rst +++ b/doc/upgrading.rst @@ -4,7 +4,8 @@ Upgrading This page describes how to upgrade the Nextstrain CLI--the ``nextstrain`` command--itself, without also upgrading the entire Nextstrain -:term:`docs:runtime`. +:term:`docs:runtime`. (To upgrade your Nextstrain runtime, use the +:doc:`/commands/update` command.) The way to upgrade depends on what kind of Nextstrain CLI installation you have (i.e. how it was first installed), so running ``nextstrain check-setup``: From 9e94891dcc131981ae74c6ecd3415240a47dfe14 Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Wed, 25 Jan 2023 13:41:27 -0800 Subject: [PATCH 02/10] docs: Document runtimes more comprehensively and centrally MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Trying to provide a more useful resource to point people to than a smattering of scattered doc snippets. There's still some places this could be better cross-referenced (e.g. in --help output) and more that could be said about runtimes in general and each runtime specifically… but that can all come in due time. I needed to find a stopping point for this work _somewhere_. A big gain here is that now there's clear places to attach additional documentation in the future. Like the recent switch from dynamically- to statically-generated rST for command documentation, it may also sooner-than-later make sense to ditch the "automodule" directive and do something static instead. Resolves . --- README.md | 8 +- doc/commands/build.rst | 2 +- doc/commands/view.rst | 2 +- doc/glossary.rst | 15 +- doc/index.rst | 19 ++- doc/installation.rst | 153 ++------------------ doc/runtimes/ambient.rst | 6 + doc/runtimes/aws-batch.rst | 6 + doc/runtimes/comparison.csv | 6 + doc/runtimes/conda.rst | 6 + doc/runtimes/docker.rst | 6 + doc/runtimes/index.rst | 97 +++++++++++++ doc/runtimes/singularity.rst | 6 + nextstrain/cli/runner/ambient.py | 28 +++- nextstrain/cli/runner/aws_batch/__init__.py | 49 +++++++ nextstrain/cli/runner/conda.py | 22 +++ nextstrain/cli/runner/docker.py | 48 ++++++ nextstrain/cli/runner/singularity.py | 54 ++++++- 18 files changed, 366 insertions(+), 167 deletions(-) create mode 100644 doc/runtimes/ambient.rst create mode 100644 doc/runtimes/aws-batch.rst create mode 100644 doc/runtimes/comparison.csv create mode 100644 doc/runtimes/conda.rst create mode 100644 doc/runtimes/docker.rst create mode 100644 doc/runtimes/index.rst create mode 100644 doc/runtimes/singularity.rst diff --git a/README.md b/README.md index 517d3448..55045a62 100644 --- a/README.md +++ b/README.md @@ -18,10 +18,10 @@ email to . [Augur]: https://docs.nextstrain.org/projects/augur/ [Auspice]: https://docs.nextstrain.org/projects/auspice/ -[Docker]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#docker -[Conda]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#conda -[Singularity]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#singularity -[AWS Batch]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#aws-batch +[Docker]: https://docs.nextstrain.org/projects/cli/page/runtimes/docker/ +[Conda]: https://docs.nextstrain.org/projects/cli/page/runtimes/conda/ +[Singularity]: https://docs.nextstrain.org/projects/cli/page/runtimes/singularity/ +[AWS Batch]: https://docs.nextstrain.org/projects/cli/page/runtimes/aws-batch/ [documentation]: https://docs.nextstrain.org/projects/cli/ [development docs]: https://docs.nextstrain.org/projects/cli/page/development/ [open an issue]: https://github.com/nextstrain/cli/issues/new diff --git a/doc/commands/build.rst b/doc/commands/build.rst index bf3a48f8..0a217bbe 100644 --- a/doc/commands/build.rst +++ b/doc/commands/build.rst @@ -120,7 +120,7 @@ default is not suitable. .. option:: --ambient - Run commands in the ambient environment, outside of any container image. + Run commands in the ambient environment, outside of any container image or managed environment. .. option:: --aws-batch diff --git a/doc/commands/view.rst b/doc/commands/view.rst index c91508ad..d15dd3fe 100644 --- a/doc/commands/view.rst +++ b/doc/commands/view.rst @@ -115,7 +115,7 @@ default is not suitable. .. option:: --ambient - Run commands in the ambient environment, outside of any container image. + Run commands in the ambient environment, outside of any container image or managed environment. .. option:: --conda diff --git a/doc/glossary.rst b/doc/glossary.rst index bd87e5c1..7bf2c36e 100644 --- a/doc/glossary.rst +++ b/doc/glossary.rst @@ -36,17 +36,4 @@ documentation. environment) in which Nextstrain CLI expects to find and execute other Nextstrain programs. - Current runtimes: - - - Docker with the `nextstrain/base image `_ - - Conda with the `nextstrain-base meta-package `__ - - Ambient - - AWS Batch with the `nextstrain/base image`_ - - Each runtime provides specific versions of Nextstrain's software - components, like Augur and Auspice. - - Runtimes are managed (maintained, tested, versioned, released) by the - Nextstrain team, except for the ambient runtime. The ambient runtime - is special in that it's whatever computing environment in which - Nextstrain CLI itself is running (i.e. managed by the user). + See the :doc:`runtimes overview `. diff --git a/doc/index.rst b/doc/index.rst index e8c3b038..0a2d06e8 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -11,10 +11,10 @@ The Nextstrain CLI is a program called ``nextstrain``. It aims to provide a consistent way to run and visualize pathogen builds and access Nextstrain components like :doc:`Augur ` and :doc:`Auspice ` computing platforms, such as -:ref:`installation/docker`, -:ref:`installation/conda`, -:ref:`installation/singularity`, and -:ref:`installation/aws-batch`. +:doc:`/runtimes/docker`, +:doc:`/runtimes/conda`, +:doc:`/runtimes/singularity`, and +:doc:`/runtimes/aws-batch`. .. note:: If you're unfamiliar with Nextstrain builds, you may want to follow our @@ -64,6 +64,17 @@ Table of Contents nextstrain.org Amazon S3 +.. toctree:: + :caption: Runtimes + :titlesonly: + + Overview + Docker + Conda + Singularity + AWS Batch + Ambient + .. toctree:: :caption: Development :titlesonly: diff --git a/doc/installation.rst b/doc/installation.rst index 9d9d67d3..f4e46001 100644 --- a/doc/installation.rst +++ b/doc/installation.rst @@ -106,156 +106,29 @@ The version you get will probably be different than the one shown in the example above. -Runtimes -======== +A Nextstrain runtime +==================== -.. XXX TODO: Move this heading and subheadings (with modification) to their own - top-level doc section (e.g. like Remotes). - -trs, 12 Jan 2023 +If you intend to run commands like :doc:`/commands/build` and +:doc:`/commands/view`, then you'll need to set up at least one :term:`runtime`. +See the :doc:`runtimes overview ` for a comparison of the +options and brief set up instructions for each. Runtime set up typically +concludes by running: -The Nextstrain CLI provides a consistent interface and computing environment -for running and visualizing Nextstrain pathogen builds across several different -computing platforms, such as -:ref:`installation/docker`, -:ref:`installation/conda`, -:ref:`installation/singularity`, and -:ref:`installation/aws-batch`. - -We call the provided computing environments the :term:`Nextstrain runtimes -`. Each runtime provides specific versions of Nextstrain's -software components, like `Augur `__ and -`Auspice `__. - -At least one of these runtimes must be setup in order for many of -``nextstrain``'s subcommands to work, such as ``nextstrain build`` and -``nextstrain view``. - -The default runtime is Docker, using the `nextstrain/base`_ container image. -Containers provide a tremendous amount of benefit for scientific workflows by -isolating dependencies and increasing reproducibility. However, they're not -always appropriate, so a Conda runtime, Singularity runtime, and "ambient" -runtime are also supported. The installation and setup of supported runtimes -is described below. - -.. _nextstrain/base: https://github.com/nextstrain/docker-base - -.. _installation/docker: - -Docker ------- - -`Docker `__ is a very popular container system -freely-available for all platforms. When you use Docker with the Nextstrain -CLI, you don't need to install any other Nextstrain software dependencies as -validated versions are already bundled into a container image by the Nextstrain -team. - -On macOS, download and install `Docker Desktop`_, also known previously as -"Docker for Mac". - -On Linux, install Docker with the standard package manager. For example, on -Ubuntu, you can install Docker with ``sudo apt install docker.io``. - -On Windows, install `Docker Desktop`_ with its support for a WSL2 backend. - -Once you've installed Docker, proceed with :ref:`checking your setup -`. - -.. _Docker Desktop: https://www.docker.com/products/docker-desktop - -.. _installation/conda: - -Conda ------ - -`Conda `__ is a very popular -packaging system freely-available for all platforms. When you use Nextstrain -CLI's built-in Conda support, you don't need to install any other Nextstrain -software dependencies yourself as they're automatically managed in an isolated -location (isolated even from other Conda environments you may manage yourself). - -On macOS and Linux, run ``nextstrain setup conda`` to get started. - -This runtime is not directly supported on Windows, but you can use `WSL2 -`__ to "switch" to -Linux and run the above setup command. - -.. _installation/singularity: - -Singularity ------------ - -Singularity is a container system freely-available for Linux platforms. It is -commonly available on institutional HPC systems as an alternative to Docker, -which is often not supported on such systems. When you use Singularity with -the Nextstrain CLI, you don't need to install any other Nextstrain software -dependencies as validated versions are already bundled into a container image -by the Nextstrain team. - -Run ``nextstrain setup singularity`` to get started. -Singularity version 3.0.0 or newer is required, but we recommend at least -version 3.10.0 or newer when possible. - -Note that the Singularity project forked into two separate projects in late -2021: `SingularityCE`_ under `Sylabs`_ and `Apptainer`_ under the `Linux -Foundation`_. Either fork should work with Nextstrain CLI, as both projects -still provide very similar interfaces and functionality via the ``singularity`` -command. You can read `Sylab's announcement`_ and `Apptainer's announcement`_ -for more information on the fork. - -.. _SingularityCE: https://sylabs.io/singularity/ -.. _Sylabs: https://sylabs.io/ -.. _Apptainer: https://apptainer.org -.. _Linux Foundation: https://www.linuxfoundation.org/ -.. _Sylab's announcement: https://sylabs.io/2022/06/singularityce-is-singularity/ -.. _Apptainer's announcement: https://apptainer.org/news/community-announcement-20211130 - -Ambient -------- - -The "ambient" runtime allows you to use the Nextstrain CLI with your own ambient -setup, for when you cannot or do not want to have Nextstrain CLI manage its own -runtime. - -However, you will need to make sure all of the Nextstrain software dependencies -are available locally or "ambiently" on your computer. A common way to do this -is by manually using `Conda `__ -to manage your own environment that includes the required software, however -you're responsible for making sure the correct software is installed and kept -up-to-date. It is also possible to install the required Nextstrain software -`Augur `__ and `Auspice -`__ and their dependencies manually, -although this is not recommended. - -Once you've installed dependencies, proceed with :ref:`checking your setup -`. - -.. _installation/aws-batch: - -AWS Batch ---------- - -`AWS Batch `__ is an advanced computing -platform which allows you to launch and monitor Nextstrain builds in the -cloud from the comfort of your own computer. The same image used by the local -Docker runtime is used by AWS Batch, making your builds more reproducible, and -builds have access to computers with very large CPU and memory allocations if -necessary. +.. code-block:: bash -The initial setup is quite a bit more involved, but :doc:`detailed instructions -` are available. + nextstrain setup -Once you've setup AWS, proceed with :ref:`checking your setup -`. .. _installation/check-setup: Checking your setup =================== -After installation and setup, run ``nextstrain check-setup --set-default`` to -ensure everything works and automatically pick an appropriate default runtime -based on what's available. You should see output similar to the following: +After installation and runtime set up, run ``nextstrain check-setup +--set-default`` to ensure everything works and automatically pick an +appropriate default runtime based on what's available. You should see output +similar to the following: .. code-block:: console diff --git a/doc/runtimes/ambient.rst b/doc/runtimes/ambient.rst new file mode 100644 index 00000000..621534a1 --- /dev/null +++ b/doc/runtimes/ambient.rst @@ -0,0 +1,6 @@ +=============== +Ambient runtime +=============== + +.. automodule:: nextstrain.cli.runner.ambient + :noindex: diff --git a/doc/runtimes/aws-batch.rst b/doc/runtimes/aws-batch.rst new file mode 100644 index 00000000..44810251 --- /dev/null +++ b/doc/runtimes/aws-batch.rst @@ -0,0 +1,6 @@ +================= +AWS Batch runtime +================= + +.. automodule:: nextstrain.cli.runner.aws_batch + :noindex: diff --git a/doc/runtimes/comparison.csv b/doc/runtimes/comparison.csv new file mode 100644 index 00000000..786c322c --- /dev/null +++ b/doc/runtimes/comparison.csv @@ -0,0 +1,6 @@ +,Isolation level,Containerized?,Locality,External dependencies +Docker,great (3),yes,local or remote,``docker`` command +Conda,some (1),no,local,none +Singularity,good (2),yes,local,``singularity`` command +Ambient,none (0),no,local,many +AWS Batch,great (3),yes,remote,AWS account with Batch set up diff --git a/doc/runtimes/conda.rst b/doc/runtimes/conda.rst new file mode 100644 index 00000000..553f9bfc --- /dev/null +++ b/doc/runtimes/conda.rst @@ -0,0 +1,6 @@ +============= +Conda runtime +============= + +.. automodule:: nextstrain.cli.runner.conda + :noindex: diff --git a/doc/runtimes/docker.rst b/doc/runtimes/docker.rst new file mode 100644 index 00000000..f6899ca1 --- /dev/null +++ b/doc/runtimes/docker.rst @@ -0,0 +1,6 @@ +============== +Docker runtime +============== + +.. automodule:: nextstrain.cli.runner.docker + :noindex: diff --git a/doc/runtimes/index.rst b/doc/runtimes/index.rst new file mode 100644 index 00000000..3585131a --- /dev/null +++ b/doc/runtimes/index.rst @@ -0,0 +1,97 @@ +======== +Runtimes +======== + +Nextstrain's runtimes are specific :term:`computing environments ` in which Nextstrain CLI expects to find and run other Nextstrain +programs, like :doc:`Augur ` and :doc:`Auspice `. +In turn, Nextstrain CLI provides a consistent set of commands to run and +visualize Nextstrain pathogen builds regardless of the underlying runtime in +use. Together, this allows Nextstrain to be used across many different +computing platforms and operating systems. + +The :doc:`/commands/build`, :doc:`/commands/view`, and :doc:`/commands/shell` +commands all require a runtime, as they require access to other Nextstrain +software. + +The :doc:`/commands/setup`, :doc:`/commands/check-setup`, and +:doc:`/commands/update` commands manage the runtimes available for use by the +commands above. The :doc:`/commands/version` command's :option:`--verbose +` option reports detailed version information +about all available runtimes. + +Other Nextstrain CLI commands, such as the :doc:`/commands/remote/index` family +of commands and the related :doc:`/commands/login` and :doc:`/commands/logout` +commands, do not require a runtime. They may be used without any prior set up +of a runtime. + +The runtimes currently available are the: + + - :doc:`/runtimes/docker` + - :doc:`/runtimes/singularity` + - :doc:`/runtimes/conda` + - :doc:`/runtimes/ambient` + - :doc:`/runtimes/aws-batch` + +Runtimes are managed (maintained, tested, versioned, released) by the +Nextstrain team, except for the ambient runtime. The ambient runtime is +special in that it's whatever computing environment in which Nextstrain CLI +itself is running (that is, it's managed by the user). + +You can set up and use multiple runtimes on the same computer, for example if +you want to use them in different contexts. Runtime-using commands let you +select a different runtime than your default with command-line options (e.g. +``--docker``, ``--conda``, and so on). For example, you might use the Docker +runtime to run small builds locally and then use the AWS Batch runtime to run +large scale builds with more computing power. + +If you pick one runtime and later realize you want to switch, you can go back +and set up the other. + + +Comparison +========== + +.. csv-table:: + :file: comparison.csv + :header-rows: 1 + :stub-columns: 1 + +Isolation level + A relative ranking from least isolated (*none*, 0) to most isolated + (*great*, 3) from the underlying computer system. + +Containerized? + A containerized :term:`computing platform` provides a higher degree of + isolation, which in turn usually means a higher degree of portabililty and + reproducibility across different computers and users. + +Locality + *Local* means "on the same computer where ``nextstrain`` is being run". + *Remote* means "on a different computer". + + Docker is most often used to run containers locally, but can also be used + to run them remotely. + +External dependencies + Third-party programs or configuration which are required to use a runtime + but not managed by :doc:`/commands/setup` and :doc:`/commands/update`. + + +Compatibility +============= + +Switching runtimes or updating the version of a runtime may result in different +versions of Nextstrain components like Augur and Auspice as well as other +programs, and thus different behaviour. + +Exact behavioural compatibility is not guaranteed between different runtimes +(e.g. between the Docker vs. Conda runtimes) or between versions of the same +runtime (e.g. between Docker runtime images +``nextstrain/base:build-20230714T205715Z`` and +``nextstrain/base:build-20230720T001758Z``). However, the containerized +runtimes (Docker, Singularity, AWS Batch; see comparison_ above) will usually +have identical behaviour given the same runtime version (e.g. ``build-…``) as +they are all based the same runtime image (``nextstrain/base``). Any variance +is typically due to use of external resources (intentional or otherwise) from +outside the container. diff --git a/doc/runtimes/singularity.rst b/doc/runtimes/singularity.rst new file mode 100644 index 00000000..4d526765 --- /dev/null +++ b/doc/runtimes/singularity.rst @@ -0,0 +1,6 @@ +=================== +Singularity runtime +=================== + +.. automodule:: nextstrain.cli.runner.singularity + :noindex: diff --git a/nextstrain/cli/runner/ambient.py b/nextstrain/cli/runner/ambient.py index 7d1c4574..1d4cd445 100644 --- a/nextstrain/cli/runner/ambient.py +++ b/nextstrain/cli/runner/ambient.py @@ -1,5 +1,31 @@ """ -Run commands in the ambient environment, outside of any container image. +Run commands in the ambient environment, outside of any container image or managed environment. + +The "ambient" runtime allows you to use the Nextstrain CLI with your own ambient +setup, for when you cannot or do not want to have Nextstrain CLI manage its own +runtime. + +.. versionadded:: 1.5.0 +.. versionchanged:: 5.0.0 + Renamed from "native" to "ambient". + + +.. _ambient-setup: + +Setup +===== + +You will need to make sure all of the Nextstrain software dependencies +are available locally or "ambiently" on your computer. A common way to do this +is by manually using `Conda `__ +to manage your own environment that includes the required software, however +you're responsible for making sure the correct software is installed and kept +up-to-date. It is also possible to install the required Nextstrain software +`Augur `__ and `Auspice +`__ and their dependencies manually, +although this is not recommended. + +Once you've installed dependencies, proceed with ``nextstrain setup ambient``. """ import os diff --git a/nextstrain/cli/runner/aws_batch/__init__.py b/nextstrain/cli/runner/aws_batch/__init__.py index c270f508..23a34762 100644 --- a/nextstrain/cli/runner/aws_batch/__init__.py +++ b/nextstrain/cli/runner/aws_batch/__init__.py @@ -1,5 +1,54 @@ """ Run commands remotely on AWS Batch inside the Nextstrain container image. + +`AWS Batch `__ is an advanced computing +platform which allows you to launch and monitor Nextstrain builds in the +cloud from the comfort of your own computer. The same image used by the local +Docker runtime is used by AWS Batch, making your builds more reproducible, and +builds have access to computers with very large CPU and memory allocations if +necessary. + +.. versionadded:: 1.7.0 + + +.. _aws-batch-setup: + +Setup +===== + +The initial setup is quite a bit more involved than most runtimes, but +:doc:`detailed instructions ` are available. + +Once you've set up AWS, proceed with ``nextstrain setup aws-batch``. + + +.. _aws-batch-env: + +Environment variables +===================== + +Defaults for the corresponding command line options, potentially overriding +defaults set by config file variables. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_JOB + + Default for ``--aws-batch-job``. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_QUEUE + + Default for ``--aws-batch-queue``. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_S3_BUCKET + + Default for ``--aws-batch-bucket``. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_CPUS + + Default for ``--aws-batch-cpus``. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_MEMORY + + Default for ``--aws-batch-memory``. """ import os diff --git a/nextstrain/cli/runner/conda.py b/nextstrain/cli/runner/conda.py index fc35d727..929b02c7 100644 --- a/nextstrain/cli/runner/conda.py +++ b/nextstrain/cli/runner/conda.py @@ -1,6 +1,28 @@ """ Run commands with access to a fully-managed Conda environment. +`Conda `__ is a very popular +packaging system freely-available for all platforms. When you use Nextstrain +CLI's built-in Conda support, you don't need to install any other Nextstrain +software dependencies yourself as they're automatically managed in an isolated +location (isolated even from other Conda environments you may manage yourself). + +.. versionadded:: 5.0.0 + + +.. _conda-setup: + +Setup +===== + +On macOS and Linux, run ``nextstrain setup conda`` to get started. + +This runtime is not directly supported on Windows, but you can use `WSL2 +`__ to "switch" to +Linux and run the above setup command. + + +.. _conda-env: Environment variables ===================== diff --git a/nextstrain/cli/runner/docker.py b/nextstrain/cli/runner/docker.py index 4ac04ddf..dab6199d 100644 --- a/nextstrain/cli/runner/docker.py +++ b/nextstrain/cli/runner/docker.py @@ -1,5 +1,53 @@ """ Run commands inside a container image using Docker. + +`Docker `__ is a very popular container system +freely-available for all platforms. When you use Docker with the Nextstrain +CLI, you don't need to install any other Nextstrain software dependencies as +validated versions are already bundled into a container image +(`nextstrain/base`_) by the Nextstrain team. + + +.. _nextstrain/base: https://github.com/nextstrain/docker-base + + +.. _docker-setup: + +Setup +===== + +.. hint:: + This is a reference page with brief pointers for set up. For a more + comprehensive installation guide, please see `our general Nextstrain + installation page `__. + +On macOS, download and install `Docker Desktop`_, also known previously as +"Docker for Mac". + +On Linux, install Docker with the standard package manager. For example, on +Ubuntu, you can install Docker with ``sudo apt install docker.io``. + +On Windows, install `Docker Desktop`_ with its support for a WSL2 backend. + +Once you've installed Docker, proceed with ``nextstrain setup docker``. + +.. _Docker Desktop: https://www.docker.com/products/docker-desktop + + +.. _docker-env: + +Environment variables +===================== + +.. warning:: + For development only. You don't need to set these during normal operation. + +Defaults for the corresponding command line options, potentially overriding +defaults set by config file variables. + +.. envvar:: NEXTSTRAIN_DOCKER_IMAGE + + Default for ``--image`` when using the Docker runtime. """ import os diff --git a/nextstrain/cli/runner/singularity.py b/nextstrain/cli/runner/singularity.py index c8e60c71..f4080258 100644 --- a/nextstrain/cli/runner/singularity.py +++ b/nextstrain/cli/runner/singularity.py @@ -1,9 +1,59 @@ """ Run commands inside a container image using Singularity. -Uses the images built for the Docker runtime by automatically converting them -to local Singularity images. Local images are stored as files named +Singularity is a container system freely-available for Linux platforms. It is +commonly available on institutional HPC systems as an alternative to Docker, +which is often not supported on such systems. When you use Singularity with +the Nextstrain CLI, you don't need to install any other Nextstrain software +dependencies as validated versions are already bundled into a container image +by the Nextstrain team. + +The Singularity runtime uses the same images built for the Docker runtime by +automatically converting them to local Singularity images. Local images are +stored as files named :file:`~/.nextstrain/runtimes/singularity/images/{repository}/{tag}.sif`. + +.. versionadded:: 6.1.0 + + +.. _singularity-setup: + +Setup +===== + +Run ``nextstrain setup singularity`` to get started. +Singularity version 3.0.0 or newer is required, but we recommend at least +version 3.10.0 or newer when possible. + +Note that the Singularity project forked into two separate projects in late +2021: `SingularityCE`_ under `Sylabs`_ and `Apptainer`_ under the `Linux +Foundation`_. Either fork should work with Nextstrain CLI, as both projects +still provide very similar interfaces and functionality via the ``singularity`` +command. You can read `Sylab's announcement`_ and `Apptainer's announcement`_ +for more information on the fork. + +.. _SingularityCE: https://sylabs.io/singularity/ +.. _Sylabs: https://sylabs.io/ +.. _Apptainer: https://apptainer.org +.. _Linux Foundation: https://www.linuxfoundation.org/ +.. _Sylab's announcement: https://sylabs.io/2022/06/singularityce-is-singularity/ +.. _Apptainer's announcement: https://apptainer.org/news/community-announcement-20211130 + + +.. _singularity-env: + +Environment variables +===================== + +.. warning:: + For development only. You don't need to set these during normal operation. + +Defaults for the corresponding command line options, potentially overriding +defaults set by config file variables. + +.. envvar:: NEXTSTRAIN_SINGULARITY_IMAGE + + Default for ``--image`` when using the Singularity runtime. """ import itertools From 8ea4f33fb745167458cdd55f9cd1cd0a28610cbd Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Wed, 2 Aug 2023 16:36:09 -0700 Subject: [PATCH 03/10] docs: Document config file variables and path configuration Previously undocumented! --- doc/config/file.rst | 46 ++++++++++++++++++++ doc/config/paths.rst | 48 +++++++++++++++++++++ doc/index.rst | 7 +++ nextstrain/cli/paths.py | 4 ++ nextstrain/cli/runner/aws_batch/__init__.py | 28 +++++++++++- nextstrain/cli/runner/docker.py | 20 ++++++++- nextstrain/cli/runner/singularity.py | 20 ++++++++- 7 files changed, 170 insertions(+), 3 deletions(-) create mode 100644 doc/config/file.rst create mode 100644 doc/config/paths.rst diff --git a/doc/config/file.rst b/doc/config/file.rst new file mode 100644 index 00000000..f85510c0 --- /dev/null +++ b/doc/config/file.rst @@ -0,0 +1,46 @@ +=========== +Config file +=========== + +Nextstrain CLI uses an INI-style configuration file to store information about +the runtimes that are set up. For example: + +.. code-block:: ini + + [core] + runner = docker + + [docker] + image = nextstrain/base:build-20230623T174208Z + +The default configuration file is :file:`~/.nextstrain/config`. This path may +be overridden entirely by the :envvar:`NEXTSTRAIN_CONFIG` environment variable. +Alternatively, the path of the containing directory (i.e. +:file:`~/.nextstrain/`) may be overridden by the :envvar:`NEXTSTRAIN_HOME` +environment variable. + + +Sections +======== + +- `Core variables`_ +- :ref:`Docker runtime variables ` +- :ref:`Singularity runtime variables ` +- :ref:`AWS Batch runtime variables ` + + +Core variables +============== + +.. glossary:: + + :index:`core.runner ` + Short name of the default :term:`runtime`. Typically set by running + one of: + + .. code-block:: none + + nextstrain setup --set-default + nextstrain check-setup --set-default + + If not set, the :doc:`/runtimes/docker` (``docker``) is used. diff --git a/doc/config/paths.rst b/doc/config/paths.rst new file mode 100644 index 00000000..07ed5cb4 --- /dev/null +++ b/doc/config/paths.rst @@ -0,0 +1,48 @@ +=================== +Paths configuration +=================== + +Nextstrain CLI uses various local filesystem paths for config and runtime data. +If necessary, the defaults can be overridden by environment variables. + +.. envvar:: NEXTSTRAIN_HOME + + Directory for config and other application data. Used as the basis for all + other paths. + + Default is :file:`~/.nextstrain/`, assuming a home directory is + discernable. If not, :file:`.nextstrain` (i.e. in the current directory) + is used as a last resort. + +.. envvar:: NEXTSTRAIN_CONFIG + + File for :doc:`configuration `. + + Default is :file:`{${NEXTSTRAIN_HOME}}/config`. + +.. envvar:: NEXTSTRAIN_SECRETS + + File for secrets (e.g. nextstrain.org tokens) managed by + :doc:`/commands/login` and :doc:`/commands/logout`. + + Default is :file:`{${NEXTSTRAIN_HOME}}/secrets`. + +.. envvar:: NEXTSTRAIN_LOCK + + File for serializing access to other config files to prevent corruption and + other bugs. + + Default is :file:`{${NEXTSTRAIN_HOME}}/lock`. + +.. envvar:: NEXTSTRAIN_RUNTIMES + + Directory for runtime-specific data, e.g. Singularity images or a Conda + environment. Each runtime uses a subdirectory within here. + + Default is :file:`{${NEXTSTRAIN_HOME}}/runtimes/`. + +.. envvar:: NEXTSTRAIN_SHELL_HISTORY + + File for preserving command history across :doc:`/commands/shell` invocations. + + Default is :file:`{${NEXTSTRAIN_HOME}}/shell-history`. diff --git a/doc/index.rst b/doc/index.rst index 0a2d06e8..c2a5d711 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -75,6 +75,13 @@ Table of Contents AWS Batch Ambient +.. toctree:: + :caption: Configuration + :titlesonly: + + Config file + Paths + .. toctree:: :caption: Development :titlesonly: diff --git a/nextstrain/cli/paths.py b/nextstrain/cli/paths.py index 53184ccd..91f5889e 100644 --- a/nextstrain/cli/paths.py +++ b/nextstrain/cli/paths.py @@ -1,5 +1,9 @@ """ Filesystem paths. + +Documented manually in :file:`doc/config/paths.rst` because the autodoc +directives make it too difficult to produce reasonable output for this module. +(Granted, it's a bit of an off-label use.) """ import os from pathlib import Path diff --git a/nextstrain/cli/runner/aws_batch/__init__.py b/nextstrain/cli/runner/aws_batch/__init__.py index 23a34762..8be04681 100644 --- a/nextstrain/cli/runner/aws_batch/__init__.py +++ b/nextstrain/cli/runner/aws_batch/__init__.py @@ -22,13 +22,39 @@ Once you've set up AWS, proceed with ``nextstrain setup aws-batch``. +.. _aws-batch-config: + +Config file variables +===================== + +Defaults for the corresponding command line options, specified in the +:doc:`config file `. + +.. glossary:: + + :index:`aws-batch.job ` + Default for ``--aws-batch-job``. + + :index:`aws-batch.queue ` + Default for ``--aws-batch-queue``. + + :index:`aws-batch.s3-bucket ` + Default for ``--aws-batch-bucket``. + + :index:`aws-batch.cpus ` + Default for ``--aws-batch-cpus``. + + :index:`aws-batch.memory ` + Default for ``--aws-batch-memory``. + + .. _aws-batch-env: Environment variables ===================== Defaults for the corresponding command line options, potentially overriding -defaults set by config file variables. +defaults set by `config file variables`_. .. envvar:: NEXTSTRAIN_AWS_BATCH_JOB diff --git a/nextstrain/cli/runner/docker.py b/nextstrain/cli/runner/docker.py index dab6199d..f520f551 100644 --- a/nextstrain/cli/runner/docker.py +++ b/nextstrain/cli/runner/docker.py @@ -34,6 +34,24 @@ .. _Docker Desktop: https://www.docker.com/products/docker-desktop +.. _docker-config: + +Config file variables +===================== + +Defaults for the corresponding command line options, specified in the +:doc:`config file `. + +.. glossary:: + + :index:`docker.image ` + Default for ``--image`` when using the Docker runtime, e.g. + ``nextstrain/base:build-20230623T174208Z``. + + Typically set initially by ``nextstrain setup`` and subsequently by + ``nextstrain update``. + + .. _docker-env: Environment variables @@ -43,7 +61,7 @@ For development only. You don't need to set these during normal operation. Defaults for the corresponding command line options, potentially overriding -defaults set by config file variables. +defaults set by `config file variables`_. .. envvar:: NEXTSTRAIN_DOCKER_IMAGE diff --git a/nextstrain/cli/runner/singularity.py b/nextstrain/cli/runner/singularity.py index f4080258..0c3727c5 100644 --- a/nextstrain/cli/runner/singularity.py +++ b/nextstrain/cli/runner/singularity.py @@ -40,6 +40,24 @@ .. _Apptainer's announcement: https://apptainer.org/news/community-announcement-20211130 +.. _singularity-config: + +Config file variables +===================== + +Defaults for the corresponding command line options, specified in the +:doc:`config file `. + +.. glossary:: + + :index:`singularity.image ` + Default for ``--image`` when using the Singularity runtime, e.g. + ``docker://nextstrain/base:build-20230525T143814Z``. + + Typically set initially by ``nextstrain setup`` and subsequently by + ``nextstrain update``. + + .. _singularity-env: Environment variables @@ -49,7 +67,7 @@ For development only. You don't need to set these during normal operation. Defaults for the corresponding command line options, potentially overriding -defaults set by config file variables. +defaults set by `config file variables`_. .. envvar:: NEXTSTRAIN_SINGULARITY_IMAGE From 3d4715245512584ac71f3d89a457f7259099e9a8 Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Thu, 3 Aug 2023 12:07:43 -0700 Subject: [PATCH 04/10] runner.singularity: Clean up caches during update and setup Since we immediately convert the Docker/OCI image to SIF, the Docker/OCI image layer caches end up being duplicative. Not keeping them cached means the cache can't be used to avoid downloading layers that haven't changed in subsequent updates, but I think when choosing between minimizing transfer or storage for our users, we should minimize storage. If we want to have our cake and eat it to (i.e. minimize both), we might be able to change tack to newer Singularity features where (I think) OCI images can be used directly without first converting to SIF. Then, like with our Docker runtime, the image we use and layer caches are one and the same. That's a larger change for a future time, though. --- nextstrain/cli/runner/singularity.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/nextstrain/cli/runner/singularity.py b/nextstrain/cli/runner/singularity.py index 0c3727c5..60f96c13 100644 --- a/nextstrain/cli/runner/singularity.py +++ b/nextstrain/cli/runner/singularity.py @@ -373,6 +373,24 @@ def _update(dry_run: bool = False) -> RunnerUpdateStatus: # default going forward. config.set("singularity", "image", latest_image) + # Clean up unnecessary caches + print() + print(colored("bold", "Cleaning up…")) + print() + + if not dry_run: + argv = ("singularity", "cache", "clean", "--type=all", "--force") + env = { + **os.environ.copy(), + **SINGULARITY_CONFIG_ENV, + } + + try: + subprocess.run(argv, env = env, check = True) + except (OSError, subprocess.CalledProcessError) as err: + warn(f"Error running {argv!r}: {err}") + warn(f"Continuing anyway.") + # Prune any old images to avoid leaving lots of hidden disk use around. print() print(colored("bold", "Pruning old images…")) From 23281d54543d84244d4f94231b932432492ca738 Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Thu, 3 Aug 2023 12:15:29 -0700 Subject: [PATCH 05/10] docs: Document the transfer and storage requirements for runtimes Resolves . --- nextstrain/cli/runner/conda.py | 6 ++++++ nextstrain/cli/runner/docker.py | 6 ++++++ nextstrain/cli/runner/singularity.py | 6 ++++++ 3 files changed, 18 insertions(+) diff --git a/nextstrain/cli/runner/conda.py b/nextstrain/cli/runner/conda.py index 929b02c7..4a990fc3 100644 --- a/nextstrain/cli/runner/conda.py +++ b/nextstrain/cli/runner/conda.py @@ -17,6 +17,12 @@ On macOS and Linux, run ``nextstrain setup conda`` to get started. +This will download compressed packages totaling about 450 MB in size which +expand to a final on-disk size of about 2 GB. Transient disk usage during this +process peaks at about 3 GB. These numbers are current as of August 2023, as +observed on Linux. Numbers will vary over time, with a tendency to slowly +increase, and vary slightly by OS. + This runtime is not directly supported on Windows, but you can use `WSL2 `__ to "switch" to Linux and run the above setup command. diff --git a/nextstrain/cli/runner/docker.py b/nextstrain/cli/runner/docker.py index f520f551..d81a6c21 100644 --- a/nextstrain/cli/runner/docker.py +++ b/nextstrain/cli/runner/docker.py @@ -31,6 +31,12 @@ Once you've installed Docker, proceed with ``nextstrain setup docker``. +This will download compressed image layers totaling about 750 MB in size which +expand to a final on-disk size of about 2 GB. Transient disk usage during this +process peaks at about 3 GB. These numbers are current as of August 2023, as +observed on Linux. Numbers will vary over time, with a tendency to slowly +increase, and vary slightly by OS. + .. _Docker Desktop: https://www.docker.com/products/docker-desktop diff --git a/nextstrain/cli/runner/singularity.py b/nextstrain/cli/runner/singularity.py index 60f96c13..535fa361 100644 --- a/nextstrain/cli/runner/singularity.py +++ b/nextstrain/cli/runner/singularity.py @@ -25,6 +25,12 @@ Singularity version 3.0.0 or newer is required, but we recommend at least version 3.10.0 or newer when possible. +This will download image layers totaling about 750 MB in size which are +converted to a final on-disk image of about 650 MB. Transient disk usage +during this process peaks at about 2 GB. These numbers are current as of +August 2023, as observed on Linux. Numbers will vary over time, with a +tendency to slowly increase, and vary slightly by OS. + Note that the Singularity project forked into two separate projects in late 2021: `SingularityCE`_ under `Sylabs`_ and `Apptainer`_ under the `Linux Foundation`_. Either fork should work with Nextstrain CLI, as both projects From 50e18eb03b225eb4fbe1d3a38a92e573c0877a6e Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Thu, 3 Aug 2023 12:23:25 -0700 Subject: [PATCH 06/10] docs: Replace reference to "runtime" term in our umbrella doc's glossary with ours More internally consistent now that we have the glossary term, which in turn links to our overview page about them for more details. --- doc/upgrading.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/doc/upgrading.rst b/doc/upgrading.rst index 327ac1bb..f266df2f 100644 --- a/doc/upgrading.rst +++ b/doc/upgrading.rst @@ -3,9 +3,8 @@ Upgrading ========= This page describes how to upgrade the Nextstrain CLI--the ``nextstrain`` -command--itself, without also upgrading the entire Nextstrain -:term:`docs:runtime`. (To upgrade your Nextstrain runtime, use the -:doc:`/commands/update` command.) +command--itself, without also upgrading the entire Nextstrain :term:`runtime`. +(To upgrade your Nextstrain runtime, use the :doc:`/commands/update` command.) The way to upgrade depends on what kind of Nextstrain CLI installation you have (i.e. how it was first installed), so running ``nextstrain check-setup``: From 8b90a0bb7bbb4b423cfb66f7d64be02854676dcf Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Thu, 17 Aug 2023 11:37:18 -0700 Subject: [PATCH 07/10] =?UTF-8?q?docs:=20Reiterate=20how=20to=20report=20r?= =?UTF-8?q?untime=20version=20info=20in=20=C2=B6=20about=20switching/updat?= =?UTF-8?q?ing=20runtimes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A good suggestion by @joverlee521 in review. --- doc/runtimes/index.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/runtimes/index.rst b/doc/runtimes/index.rst index 3585131a..cf8a53b6 100644 --- a/doc/runtimes/index.rst +++ b/doc/runtimes/index.rst @@ -83,7 +83,9 @@ Compatibility Switching runtimes or updating the version of a runtime may result in different versions of Nextstrain components like Augur and Auspice as well as other -programs, and thus different behaviour. +programs, and thus different behaviour. Use the :doc:`/commands/version` +command's :option:`--verbose ` option to report +detailed version information about all available runtimes. Exact behavioural compatibility is not guaranteed between different runtimes (e.g. between the Docker vs. Conda runtimes) or between versions of the same From 93237e6640b77c61306478e9de4f3568b639a665 Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Thu, 17 Aug 2023 11:47:56 -0700 Subject: [PATCH 08/10] docs: Link to our conda-base repo in the Conda runtime description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit à la how we do it for the docker-base repo and the Docker runtime. A good suggestion by @joverlee521 in review. --- nextstrain/cli/runner/conda.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/nextstrain/cli/runner/conda.py b/nextstrain/cli/runner/conda.py index 4a990fc3..3bd0a635 100644 --- a/nextstrain/cli/runner/conda.py +++ b/nextstrain/cli/runner/conda.py @@ -4,8 +4,13 @@ `Conda `__ is a very popular packaging system freely-available for all platforms. When you use Nextstrain CLI's built-in Conda support, you don't need to install any other Nextstrain -software dependencies yourself as they're automatically managed in an isolated -location (isolated even from other Conda environments you may manage yourself). +software dependencies yourself as validated versions are already bundled into a +package (`nextstrain-base`_) by the Nextstrain team and automatically managed +in an isolated location (isolated even from other Conda environments you may +manage yourself). + +.. _nextstrain-base: https://github.com/nextstrain/conda-base + .. versionadded:: 5.0.0 From acf3453b8e28157b3542a22da78266c27504ac63 Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Thu, 17 Aug 2023 11:57:17 -0700 Subject: [PATCH 09/10] docs: Refer to our general installation page from the setup section of our ambient runtime The general installation page contains a good example of ambient setup. A good suggestion by @victorlin in review. --- nextstrain/cli/runner/ambient.py | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/nextstrain/cli/runner/ambient.py b/nextstrain/cli/runner/ambient.py index 1d4cd445..f1341a79 100644 --- a/nextstrain/cli/runner/ambient.py +++ b/nextstrain/cli/runner/ambient.py @@ -16,12 +16,17 @@ ===== You will need to make sure all of the Nextstrain software dependencies -are available locally or "ambiently" on your computer. A common way to do this -is by manually using `Conda `__ -to manage your own environment that includes the required software, however -you're responsible for making sure the correct software is installed and kept -up-to-date. It is also possible to install the required Nextstrain software -`Augur `__ and `Auspice +are available locally or "ambiently" on your computer. + +A common way to do this is by manually using `Conda +`__ to manage your own +environment that includes the required software, however you're responsible for +making sure the correct software is installed and kept up-to-date. Our +:doc:`general Nextstrain installation page ` describes more +comprehensively how to do this. + +It is also possible to install the required Nextstrain software `Augur +`__ and `Auspice `__ and their dependencies manually, although this is not recommended. From f03bd97b9cddde8f073d4a11e8a69e7847f907ef Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Thu, 17 Aug 2023 12:14:10 -0700 Subject: [PATCH 10/10] CHANGES: Document documentation improvements and Singularity cache clean up --- CHANGES.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/CHANGES.md b/CHANGES.md index 3f658bed..14699e60 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -13,6 +13,15 @@ development source code and as such may not be routinely kept up to date. # __NEXT__ +## Bug fixes + +* The Singularity runtime now cleans up unnecessary caches during `nextstrain + update` and `nextstrain setup`. This reduces the storage space required by + the runtime and prevents it from growing unbounded over time. However, this + also results in increases in network transfer during `nextstrain update`. We + may mitigate the additional transfer in the future. + ([#303][]) + ## Documentation * The command-line `--help` output for commands and the corresponding @@ -22,6 +31,21 @@ development source code and as such may not be routinely kept up to date. and reliably readable. ([#299](https://github.com/nextstrain/cli/pull/299)) +* We've added a large section of new and expanded [documentation on + runtimes](https://docs.nextstrain.org/projects/cli/page/runtimes/), both in + general and for each individual runtime. + ([#303][]) + +* We've added and expanded documentation about configuration, including the + [config file](https://docs.nextstrain.org/projects/cli/page/config/file/) and + the [paths](https://docs.nextstrain.org/projects/cli/page/config/paths/) used + by Nextstrain CLI. Documentation in general now includes more about the + environment variables that can be used to configure Nextstrain CLI. + ([#303][]) + + +[#303]: https://github.com/nextstrain/cli/pull/303 + # 7.1.0 (22 June 2023)