diff --git a/CHANGES.md b/CHANGES.md index 3f658bed..14699e60 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -13,6 +13,15 @@ development source code and as such may not be routinely kept up to date. # __NEXT__ +## Bug fixes + +* The Singularity runtime now cleans up unnecessary caches during `nextstrain + update` and `nextstrain setup`. This reduces the storage space required by + the runtime and prevents it from growing unbounded over time. However, this + also results in increases in network transfer during `nextstrain update`. We + may mitigate the additional transfer in the future. + ([#303][]) + ## Documentation * The command-line `--help` output for commands and the corresponding @@ -22,6 +31,21 @@ development source code and as such may not be routinely kept up to date. and reliably readable. ([#299](https://github.com/nextstrain/cli/pull/299)) +* We've added a large section of new and expanded [documentation on + runtimes](https://docs.nextstrain.org/projects/cli/page/runtimes/), both in + general and for each individual runtime. + ([#303][]) + +* We've added and expanded documentation about configuration, including the + [config file](https://docs.nextstrain.org/projects/cli/page/config/file/) and + the [paths](https://docs.nextstrain.org/projects/cli/page/config/paths/) used + by Nextstrain CLI. Documentation in general now includes more about the + environment variables that can be used to configure Nextstrain CLI. + ([#303][]) + + +[#303]: https://github.com/nextstrain/cli/pull/303 + # 7.1.0 (22 June 2023) diff --git a/README.md b/README.md index 517d3448..55045a62 100644 --- a/README.md +++ b/README.md @@ -18,10 +18,10 @@ email to . [Augur]: https://docs.nextstrain.org/projects/augur/ [Auspice]: https://docs.nextstrain.org/projects/auspice/ -[Docker]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#docker -[Conda]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#conda -[Singularity]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#singularity -[AWS Batch]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#aws-batch +[Docker]: https://docs.nextstrain.org/projects/cli/page/runtimes/docker/ +[Conda]: https://docs.nextstrain.org/projects/cli/page/runtimes/conda/ +[Singularity]: https://docs.nextstrain.org/projects/cli/page/runtimes/singularity/ +[AWS Batch]: https://docs.nextstrain.org/projects/cli/page/runtimes/aws-batch/ [documentation]: https://docs.nextstrain.org/projects/cli/ [development docs]: https://docs.nextstrain.org/projects/cli/page/development/ [open an issue]: https://github.com/nextstrain/cli/issues/new diff --git a/doc/commands/build.rst b/doc/commands/build.rst index bf3a48f8..0a217bbe 100644 --- a/doc/commands/build.rst +++ b/doc/commands/build.rst @@ -120,7 +120,7 @@ default is not suitable. .. option:: --ambient - Run commands in the ambient environment, outside of any container image. + Run commands in the ambient environment, outside of any container image or managed environment. .. option:: --aws-batch diff --git a/doc/commands/view.rst b/doc/commands/view.rst index c91508ad..d15dd3fe 100644 --- a/doc/commands/view.rst +++ b/doc/commands/view.rst @@ -115,7 +115,7 @@ default is not suitable. .. option:: --ambient - Run commands in the ambient environment, outside of any container image. + Run commands in the ambient environment, outside of any container image or managed environment. .. option:: --conda diff --git a/doc/config/file.rst b/doc/config/file.rst new file mode 100644 index 00000000..f85510c0 --- /dev/null +++ b/doc/config/file.rst @@ -0,0 +1,46 @@ +=========== +Config file +=========== + +Nextstrain CLI uses an INI-style configuration file to store information about +the runtimes that are set up. For example: + +.. code-block:: ini + + [core] + runner = docker + + [docker] + image = nextstrain/base:build-20230623T174208Z + +The default configuration file is :file:`~/.nextstrain/config`. This path may +be overridden entirely by the :envvar:`NEXTSTRAIN_CONFIG` environment variable. +Alternatively, the path of the containing directory (i.e. +:file:`~/.nextstrain/`) may be overridden by the :envvar:`NEXTSTRAIN_HOME` +environment variable. + + +Sections +======== + +- `Core variables`_ +- :ref:`Docker runtime variables ` +- :ref:`Singularity runtime variables ` +- :ref:`AWS Batch runtime variables ` + + +Core variables +============== + +.. glossary:: + + :index:`core.runner ` + Short name of the default :term:`runtime`. Typically set by running + one of: + + .. code-block:: none + + nextstrain setup --set-default + nextstrain check-setup --set-default + + If not set, the :doc:`/runtimes/docker` (``docker``) is used. diff --git a/doc/config/paths.rst b/doc/config/paths.rst new file mode 100644 index 00000000..07ed5cb4 --- /dev/null +++ b/doc/config/paths.rst @@ -0,0 +1,48 @@ +=================== +Paths configuration +=================== + +Nextstrain CLI uses various local filesystem paths for config and runtime data. +If necessary, the defaults can be overridden by environment variables. + +.. envvar:: NEXTSTRAIN_HOME + + Directory for config and other application data. Used as the basis for all + other paths. + + Default is :file:`~/.nextstrain/`, assuming a home directory is + discernable. If not, :file:`.nextstrain` (i.e. in the current directory) + is used as a last resort. + +.. envvar:: NEXTSTRAIN_CONFIG + + File for :doc:`configuration `. + + Default is :file:`{${NEXTSTRAIN_HOME}}/config`. + +.. envvar:: NEXTSTRAIN_SECRETS + + File for secrets (e.g. nextstrain.org tokens) managed by + :doc:`/commands/login` and :doc:`/commands/logout`. + + Default is :file:`{${NEXTSTRAIN_HOME}}/secrets`. + +.. envvar:: NEXTSTRAIN_LOCK + + File for serializing access to other config files to prevent corruption and + other bugs. + + Default is :file:`{${NEXTSTRAIN_HOME}}/lock`. + +.. envvar:: NEXTSTRAIN_RUNTIMES + + Directory for runtime-specific data, e.g. Singularity images or a Conda + environment. Each runtime uses a subdirectory within here. + + Default is :file:`{${NEXTSTRAIN_HOME}}/runtimes/`. + +.. envvar:: NEXTSTRAIN_SHELL_HISTORY + + File for preserving command history across :doc:`/commands/shell` invocations. + + Default is :file:`{${NEXTSTRAIN_HOME}}/shell-history`. diff --git a/doc/glossary.rst b/doc/glossary.rst index bd87e5c1..7bf2c36e 100644 --- a/doc/glossary.rst +++ b/doc/glossary.rst @@ -36,17 +36,4 @@ documentation. environment) in which Nextstrain CLI expects to find and execute other Nextstrain programs. - Current runtimes: - - - Docker with the `nextstrain/base image `_ - - Conda with the `nextstrain-base meta-package `__ - - Ambient - - AWS Batch with the `nextstrain/base image`_ - - Each runtime provides specific versions of Nextstrain's software - components, like Augur and Auspice. - - Runtimes are managed (maintained, tested, versioned, released) by the - Nextstrain team, except for the ambient runtime. The ambient runtime - is special in that it's whatever computing environment in which - Nextstrain CLI itself is running (i.e. managed by the user). + See the :doc:`runtimes overview `. diff --git a/doc/index.rst b/doc/index.rst index e8c3b038..c2a5d711 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -11,10 +11,10 @@ The Nextstrain CLI is a program called ``nextstrain``. It aims to provide a consistent way to run and visualize pathogen builds and access Nextstrain components like :doc:`Augur ` and :doc:`Auspice ` computing platforms, such as -:ref:`installation/docker`, -:ref:`installation/conda`, -:ref:`installation/singularity`, and -:ref:`installation/aws-batch`. +:doc:`/runtimes/docker`, +:doc:`/runtimes/conda`, +:doc:`/runtimes/singularity`, and +:doc:`/runtimes/aws-batch`. .. note:: If you're unfamiliar with Nextstrain builds, you may want to follow our @@ -64,6 +64,24 @@ Table of Contents nextstrain.org Amazon S3 +.. toctree:: + :caption: Runtimes + :titlesonly: + + Overview + Docker + Conda + Singularity + AWS Batch + Ambient + +.. toctree:: + :caption: Configuration + :titlesonly: + + Config file + Paths + .. toctree:: :caption: Development :titlesonly: diff --git a/doc/installation.rst b/doc/installation.rst index 9d9d67d3..f4e46001 100644 --- a/doc/installation.rst +++ b/doc/installation.rst @@ -106,156 +106,29 @@ The version you get will probably be different than the one shown in the example above. -Runtimes -======== +A Nextstrain runtime +==================== -.. XXX TODO: Move this heading and subheadings (with modification) to their own - top-level doc section (e.g. like Remotes). - -trs, 12 Jan 2023 +If you intend to run commands like :doc:`/commands/build` and +:doc:`/commands/view`, then you'll need to set up at least one :term:`runtime`. +See the :doc:`runtimes overview ` for a comparison of the +options and brief set up instructions for each. Runtime set up typically +concludes by running: -The Nextstrain CLI provides a consistent interface and computing environment -for running and visualizing Nextstrain pathogen builds across several different -computing platforms, such as -:ref:`installation/docker`, -:ref:`installation/conda`, -:ref:`installation/singularity`, and -:ref:`installation/aws-batch`. - -We call the provided computing environments the :term:`Nextstrain runtimes -`. Each runtime provides specific versions of Nextstrain's -software components, like `Augur `__ and -`Auspice `__. - -At least one of these runtimes must be setup in order for many of -``nextstrain``'s subcommands to work, such as ``nextstrain build`` and -``nextstrain view``. - -The default runtime is Docker, using the `nextstrain/base`_ container image. -Containers provide a tremendous amount of benefit for scientific workflows by -isolating dependencies and increasing reproducibility. However, they're not -always appropriate, so a Conda runtime, Singularity runtime, and "ambient" -runtime are also supported. The installation and setup of supported runtimes -is described below. - -.. _nextstrain/base: https://github.com/nextstrain/docker-base - -.. _installation/docker: - -Docker ------- - -`Docker `__ is a very popular container system -freely-available for all platforms. When you use Docker with the Nextstrain -CLI, you don't need to install any other Nextstrain software dependencies as -validated versions are already bundled into a container image by the Nextstrain -team. - -On macOS, download and install `Docker Desktop`_, also known previously as -"Docker for Mac". - -On Linux, install Docker with the standard package manager. For example, on -Ubuntu, you can install Docker with ``sudo apt install docker.io``. - -On Windows, install `Docker Desktop`_ with its support for a WSL2 backend. - -Once you've installed Docker, proceed with :ref:`checking your setup -`. - -.. _Docker Desktop: https://www.docker.com/products/docker-desktop - -.. _installation/conda: - -Conda ------ - -`Conda `__ is a very popular -packaging system freely-available for all platforms. When you use Nextstrain -CLI's built-in Conda support, you don't need to install any other Nextstrain -software dependencies yourself as they're automatically managed in an isolated -location (isolated even from other Conda environments you may manage yourself). - -On macOS and Linux, run ``nextstrain setup conda`` to get started. - -This runtime is not directly supported on Windows, but you can use `WSL2 -`__ to "switch" to -Linux and run the above setup command. - -.. _installation/singularity: - -Singularity ------------ - -Singularity is a container system freely-available for Linux platforms. It is -commonly available on institutional HPC systems as an alternative to Docker, -which is often not supported on such systems. When you use Singularity with -the Nextstrain CLI, you don't need to install any other Nextstrain software -dependencies as validated versions are already bundled into a container image -by the Nextstrain team. - -Run ``nextstrain setup singularity`` to get started. -Singularity version 3.0.0 or newer is required, but we recommend at least -version 3.10.0 or newer when possible. - -Note that the Singularity project forked into two separate projects in late -2021: `SingularityCE`_ under `Sylabs`_ and `Apptainer`_ under the `Linux -Foundation`_. Either fork should work with Nextstrain CLI, as both projects -still provide very similar interfaces and functionality via the ``singularity`` -command. You can read `Sylab's announcement`_ and `Apptainer's announcement`_ -for more information on the fork. - -.. _SingularityCE: https://sylabs.io/singularity/ -.. _Sylabs: https://sylabs.io/ -.. _Apptainer: https://apptainer.org -.. _Linux Foundation: https://www.linuxfoundation.org/ -.. _Sylab's announcement: https://sylabs.io/2022/06/singularityce-is-singularity/ -.. _Apptainer's announcement: https://apptainer.org/news/community-announcement-20211130 - -Ambient -------- - -The "ambient" runtime allows you to use the Nextstrain CLI with your own ambient -setup, for when you cannot or do not want to have Nextstrain CLI manage its own -runtime. - -However, you will need to make sure all of the Nextstrain software dependencies -are available locally or "ambiently" on your computer. A common way to do this -is by manually using `Conda `__ -to manage your own environment that includes the required software, however -you're responsible for making sure the correct software is installed and kept -up-to-date. It is also possible to install the required Nextstrain software -`Augur `__ and `Auspice -`__ and their dependencies manually, -although this is not recommended. - -Once you've installed dependencies, proceed with :ref:`checking your setup -`. - -.. _installation/aws-batch: - -AWS Batch ---------- - -`AWS Batch `__ is an advanced computing -platform which allows you to launch and monitor Nextstrain builds in the -cloud from the comfort of your own computer. The same image used by the local -Docker runtime is used by AWS Batch, making your builds more reproducible, and -builds have access to computers with very large CPU and memory allocations if -necessary. +.. code-block:: bash -The initial setup is quite a bit more involved, but :doc:`detailed instructions -` are available. + nextstrain setup -Once you've setup AWS, proceed with :ref:`checking your setup -`. .. _installation/check-setup: Checking your setup =================== -After installation and setup, run ``nextstrain check-setup --set-default`` to -ensure everything works and automatically pick an appropriate default runtime -based on what's available. You should see output similar to the following: +After installation and runtime set up, run ``nextstrain check-setup +--set-default`` to ensure everything works and automatically pick an +appropriate default runtime based on what's available. You should see output +similar to the following: .. code-block:: console diff --git a/doc/runtimes/ambient.rst b/doc/runtimes/ambient.rst new file mode 100644 index 00000000..621534a1 --- /dev/null +++ b/doc/runtimes/ambient.rst @@ -0,0 +1,6 @@ +=============== +Ambient runtime +=============== + +.. automodule:: nextstrain.cli.runner.ambient + :noindex: diff --git a/doc/runtimes/aws-batch.rst b/doc/runtimes/aws-batch.rst new file mode 100644 index 00000000..44810251 --- /dev/null +++ b/doc/runtimes/aws-batch.rst @@ -0,0 +1,6 @@ +================= +AWS Batch runtime +================= + +.. automodule:: nextstrain.cli.runner.aws_batch + :noindex: diff --git a/doc/runtimes/comparison.csv b/doc/runtimes/comparison.csv new file mode 100644 index 00000000..786c322c --- /dev/null +++ b/doc/runtimes/comparison.csv @@ -0,0 +1,6 @@ +,Isolation level,Containerized?,Locality,External dependencies +Docker,great (3),yes,local or remote,``docker`` command +Conda,some (1),no,local,none +Singularity,good (2),yes,local,``singularity`` command +Ambient,none (0),no,local,many +AWS Batch,great (3),yes,remote,AWS account with Batch set up diff --git a/doc/runtimes/conda.rst b/doc/runtimes/conda.rst new file mode 100644 index 00000000..553f9bfc --- /dev/null +++ b/doc/runtimes/conda.rst @@ -0,0 +1,6 @@ +============= +Conda runtime +============= + +.. automodule:: nextstrain.cli.runner.conda + :noindex: diff --git a/doc/runtimes/docker.rst b/doc/runtimes/docker.rst new file mode 100644 index 00000000..f6899ca1 --- /dev/null +++ b/doc/runtimes/docker.rst @@ -0,0 +1,6 @@ +============== +Docker runtime +============== + +.. automodule:: nextstrain.cli.runner.docker + :noindex: diff --git a/doc/runtimes/index.rst b/doc/runtimes/index.rst new file mode 100644 index 00000000..cf8a53b6 --- /dev/null +++ b/doc/runtimes/index.rst @@ -0,0 +1,99 @@ +======== +Runtimes +======== + +Nextstrain's runtimes are specific :term:`computing environments ` in which Nextstrain CLI expects to find and run other Nextstrain +programs, like :doc:`Augur ` and :doc:`Auspice `. +In turn, Nextstrain CLI provides a consistent set of commands to run and +visualize Nextstrain pathogen builds regardless of the underlying runtime in +use. Together, this allows Nextstrain to be used across many different +computing platforms and operating systems. + +The :doc:`/commands/build`, :doc:`/commands/view`, and :doc:`/commands/shell` +commands all require a runtime, as they require access to other Nextstrain +software. + +The :doc:`/commands/setup`, :doc:`/commands/check-setup`, and +:doc:`/commands/update` commands manage the runtimes available for use by the +commands above. The :doc:`/commands/version` command's :option:`--verbose +` option reports detailed version information +about all available runtimes. + +Other Nextstrain CLI commands, such as the :doc:`/commands/remote/index` family +of commands and the related :doc:`/commands/login` and :doc:`/commands/logout` +commands, do not require a runtime. They may be used without any prior set up +of a runtime. + +The runtimes currently available are the: + + - :doc:`/runtimes/docker` + - :doc:`/runtimes/singularity` + - :doc:`/runtimes/conda` + - :doc:`/runtimes/ambient` + - :doc:`/runtimes/aws-batch` + +Runtimes are managed (maintained, tested, versioned, released) by the +Nextstrain team, except for the ambient runtime. The ambient runtime is +special in that it's whatever computing environment in which Nextstrain CLI +itself is running (that is, it's managed by the user). + +You can set up and use multiple runtimes on the same computer, for example if +you want to use them in different contexts. Runtime-using commands let you +select a different runtime than your default with command-line options (e.g. +``--docker``, ``--conda``, and so on). For example, you might use the Docker +runtime to run small builds locally and then use the AWS Batch runtime to run +large scale builds with more computing power. + +If you pick one runtime and later realize you want to switch, you can go back +and set up the other. + + +Comparison +========== + +.. csv-table:: + :file: comparison.csv + :header-rows: 1 + :stub-columns: 1 + +Isolation level + A relative ranking from least isolated (*none*, 0) to most isolated + (*great*, 3) from the underlying computer system. + +Containerized? + A containerized :term:`computing platform` provides a higher degree of + isolation, which in turn usually means a higher degree of portabililty and + reproducibility across different computers and users. + +Locality + *Local* means "on the same computer where ``nextstrain`` is being run". + *Remote* means "on a different computer". + + Docker is most often used to run containers locally, but can also be used + to run them remotely. + +External dependencies + Third-party programs or configuration which are required to use a runtime + but not managed by :doc:`/commands/setup` and :doc:`/commands/update`. + + +Compatibility +============= + +Switching runtimes or updating the version of a runtime may result in different +versions of Nextstrain components like Augur and Auspice as well as other +programs, and thus different behaviour. Use the :doc:`/commands/version` +command's :option:`--verbose ` option to report +detailed version information about all available runtimes. + +Exact behavioural compatibility is not guaranteed between different runtimes +(e.g. between the Docker vs. Conda runtimes) or between versions of the same +runtime (e.g. between Docker runtime images +``nextstrain/base:build-20230714T205715Z`` and +``nextstrain/base:build-20230720T001758Z``). However, the containerized +runtimes (Docker, Singularity, AWS Batch; see comparison_ above) will usually +have identical behaviour given the same runtime version (e.g. ``build-…``) as +they are all based the same runtime image (``nextstrain/base``). Any variance +is typically due to use of external resources (intentional or otherwise) from +outside the container. diff --git a/doc/runtimes/singularity.rst b/doc/runtimes/singularity.rst new file mode 100644 index 00000000..4d526765 --- /dev/null +++ b/doc/runtimes/singularity.rst @@ -0,0 +1,6 @@ +=================== +Singularity runtime +=================== + +.. automodule:: nextstrain.cli.runner.singularity + :noindex: diff --git a/doc/upgrading.rst b/doc/upgrading.rst index 76439a9a..f266df2f 100644 --- a/doc/upgrading.rst +++ b/doc/upgrading.rst @@ -3,8 +3,8 @@ Upgrading ========= This page describes how to upgrade the Nextstrain CLI--the ``nextstrain`` -command--itself, without also upgrading the entire Nextstrain -:term:`docs:runtime`. +command--itself, without also upgrading the entire Nextstrain :term:`runtime`. +(To upgrade your Nextstrain runtime, use the :doc:`/commands/update` command.) The way to upgrade depends on what kind of Nextstrain CLI installation you have (i.e. how it was first installed), so running ``nextstrain check-setup``: diff --git a/nextstrain/cli/paths.py b/nextstrain/cli/paths.py index 53184ccd..91f5889e 100644 --- a/nextstrain/cli/paths.py +++ b/nextstrain/cli/paths.py @@ -1,5 +1,9 @@ """ Filesystem paths. + +Documented manually in :file:`doc/config/paths.rst` because the autodoc +directives make it too difficult to produce reasonable output for this module. +(Granted, it's a bit of an off-label use.) """ import os from pathlib import Path diff --git a/nextstrain/cli/runner/ambient.py b/nextstrain/cli/runner/ambient.py index 7d1c4574..f1341a79 100644 --- a/nextstrain/cli/runner/ambient.py +++ b/nextstrain/cli/runner/ambient.py @@ -1,5 +1,36 @@ """ -Run commands in the ambient environment, outside of any container image. +Run commands in the ambient environment, outside of any container image or managed environment. + +The "ambient" runtime allows you to use the Nextstrain CLI with your own ambient +setup, for when you cannot or do not want to have Nextstrain CLI manage its own +runtime. + +.. versionadded:: 1.5.0 +.. versionchanged:: 5.0.0 + Renamed from "native" to "ambient". + + +.. _ambient-setup: + +Setup +===== + +You will need to make sure all of the Nextstrain software dependencies +are available locally or "ambiently" on your computer. + +A common way to do this is by manually using `Conda +`__ to manage your own +environment that includes the required software, however you're responsible for +making sure the correct software is installed and kept up-to-date. Our +:doc:`general Nextstrain installation page ` describes more +comprehensively how to do this. + +It is also possible to install the required Nextstrain software `Augur +`__ and `Auspice +`__ and their dependencies manually, +although this is not recommended. + +Once you've installed dependencies, proceed with ``nextstrain setup ambient``. """ import os diff --git a/nextstrain/cli/runner/aws_batch/__init__.py b/nextstrain/cli/runner/aws_batch/__init__.py index c270f508..8be04681 100644 --- a/nextstrain/cli/runner/aws_batch/__init__.py +++ b/nextstrain/cli/runner/aws_batch/__init__.py @@ -1,5 +1,80 @@ """ Run commands remotely on AWS Batch inside the Nextstrain container image. + +`AWS Batch `__ is an advanced computing +platform which allows you to launch and monitor Nextstrain builds in the +cloud from the comfort of your own computer. The same image used by the local +Docker runtime is used by AWS Batch, making your builds more reproducible, and +builds have access to computers with very large CPU and memory allocations if +necessary. + +.. versionadded:: 1.7.0 + + +.. _aws-batch-setup: + +Setup +===== + +The initial setup is quite a bit more involved than most runtimes, but +:doc:`detailed instructions ` are available. + +Once you've set up AWS, proceed with ``nextstrain setup aws-batch``. + + +.. _aws-batch-config: + +Config file variables +===================== + +Defaults for the corresponding command line options, specified in the +:doc:`config file `. + +.. glossary:: + + :index:`aws-batch.job ` + Default for ``--aws-batch-job``. + + :index:`aws-batch.queue ` + Default for ``--aws-batch-queue``. + + :index:`aws-batch.s3-bucket ` + Default for ``--aws-batch-bucket``. + + :index:`aws-batch.cpus ` + Default for ``--aws-batch-cpus``. + + :index:`aws-batch.memory ` + Default for ``--aws-batch-memory``. + + +.. _aws-batch-env: + +Environment variables +===================== + +Defaults for the corresponding command line options, potentially overriding +defaults set by `config file variables`_. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_JOB + + Default for ``--aws-batch-job``. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_QUEUE + + Default for ``--aws-batch-queue``. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_S3_BUCKET + + Default for ``--aws-batch-bucket``. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_CPUS + + Default for ``--aws-batch-cpus``. + +.. envvar:: NEXTSTRAIN_AWS_BATCH_MEMORY + + Default for ``--aws-batch-memory``. """ import os diff --git a/nextstrain/cli/runner/conda.py b/nextstrain/cli/runner/conda.py index fc35d727..3bd0a635 100644 --- a/nextstrain/cli/runner/conda.py +++ b/nextstrain/cli/runner/conda.py @@ -1,6 +1,39 @@ """ Run commands with access to a fully-managed Conda environment. +`Conda `__ is a very popular +packaging system freely-available for all platforms. When you use Nextstrain +CLI's built-in Conda support, you don't need to install any other Nextstrain +software dependencies yourself as validated versions are already bundled into a +package (`nextstrain-base`_) by the Nextstrain team and automatically managed +in an isolated location (isolated even from other Conda environments you may +manage yourself). + +.. _nextstrain-base: https://github.com/nextstrain/conda-base + + +.. versionadded:: 5.0.0 + + +.. _conda-setup: + +Setup +===== + +On macOS and Linux, run ``nextstrain setup conda`` to get started. + +This will download compressed packages totaling about 450 MB in size which +expand to a final on-disk size of about 2 GB. Transient disk usage during this +process peaks at about 3 GB. These numbers are current as of August 2023, as +observed on Linux. Numbers will vary over time, with a tendency to slowly +increase, and vary slightly by OS. + +This runtime is not directly supported on Windows, but you can use `WSL2 +`__ to "switch" to +Linux and run the above setup command. + + +.. _conda-env: Environment variables ===================== diff --git a/nextstrain/cli/runner/docker.py b/nextstrain/cli/runner/docker.py index 4ac04ddf..d81a6c21 100644 --- a/nextstrain/cli/runner/docker.py +++ b/nextstrain/cli/runner/docker.py @@ -1,5 +1,77 @@ """ Run commands inside a container image using Docker. + +`Docker `__ is a very popular container system +freely-available for all platforms. When you use Docker with the Nextstrain +CLI, you don't need to install any other Nextstrain software dependencies as +validated versions are already bundled into a container image +(`nextstrain/base`_) by the Nextstrain team. + + +.. _nextstrain/base: https://github.com/nextstrain/docker-base + + +.. _docker-setup: + +Setup +===== + +.. hint:: + This is a reference page with brief pointers for set up. For a more + comprehensive installation guide, please see `our general Nextstrain + installation page `__. + +On macOS, download and install `Docker Desktop`_, also known previously as +"Docker for Mac". + +On Linux, install Docker with the standard package manager. For example, on +Ubuntu, you can install Docker with ``sudo apt install docker.io``. + +On Windows, install `Docker Desktop`_ with its support for a WSL2 backend. + +Once you've installed Docker, proceed with ``nextstrain setup docker``. + +This will download compressed image layers totaling about 750 MB in size which +expand to a final on-disk size of about 2 GB. Transient disk usage during this +process peaks at about 3 GB. These numbers are current as of August 2023, as +observed on Linux. Numbers will vary over time, with a tendency to slowly +increase, and vary slightly by OS. + +.. _Docker Desktop: https://www.docker.com/products/docker-desktop + + +.. _docker-config: + +Config file variables +===================== + +Defaults for the corresponding command line options, specified in the +:doc:`config file `. + +.. glossary:: + + :index:`docker.image ` + Default for ``--image`` when using the Docker runtime, e.g. + ``nextstrain/base:build-20230623T174208Z``. + + Typically set initially by ``nextstrain setup`` and subsequently by + ``nextstrain update``. + + +.. _docker-env: + +Environment variables +===================== + +.. warning:: + For development only. You don't need to set these during normal operation. + +Defaults for the corresponding command line options, potentially overriding +defaults set by `config file variables`_. + +.. envvar:: NEXTSTRAIN_DOCKER_IMAGE + + Default for ``--image`` when using the Docker runtime. """ import os diff --git a/nextstrain/cli/runner/singularity.py b/nextstrain/cli/runner/singularity.py index c8e60c71..535fa361 100644 --- a/nextstrain/cli/runner/singularity.py +++ b/nextstrain/cli/runner/singularity.py @@ -1,9 +1,83 @@ """ Run commands inside a container image using Singularity. -Uses the images built for the Docker runtime by automatically converting them -to local Singularity images. Local images are stored as files named +Singularity is a container system freely-available for Linux platforms. It is +commonly available on institutional HPC systems as an alternative to Docker, +which is often not supported on such systems. When you use Singularity with +the Nextstrain CLI, you don't need to install any other Nextstrain software +dependencies as validated versions are already bundled into a container image +by the Nextstrain team. + +The Singularity runtime uses the same images built for the Docker runtime by +automatically converting them to local Singularity images. Local images are +stored as files named :file:`~/.nextstrain/runtimes/singularity/images/{repository}/{tag}.sif`. + +.. versionadded:: 6.1.0 + + +.. _singularity-setup: + +Setup +===== + +Run ``nextstrain setup singularity`` to get started. +Singularity version 3.0.0 or newer is required, but we recommend at least +version 3.10.0 or newer when possible. + +This will download image layers totaling about 750 MB in size which are +converted to a final on-disk image of about 650 MB. Transient disk usage +during this process peaks at about 2 GB. These numbers are current as of +August 2023, as observed on Linux. Numbers will vary over time, with a +tendency to slowly increase, and vary slightly by OS. + +Note that the Singularity project forked into two separate projects in late +2021: `SingularityCE`_ under `Sylabs`_ and `Apptainer`_ under the `Linux +Foundation`_. Either fork should work with Nextstrain CLI, as both projects +still provide very similar interfaces and functionality via the ``singularity`` +command. You can read `Sylab's announcement`_ and `Apptainer's announcement`_ +for more information on the fork. + +.. _SingularityCE: https://sylabs.io/singularity/ +.. _Sylabs: https://sylabs.io/ +.. _Apptainer: https://apptainer.org +.. _Linux Foundation: https://www.linuxfoundation.org/ +.. _Sylab's announcement: https://sylabs.io/2022/06/singularityce-is-singularity/ +.. _Apptainer's announcement: https://apptainer.org/news/community-announcement-20211130 + + +.. _singularity-config: + +Config file variables +===================== + +Defaults for the corresponding command line options, specified in the +:doc:`config file `. + +.. glossary:: + + :index:`singularity.image ` + Default for ``--image`` when using the Singularity runtime, e.g. + ``docker://nextstrain/base:build-20230525T143814Z``. + + Typically set initially by ``nextstrain setup`` and subsequently by + ``nextstrain update``. + + +.. _singularity-env: + +Environment variables +===================== + +.. warning:: + For development only. You don't need to set these during normal operation. + +Defaults for the corresponding command line options, potentially overriding +defaults set by `config file variables`_. + +.. envvar:: NEXTSTRAIN_SINGULARITY_IMAGE + + Default for ``--image`` when using the Singularity runtime. """ import itertools @@ -305,6 +379,24 @@ def _update(dry_run: bool = False) -> RunnerUpdateStatus: # default going forward. config.set("singularity", "image", latest_image) + # Clean up unnecessary caches + print() + print(colored("bold", "Cleaning up…")) + print() + + if not dry_run: + argv = ("singularity", "cache", "clean", "--type=all", "--force") + env = { + **os.environ.copy(), + **SINGULARITY_CONFIG_ENV, + } + + try: + subprocess.run(argv, env = env, check = True) + except (OSError, subprocess.CalledProcessError) as err: + warn(f"Error running {argv!r}: {err}") + warn(f"Continuing anyway.") + # Prune any old images to avoid leaving lots of hidden disk use around. print() print(colored("bold", "Pruning old images…"))