nextstrain · tsibley · Aug 17, 2023 · Aug 2, 2023 · Jan 25, 2023 · Aug 2, 2023
diff --git a/CHANGES.md b/CHANGES.md
@@ -13,6 +13,15 @@ development source code and as such may not be routinely kept up to date.
 
 # __NEXT__
 
+## Bug fixes
+
+* The Singularity runtime now cleans up unnecessary caches during `nextstrain
+  update` and `nextstrain setup`.  This reduces the storage space required by
+  the runtime and prevents it from growing unbounded over time.  However, this
+  also results in increases in network transfer during `nextstrain update`.  We
+  may mitigate the additional transfer in the future.
+  ([#303][])
+
 ## Documentation
 
 * The command-line `--help` output for commands and the corresponding
@@ -22,6 +31,21 @@ development source code and as such may not be routinely kept up to date.
   and reliably readable.
   ([#299](https://github.com/nextstrain/cli/pull/299))
 
+* We've added a large section of new and expanded [documentation on
+  runtimes](https://docs.nextstrain.org/projects/cli/page/runtimes/), both in
+  general and for each individual runtime.
+  ([#303][])
+
+* We've added and expanded documentation about configuration, including the
+  [config file](https://docs.nextstrain.org/projects/cli/page/config/file/) and
+  the [paths](https://docs.nextstrain.org/projects/cli/page/config/paths/) used
+  by Nextstrain CLI.  Documentation in general now includes more about the
+  environment variables that can be used to configure Nextstrain CLI.
+  ([#303][])
+
+
+[#303]: https://github.com/nextstrain/cli/pull/303
+
 
 # 7.1.0 (22 June 2023)
 

diff --git a/README.md b/README.md
@@ -18,10 +18,10 @@ email to <[email protected]>.
 
 [Augur]: https://docs.nextstrain.org/projects/augur/
 [Auspice]: https://docs.nextstrain.org/projects/auspice/
-[Docker]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#docker
-[Conda]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#conda
-[Singularity]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#singularity
-[AWS Batch]: https://docs.nextstrain.org/projects/cli/en/stable/installation/#aws-batch
+[Docker]: https://docs.nextstrain.org/projects/cli/page/runtimes/docker/
+[Conda]: https://docs.nextstrain.org/projects/cli/page/runtimes/conda/
+[Singularity]: https://docs.nextstrain.org/projects/cli/page/runtimes/singularity/
+[AWS Batch]: https://docs.nextstrain.org/projects/cli/page/runtimes/aws-batch/
 [documentation]: https://docs.nextstrain.org/projects/cli/
 [development docs]: https://docs.nextstrain.org/projects/cli/page/development/
 [open an issue]: https://github.com/nextstrain/cli/issues/new
diff --git a/doc/commands/build.rst b/doc/commands/build.rst
@@ -120,7 +120,7 @@ default is not suitable.
 
 .. option:: --ambient
 
-    Run commands in the ambient environment, outside of any container image.
+    Run commands in the ambient environment, outside of any container image or managed environment.
 
 .. option:: --aws-batch
 

diff --git a/doc/commands/view.rst b/doc/commands/view.rst
@@ -115,7 +115,7 @@ default is not suitable.
 
 .. option:: --ambient
 
-    Run commands in the ambient environment, outside of any container image.
+    Run commands in the ambient environment, outside of any container image or managed environment.
 
 .. option:: --conda
 

diff --git a/doc/config/file.rst b/doc/config/file.rst
@@ -0,0 +1,46 @@
+===========
+Config file
+===========
+
+Nextstrain CLI uses an INI-style configuration file to store information about
+the runtimes that are set up.  For example:
+
+.. code-block:: ini
+
+    [core]
+    runner = docker
+
+    [docker]
+    image = nextstrain/base:build-20230623T174208Z
+
+The default configuration file is :file:`~/.nextstrain/config`.  This path may
+be overridden entirely by the :envvar:`NEXTSTRAIN_CONFIG` environment variable.
+Alternatively, the path of the containing directory (i.e.
+:file:`~/.nextstrain/`) may be overridden by the :envvar:`NEXTSTRAIN_HOME`
+environment variable.
+
+
+Sections
+========
+
+- `Core variables`_
+- :ref:`Docker runtime variables <docker-config>`
+- :ref:`Singularity runtime variables <singularity-config>`
+- :ref:`AWS Batch runtime variables <aws-batch-config>`
+
+
+Core variables
+==============
+
+.. glossary::
+
+    :index:`core.runner <configuration variable; core.runner>`
+        Short name of the default :term:`runtime`.  Typically set by running
+        one of:
+
+        .. code-block:: none
+
+            nextstrain setup --set-default <runtime>
+            nextstrain check-setup --set-default
+
+        If not set, the :doc:`/runtimes/docker` (``docker``) is used.
diff --git a/doc/config/paths.rst b/doc/config/paths.rst
@@ -0,0 +1,48 @@
+===================
+Paths configuration
+===================
+
+Nextstrain CLI uses various local filesystem paths for config and runtime data.
+If necessary, the defaults can be overridden by environment variables.
+
+.. envvar:: NEXTSTRAIN_HOME
+
+    Directory for config and other application data.  Used as the basis for all
+    other paths.
+
+    Default is :file:`~/.nextstrain/`, assuming a home directory is
+    discernable.  If not, :file:`.nextstrain` (i.e. in the current directory)
+    is used as a last resort.
+
+.. envvar:: NEXTSTRAIN_CONFIG
+
+    File for :doc:`configuration </config/file>`.
+
+    Default is :file:`{${NEXTSTRAIN_HOME}}/config`.
+
+.. envvar:: NEXTSTRAIN_SECRETS
+
+    File for secrets (e.g. nextstrain.org tokens) managed by
+    :doc:`/commands/login` and :doc:`/commands/logout`.
+
+    Default is :file:`{${NEXTSTRAIN_HOME}}/secrets`.
+
+.. envvar:: NEXTSTRAIN_LOCK
+
+    File for serializing access to other config files to prevent corruption and
+    other bugs.
+
+    Default is :file:`{${NEXTSTRAIN_HOME}}/lock`.
+
+.. envvar:: NEXTSTRAIN_RUNTIMES
+
+    Directory for runtime-specific data, e.g. Singularity images or a Conda
+    environment.  Each runtime uses a subdirectory within here.
+
+    Default is :file:`{${NEXTSTRAIN_HOME}}/runtimes/`.
+
+.. envvar:: NEXTSTRAIN_SHELL_HISTORY
+
+    File for preserving command history across :doc:`/commands/shell` invocations.
+
+    Default is :file:`{${NEXTSTRAIN_HOME}}/shell-history`.
diff --git a/doc/glossary.rst b/doc/glossary.rst
@@ -36,17 +36,4 @@ documentation.
         environment) in which Nextstrain CLI expects to find and execute other
         Nextstrain programs.
 
-        Current runtimes:
-
-          - Docker with the `nextstrain/base image <https://hub.docker.com/r/nextstrain/base>`_
-          - Conda with the `nextstrain-base meta-package <https://anaconda.org/nextstrain/nextstrain-base>`__
-          - Ambient
-          - AWS Batch with the `nextstrain/base image`_
-
-        Each runtime provides specific versions of Nextstrain's software
-        components, like Augur and Auspice.
-
-        Runtimes are managed (maintained, tested, versioned, released) by the
-        Nextstrain team, except for the ambient runtime.  The ambient runtime
-        is special in that it's whatever computing environment in which
-        Nextstrain CLI itself is running (i.e. managed by the user).
+        See the :doc:`runtimes overview </runtimes/index>`.
diff --git a/doc/index.rst b/doc/index.rst
@@ -11,10 +11,10 @@ The Nextstrain CLI is a program called ``nextstrain``.  It aims to provide a
 consistent way to run and visualize pathogen builds and access Nextstrain
 components like :doc:`Augur <augur:index>` and :doc:`Auspice <auspice:index>`
 computing platforms, such as
-:ref:`installation/docker`,
-:ref:`installation/conda`,
-:ref:`installation/singularity`, and
-:ref:`installation/aws-batch`.
+:doc:`/runtimes/docker`,
+:doc:`/runtimes/conda`,
+:doc:`/runtimes/singularity`, and
+:doc:`/runtimes/aws-batch`.
 
 .. note::
     If you're unfamiliar with Nextstrain builds, you may want to follow our
@@ -64,6 +64,24 @@ Table of Contents
     nextstrain.org <remotes/nextstrain.org>
     Amazon S3 <remotes/s3>
 
+.. toctree::
+    :caption: Runtimes
+    :titlesonly:
+
+    Overview <runtimes/index>
+    Docker <runtimes/docker>
+    Conda <runtimes/conda>
+    Singularity <runtimes/singularity>
+    AWS Batch <runtimes/aws-batch>
+    Ambient <runtimes/ambient>
+
+.. toctree::
+    :caption: Configuration
+    :titlesonly:
+
+    Config file <config/file>
+    Paths <config/paths>
+
 .. toctree::
     :caption: Development
     :titlesonly:

diff --git a/doc/installation.rst b/doc/installation.rst
@@ -106,156 +106,29 @@ The version you get will probably be different than the one shown in the
 example above.
 
 
-Runtimes
-========
+A Nextstrain runtime
+====================
 
-.. XXX TODO: Move this heading and subheadings (with modification) to their own
-   top-level doc section (e.g. like Remotes).
-     -trs, 12 Jan 2023
+If you intend to run commands like :doc:`/commands/build` and
+:doc:`/commands/view`, then you'll need to set up at least one :term:`runtime`.
+See the :doc:`runtimes overview </runtimes/index>` for a comparison of the
+options and brief set up instructions for each.  Runtime set up typically
+concludes by running:
 
-The Nextstrain CLI provides a consistent interface and computing environment
-for running and visualizing Nextstrain pathogen builds across several different
-computing platforms, such as
-:ref:`installation/docker`,
-:ref:`installation/conda`,
-:ref:`installation/singularity`, and
-:ref:`installation/aws-batch`.
-
-We call the provided computing environments the :term:`Nextstrain runtimes
-<docs:runtime>`.  Each runtime provides specific versions of Nextstrain's
-software components, like `Augur <https://github.com/nextstrain/augur>`__ and
-`Auspice <https://github.com/nextstrain/auspice>`__.
-
-At least one of these runtimes must be setup in order for many of
-``nextstrain``'s subcommands to work, such as ``nextstrain build`` and
-``nextstrain view``.
-
-The default runtime is Docker, using the `nextstrain/base`_ container image.
-Containers provide a tremendous amount of benefit for scientific workflows by
-isolating dependencies and increasing reproducibility. However, they're not
-always appropriate, so a Conda runtime, Singularity runtime, and "ambient"
-runtime are also supported.  The installation and setup of supported runtimes
-is described below.
-
-.. _nextstrain/base: https://github.com/nextstrain/docker-base
-
-.. _installation/docker:
-
-Docker
-------
-
-`Docker <https://docker.com>`__ is a very popular container system
-freely-available for all platforms. When you use Docker with the Nextstrain
-CLI, you don't need to install any other Nextstrain software dependencies as
-validated versions are already bundled into a container image by the Nextstrain
-team.
-
-On macOS, download and install `Docker Desktop`_, also known previously as
-"Docker for Mac".
-
-On Linux, install Docker with the standard package manager. For example, on
-Ubuntu, you can install Docker with ``sudo apt install docker.io``.
-
-On Windows, install `Docker Desktop`_ with its support for a WSL2 backend.
-
-Once you've installed Docker, proceed with :ref:`checking your setup
-<installation/check-setup>`.
-
-.. _Docker Desktop: https://www.docker.com/products/docker-desktop
-
-.. _installation/conda:
-
-Conda
------
-
-`Conda <https://docs.conda.io/en/latest/miniconda.html>`__ is a very popular
-packaging system freely-available for all platforms. When you use Nextstrain
-CLI's built-in Conda support, you don't need to install any other Nextstrain
-software dependencies yourself as they're automatically managed in an isolated
-location (isolated even from other Conda environments you may manage yourself).
-
-On macOS and Linux, run ``nextstrain setup conda`` to get started.
-
-This runtime is not directly supported on Windows, but you can use `WSL2
-<https://docs.microsoft.com/en-us/windows/wsl/wsl2-index>`__ to "switch" to
-Linux and run the above setup command.
-
-.. _installation/singularity:
-
-Singularity
------------
-
-Singularity is a container system freely-available for Linux platforms.  It is
-commonly available on institutional HPC systems as an alternative to Docker,
-which is often not supported on such systems.  When you use Singularity with
-the Nextstrain CLI, you don't need to install any other Nextstrain software
-dependencies as validated versions are already bundled into a container image
-by the Nextstrain team.
-
-Run ``nextstrain setup singularity`` to get started.
-Singularity version 3.0.0 or newer is required, but we recommend at least
-version 3.10.0 or newer when possible.
-
-Note that the Singularity project forked into two separate projects in late
-2021: `SingularityCE`_ under `Sylabs`_ and `Apptainer`_ under the `Linux
-Foundation`_.  Either fork should work with Nextstrain CLI, as both projects
-still provide very similar interfaces and functionality via the ``singularity``
-command.  You can read `Sylab's announcement`_ and `Apptainer's announcement`_
-for more information on the fork.
-
-.. _SingularityCE: https://sylabs.io/singularity/
-.. _Sylabs: https://sylabs.io/
-.. _Apptainer: https://apptainer.org
-.. _Linux Foundation: https://www.linuxfoundation.org/
-.. _Sylab's announcement: https://sylabs.io/2022/06/singularityce-is-singularity/
-.. _Apptainer's announcement: https://apptainer.org/news/community-announcement-20211130
-
-Ambient
--------
-
-The "ambient" runtime allows you to use the Nextstrain CLI with your own ambient
-setup, for when you cannot or do not want to have Nextstrain CLI manage its own
-runtime.
-
-However, you will need to make sure all of the Nextstrain software dependencies
-are available locally or "ambiently" on your computer. A common way to do this
-is by manually using `Conda <https://docs.conda.io/en/latest/miniconda.html>`__
-to manage your own environment that includes the required software, however
-you're responsible for making sure the correct software is installed and kept
-up-to-date. It is also possible to install the required Nextstrain software
-`Augur <https://github.com/nextstrain/augur>`__ and `Auspice
-<https://github.com/nextstrain/auspice>`__ and their dependencies manually,
-although this is not recommended.
-
-Once you've installed dependencies, proceed with :ref:`checking your setup
-<installation/check-setup>`.
-
-.. _installation/aws-batch:
-
-AWS Batch
----------
-
-`AWS Batch <https://aws.amazon.com/batch/>`__ is an advanced computing
-platform which allows you to launch and monitor Nextstrain builds in the
-cloud from the comfort of your own computer. The same image used by the local
-Docker runtime is used by AWS Batch, making your builds more reproducible, and
-builds have access to computers with very large CPU and memory allocations if
-necessary.
+.. code-block:: bash
 
-The initial setup is quite a bit more involved, but :doc:`detailed instructions
-<aws-batch>` are available.
+    nextstrain setup <runtime>
 
-Once you've setup AWS, proceed with :ref:`checking your setup
-<installation/check-setup>`.
 
 .. _installation/check-setup:
 
 Checking your setup
 ===================
 
-After installation and setup, run ``nextstrain check-setup --set-default`` to
-ensure everything works and automatically pick an appropriate default runtime
-based on what's available. You should see output similar to the following:
+After installation and runtime set up, run ``nextstrain check-setup
+--set-default`` to ensure everything works and automatically pick an
+appropriate default runtime based on what's available. You should see output
+similar to the following:
 
 .. code-block:: console