Document OCI-SIF overlays

* overlay create
* overlay sync
* overlay seal

Fixes #241
Fixes #242

new.rst

@@ -19,7 +19,20 @@ OCI-mode
 - A new ``--layer-format tar`` flag for ``singularity push`` allows layers in an
   OCI-SIF image to be pushed to ``library://`` and ``docker://`` registries with
   layers in the standard OCI tar format. Images pushed with ``--layer-format``
-  tar can be pulled and run by other OCI runtimes. See :ref:`sec:layer-format`
+  tar can be pulled and run by other OCI runtimes. See :ref:`sec:layer-format`.
+- Persistent overlays embedded in OCI-SIF files. See :ref:`overlay-oci-sif`.
+
+  - A writable overlay can be added to an OCI-SIF file with the ``singularity
+    overlay create`` command. The overlay will be applied read-only, by default,
+    when executing the OCI-SIF. To write changes to the container into the
+    overlay, use the ``--writable`` flag. 
+  - A writable overlay is added to an OCI-SIF file as an ext3 format layer,
+    appended to the encapsulated OCI image. After the overlay has been modified,
+    use the ``singularity overlay sync`` command to synchronize the OCI digests with
+    the overlay content.
+  - A new ``singularity overlay seal`` command converts a writable overlay inside
+    an OCI-SIF image into a read-only squashfs layer. This seals changes made to
+    the image via the overlay, so that they are permanent.
 
 *******
 Runtime

oci_runtime.rst

@@ -242,6 +242,8 @@ The following features are supported in ``--oci`` mode:
   * Multiple simultaneous overlays are supported (though all but one must be
     mounted as read-only).
 
+* ``--writable`` to write to an :ref:`embedded overlay <overlay-oci-sif>`.
+
 * ``--cwd`` (synonym: ``--pwd``) to set a custom starting working-directory for
   the container.
 
@@ -469,16 +471,6 @@ addition to the image manifest and image config:
   cannot be executed using {Singularity} 4.0.
 
 
-Future Development
-==================
-
-Subsequent 4.x releases of {Singularity} will aim to improve OCI-mode to address
-missing features vs native mode, and improve the utility of the OCI-SIF format.
-Development items include:
-
-- Support for overlays embedded in OCI-SIF files.
-- Export / push from OCI-SIF to standard OCI format with `.tar.gz` layers.
-
 .. _sec:cdi:
 
 ********************************

persistent_overlays.rst

@@ -2,25 +2,19 @@
  Persistent Overlays
 #####################
 
-Persistent overlay directories allow you to overlay a writable file
-system on an immutable read-only container for the illusion of
-read-write access. You can run a container and make changes, and these
-changes are kept separately from the base container image.
+Persistent overlays allow you to overlay a writable file system on an immutable
+read-only container for the illusion of read-write access. You can run a
+container and make changes, and these changes are kept separately from the base
+container image.
 
 ********
 Overview
 ********
 
-A persistent overlay is a directory or file system image that “sits on
-top” of your immutable SIF container. When you install new software or
-create and modify files the overlay will store the changes.
-
-If you want to use a SIF container as though it were writable, you can
-create a directory, an ext3 file system image, or embed an ext3 file
-system image in SIF to use as a persistent overlay. Then you can specify
-that you want to use the directory or image as an overlay at runtime
-with the ``--overlay`` option, or ``--writable`` if you want to use the
-overlay embedded in SIF.
+A persistent overlay is a directory or file system image that “sits on top” of
+your immutable SIF container. When you install new software or create and modify
+files the overlay will store the changes. These changes will persist after the
+container exits.
 
 If you want to make changes to the image, but do not want them to
 persist, use the ``--writable-tmpfs`` option. This stores all changes in
@@ -33,6 +27,23 @@ container finishes executing.
    ``singularity.conf``. This defaults to 64MiB, and may need to be increased if
    your workflows create larger temporary files.
 
+There are 4 types of persistent overlay:
+
+- An ext3 filesytem image, stored as a separate file from the container image.
+- A directory on the host filesystem.
+- An embedded ext3 overlay inside a native SIF file.
+- An embedded ext3 overlay inside an OCI-Mode OCI-SIF file.
+
+Directories and separate ext3 images can be used in both native and OCI-Mode.
+Embedded overlays in SIF and OCI-SIF files behave differently, so are treated
+separately.
+
+If you want to use a container as though it were writable, you can create a
+directory, an ext3 file system image, or add an overlay to a SIF or OCI-SIF. You
+can then specify that you want to use the directory or image as an overlay at
+runtime with the ``--overlay`` option, or ``--writable`` if you want to use the
+overlay embedded in SIF.
+
 You can use persistent overlays with the following commands:
 
 -  ``run``
@@ -83,12 +94,6 @@ For example:
    $ singularity overlay create --size 1024 --create-dir /var/cache/nginx /tmp/nginx_overlay.img
    $ echo "test" | singularity exec --overlay /tmp/nginx_overlay.img /tmp/nginx.sif sh -c "cat > /var/cache/nginx/test"
 
-.. note::
-
-   Filesystem image overlays are only supported when {singularity} is installed in
-   setuid mode. An unprivileged installation of {Singularity} can create these
-   kinds of overlays, but cannot mount them to the container at runtime.
-
 Sparse overlay images
 ---------------------
 
@@ -173,6 +178,19 @@ into ``overlay.img``.
    correct ``uid`` and ``gid``, but it is not possible to lock down
    permissions so that the overlay is only writable by certain users.
 
+OCI-Mode
+--------
+
+Using a separate filestem image overlay is the same in OCI-Mode as in native
+mode. Use the ``--overlay`` flag as described above. For example to write files
+into an overlay, on top of an Ubuntu container:
+
+.. code::
+
+   $ singularity pull --oci docker://ubuntu:latest
+   $ singularity overlay create --size 1024 /tmp/ext3_overlay.img
+   $ singularity exec --oci --overlay /tmp/ext3_overlay.img ubuntu_latest.oci.sif touch /my-new-file
+
 Directory overlay
 =================
 
@@ -212,14 +230,27 @@ The example below shows the directory overlay in action.
 
    {Singularity} ubuntu.sif:~> exit
 
+OCI-Mode
+--------
+
+Using a directory overlay is the same in OCI-Mode as in native mode. Use the
+``--overlay`` flag as described above. For example to write files into a
+directory overlay, on top of an Ubuntu container:
+
+.. code::
+
+   $ singularity pull --oci docker://ubuntu:latest
+   $ mkdir my_overlay
+   $ singularity exec --oci --overlay my_overlay/ ubuntu_latest.oci.sif touch /my-new-file
+
 .. _overlay-sif:
 
-Overlay embedded in SIF
-=======================
+Overlay embedded in native SIF
+==============================
 
-It is possible to embed an overlay image into the SIF file that holds a
-container. This allows the read-only container image and your
-modifications to it to be managed as a single file.
+It is possible to embed an overlay image into the native SIF file that holds a
+container. This allows the read-only container image and your modifications to
+it to be managed as a single file.
 
 To add a 1 GiB writable overlay partition to an existing SIF image:
 
@@ -296,7 +327,117 @@ container with the ``--writable`` option.
 
 .. code::
 
-   $ sudo singularity shell --writable ubuntu_latest.sif
+   $ singularity shell --writable ubuntu_latest.sif
+
+.. _overlay-oci-sif:
+
+Overlay embedded in OCI-SIF
+===========================
+
+``singularity overlay create`` can be used to add an overlay to an OCI-SIF file,
+used in OCI-Mode, in the same way as for native mode SIF files. For example, to
+add a 1 GiB writable overlay to an existing OCI-SIF image:
+
+.. code::
+
+   $ singularity overlay create --size 1024 ubuntu_latest.oci.sif
+
+The embedded overlay is then always applied, and can be written to when a
+container is run with the ``--writable`` flag:
+
+.. code::
+
+   $ singularity exec --oci --writable ubuntu_latest.oci.sif touch /my-new-file
+
+   $ singularity exec --oci ubuntu_latest.oci.sif ls -lah /my-new-file
+   -rw-r--r--. 1 ubuntu ubuntu 0 Sep  3 15:52 /my-new-file
+
+Synchronizing an overlay
+------------------------
+
+An overlay in an OCI-SIF image is created by adding an additional layer to the
+OCI image in the OCI-SIF file. This layer begins as an empty ext3 filesystem and
+its details, including the digest of its content, are written into the OCI image
+manifest stored in the OCI-SIF.
+
+When you write into the overlay, the true digest of the ext3 filesystem no
+longer matches the digest stored in the OCI image manifest. This is not an issue
+when an OCI-SIF is being used locally with {Singularity}, but the OCI image
+manifest must be synchronized with the overlay content before the image is
+shared.
+
+You can manually synchronize the OCI image manifest with the content of the
+overlay using the ``singularity overlay sync`` command:
+
+.. code::
+
+   $ singularity overlay sync ubuntu_latest.oci.sif
+
+In normal use this operation is not required, because {Singularity} will
+automatically synchronize the image before pushing it to the container library,
+or an OCI registry:
+
+.. code::
+
+   $ singularity push -U ubuntu_latest.oci.sif library://dtrudg-sylabs-2/oci-overlay/test:latest
+   WARNING: Skipping container verification
+   INFO:    Pushing an OCI-SIF to the library OCI registry. Use `--oci` to pull this image.
+   INFO:    Synchronizing overlay digest to OCI image.
+
+Note the ``INFO`` message showing that synchronization was performed.
+
+Pushing & pulling images with overlays
+--------------------------------------
+
+OCI-SIF images containing overlays can be pushed to the container library, and
+other OCI registries, in the same manner as any other OCI-SIF image. Layers in
+an OCI-SIF file are pushed 'as-is' by default. Unless you specify
+``--layer-format tar`` as an option to ``singularity push``, an overlay will be
+pushed as an ext3 layer and remains writable when the image is subsequently
+pulled with {Singularity}. Note that other OCI runtimes cannot use ext3 layers,
+and will fail to pull the image.
+
+To push an an image with ``--layer-format tar``, for execution with other OCI
+runtimes, you must seal the overlay first (see below).
+
+When pulling a standard OCI image into an OCI-SIF with ``singularity pull``,
+default behaviour is to squash all of the remote image layers into a single
+SquashFS layer in the OCI-SIF. The ``--keep-layers`` flag can be used to prevent
+layers being squashed.
+
+If a remote image contains an overlay, the overlay is never squashed by
+``singularity pull``. By default all read-only layers are squashed and the
+overlay is preserved. With ``--keep-layers`` all layers including the overlay
+are preserved.
+
+Sealing an overlay
+------------------
+
+The ``singularity overlay seal`` command will seal an overlay in an OCI-SIF
+image by converting it to a standard read-only layer. This is useful when:
+
+- You have made important changes to an image using an overlay but want to
+  deploy the resulting image in production, read-only.
+- You want to push the image to a registry with ``--layer-format tar``, so it
+  can be run by other OCI runtimes. These other runtimes cannot handle ext3
+  writable overlay layers.
+
+To seal the image with overlay that was created above:
+
+.. code::
+
+   $ singularity overlay seal ubuntu_latest.oci.sif 
+
+Note that the file created in the overlay is still present in the container
+image, but the image is no longer writable:
+
+.. code::
+
+   $ singularity exec --oci ubuntu_latest.oci.sif ls -lah /my-new-file
+   -rw-r--r--. 1 ubuntu ubuntu 0 Sep  3 15:52 /my-new-file
+
+   $ singularity exec --oci --writable ubuntu_latest.oci.sif touch /another-new-file
+   FATAL:   image oci-sif:ubuntu_latest.oci.sif does not contain a writable overlay
 
 Final note
 ==========