Skip to content

Commit

Permalink
Add OCI-SIF library push/pull
Browse files Browse the repository at this point in the history
Also take the opportunity to change Sylabs Cloud -> Singularity
Container Services (SCS).

Fixes #188
  • Loading branch information
dtrudg committed Sep 8, 2023
1 parent 3c4f3d7 commit 219ad95
Show file tree
Hide file tree
Showing 4 changed files with 147 additions and 24 deletions.
142 changes: 133 additions & 9 deletions cloud_library.rst
Original file line number Diff line number Diff line change
@@ -1,21 +1,20 @@
.. _cloud_library:

#############
Cloud Library
#############
###########
SCS Library
###########

********
Overview
********

The Sylabs Cloud Library is the place to :ref:`push <push>` your
containers to the cloud so other users can :ref:`pull <pull>`,
The Singularity Container Services (SCS) Library is the place to :ref:`push
<push>` your containers to the cloud so other users can :ref:`pull <pull>`,
:ref:`verify <signNverify>`, and use them.

The Sylabs Cloud also provides a :ref:`Remote Builder <remote_builder>`,
allowing you to build containers on a secure remote service. This is
convenient so that you can build containers on systems where you do not
have root privileges.
SCS also provides a :ref:`Remote Builder <remote_builder>`, allowing you to
build containers on a secure remote service. This is convenient so that you can
build containers on systems where you do not have root privileges.

.. _make_a_account:

Expand Down Expand Up @@ -104,6 +103,92 @@ Note that when you push to a library that supports it, {Singularity} 3.7
and above will report your quota usage and the direct URL to view the
container in your web browser.

OCI-SIF Images
==============

If you are using {Singularity} 4's new OCI-mode, you may wish to push OCI-SIF
images to a ``library://`` destination. The standard ``push`` command can be
used, and {Singularity} will perform the push as an OCI image.

Instead of uploading the container as a single SIF file, the OCI configuration
and layer blobs that are encapsulated in the OCI-SIF will be uploaded to the OCI
registry that sits behind the SCS / Singularity Enterprise Library.

.. note::

The OCI image specification doesn't support SIF signatures, or any additional
partitions that can be added to SIF (including OCI-SIF) files by
{Singularity}.

If you have signed an OCI-SIF image locally, the signature(s) will not be
pushed to the library. You may wish to push the OCI-SIF, as a single file, to
an OCI registry using the ``oras://`` protocol instead.

Pushing OCI-SIF containers to the library in this manner means that they can be
accessed by other OCI tools. For example, you can use the `Skopeo CLI tool
<https://github.com/containers/skopeo>`__ to examine the image in the registry
after it has been pushed. First, ``push`` an OCI-SIF to the SCS ``library://``.
The ``-U`` option is needed because the image is unsigned.

.. code::
$ singularity push -U alpine_latest.oci.sif library://example/userdoc/alpine:latest
WARNING: Skipping container verification
INFO: Pushing an OCI-SIF to the library OCI registry. Use `--oci` to pull this image.
Now use ``skopeo`` to access the image in the library. This requires
authentication, which is handled automatically when you use ``singularity
push``. For other tools, {Singularity} provides a command ``singularity remote
get-login-password`` that will provide a token that we can use to login to
``registry.sylabs.io``, which is the address of the OCI registry backing the SCS
library.

.. code::
$ singularity remote get-login-password | \
skopeo login -u example --password-stdin registry.sylabs.io
Login Succeeded!
Finally, use ``skopeo inspect`` to examine the image pushed earlier:

.. code::
$ skopeo inspect docker://registry.sylabs.io/example/userdoc/alpine:latest
{
"Name": "registry.sylabs.io/example/userdoc/alpine",
"Digest": "sha256:d08ad9745675812310727c0a99a4472b82fb1cc81e5c42ceda023f1bc35ca34a",
"RepoTags": [
"latest"
],
"Created": "2023-08-07T20:16:26.309461618Z",
"DockerVersion": "",
"Labels": null,
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:a0c5ced3a57bd1d0d71aaf4a0ea6131d5f163a4a8c5355468c18d4ef006f5d7d"
],
"LayersData": [
{
"MIMEType": "application/vnd.sylabs.image.layer.v1.squashfs",
"Digest": "sha256:a0c5ced3a57bd1d0d71aaf4a0ea6131d5f163a4a8c5355468c18d4ef006f5d7d",
"Size": 3248128,
"Annotations": null
}
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
}
Because the OCI-SIF was pushed as an OCI image, ``skopeo inspect`` is able to
show the image configuration. This is not possible for non-OCI-SIF images:

.. code::
$ skopeo inspect docker://registry.sylabs.io/library/default/alpine:latest
FATA[0001] unsupported image-specific operation on artifact with type "application/vnd.sylabs.sif.config.v1+json"
.. _pull:

*******************
Expand Down Expand Up @@ -165,6 +250,45 @@ etc...
automatically, but it's good practice to always specify your output
file.

OCI-SIF Images
==============

If you are using {Singularity} 4's new OCI-mode and have pushed OCI-SIF
containers to the SCS library, they are stored as OCI images in the OCI registry
that backs the library. You can pull these images with the standard ``pull``
command:

.. code::
$ singularity pull library://sylabs/test/alpine-oci-sif:latest
INFO: sylabs/test/alpine-oci-sif:latest is an OCI image, attempting to fetch as an OCI-SIF
Getting image source signatures
Copying blob af32528d4445 done
Copying config a5d222bd0d done
Writing manifest to image destination
INFO: Writing OCI-SIF image
INFO: Cleaning up.
WARNING: integrity: signature not found for object group 1
WARNING: Skipping container verification
Note that {Singularity} detects the image is an OCI image, and automatically
retrieves it to an OCI-SIF file.

If the image was a non-OCI-SIF, built for {Singularity}'s default native mode,
then it would be retrieved as-is. To ensure that an image retrieved from a
``library://`` URI is an OCI-SIF, use the ``--oci`` flag. This will produce an
error if a non-OCI-SIF is pulled:

.. code::
$ singularity pull --oci library://sylabs/examples/ruby
Getting image source signatures
Copying blob a21814eefb7f done
Copying config 5211e7986c done
Writing manifest to image destination
INFO: Cleaning up.
FATAL: While pulling library image: error fetching image: while creating OCI-SIF: while checking OCI image: json: cannot unmarshal string into Go struct field ConfigFile.rootfs of type v1.RootFS
**************************
Verify/Sign your Container
**************************
Expand Down
16 changes: 8 additions & 8 deletions endpoint.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Overview
********

The ``remote`` command group allows users to manage the remote endpoint(s) that
{Singularity} interacts with (the public Sylabs Cloud, or a local Singularity
{Singularity} interacts with (Singularity Container Services, or a local Singularity
Enterprise installation).

.. note::
Expand All @@ -21,11 +21,11 @@ Enterprise installation).
to dedicated top-level command groups: :ref:`registry<registry>` and
:ref:`keyserver<keyserver>`, respectively.

*******************
Public Sylabs Cloud
*******************
******************************
Singularity Container Services
******************************

Sylabs introduced the online `Sylabs Cloud <https://cloud.sylabs.io/home>`_ to
Sylabs introduced `Singularity Container Services (SCS) <https://cloud.sylabs.io/home>`_ to
enable users to `Create <https://cloud.sylabs.io/builder>`_ container images,
`Secure <https://cloud.sylabs.io/keystore?sign=true>`_ them, and `Share
<https://cloud.sylabs.io/library>`_ them with others.
Expand Down Expand Up @@ -70,7 +70,7 @@ If you see any errors, you may need to check if your system requires the setting
of environment variables for a network proxy, or if a firewall may be blocking
access to ``*.sylabs.io``. Consult your system administrator.

You can interact with the public Sylabs Cloud using various {Singularity}
You can interact with Singularity Container Services using various {Singularity}
commands:

`pull
Expand All @@ -96,7 +96,7 @@ commands:

.. note::

Using the commands listed above will *not* interact with the Sylabs Cloud if
Using the commands listed above will *not* interact with SCS if
given URIs beginning with ``docker://``, ``oras://`` or ``shub://``.

*************************
Expand All @@ -107,7 +107,7 @@ Users can set up and switch between multiple remote endpoints, which will be
stored in their ``~/.singularity/remote.yaml`` file. Alternatively, remote
endpoints can be set on a system-wide basis by an administrator.

A remote endpoint may be the public Sylabs Cloud, a private installation of
A remote endpoint may be Singularity Container Services, a private installation of
Singularity Enterprise, or any community-developed service that is
API-compatible.

Expand Down
2 changes: 1 addition & 1 deletion index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ process of building, signing, and sharing your containers.
Remote Endpoints <endpoint>
Keyserver Management <keyserver>
OCI Registries <registry>
Sylabs Cloud Library <cloud_library>
SCS Library <cloud_library>

**************
Advanced Usage
Expand Down
11 changes: 5 additions & 6 deletions security.rst
Original file line number Diff line number Diff line change
Expand Up @@ -197,12 +197,11 @@ signed. The signature travels as part of the SIF image itself so that it
is always possible to verify that the image has not been tampered with
or corrupted.

We use private PGP keys to create a container signature, and the
corresponding public keys to verify the container. Verification of
signed containers happens automatically in ``singularity pull`` commands
against the Sylabs Cloud Container Library. A keystore in the Sylabs
Cloud makes it easier to share and obtain public keys for container
verification.
We use private PGP keys to create a container signature, and the corresponding
public keys to verify the container. Verification of signed containers happens
automatically in ``singularity pull`` commands against the Singularity Container
Services (SCS) Library. The SCS keystore makes it easier to share and obtain
public keys for container verification.

A container may be signed once, by a trusted individual who approves its use. It
could also be signed with multiple keys to signify it has passed each step in a
Expand Down

0 comments on commit 219ad95

Please sign in to comment.