diff --git a/LICENSE b/LICENSE index c8866a4..a866cfe 100644 --- a/LICENSE +++ b/LICENSE @@ -1,5 +1,5 @@ Copyright (c) 2017, SingularityWare, LLC. All rights reserved. -Copyright (c) 2018-2022, Sylabs, Inc. All rights reserved. +Copyright (c) 2018-2023, Sylabs, Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: diff --git a/cgroups.rst b/cgroups.rst index 9c938f7..4738913 100644 --- a/cgroups.rst +++ b/cgroups.rst @@ -61,6 +61,8 @@ satisfy these criteria by default. On older distributions support can often be enabled. Consult the admin documentation or speak to your system administrator about this. +.. _cgroup_flags: + *************************** Command Line Limit Flags *************************** diff --git a/conf.py b/conf.py index e33c2db..053f996 100755 --- a/conf.py +++ b/conf.py @@ -45,7 +45,7 @@ # General information about the project. project = u'SingularityCE User Guide' author = u'SingularityCE Project Contributors' -copyright = u'2017-2022, Sylabs Inc & Project Contributors' +copyright = u'2017-2023, Sylabs Inc & Project Contributors' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the diff --git a/docpage.png b/docpage.png index 81706fc..40bc743 100644 Binary files a/docpage.png and b/docpage.png differ diff --git a/index.rst b/index.rst index 9615166..8cc011e 100644 --- a/index.rst +++ b/index.rst @@ -85,7 +85,7 @@ networking and security configuration. Bind Paths and Mounts Persistent Overlays - Running Services + Instances - Running Services Environment and Metadata Plugins Security Options diff --git a/running_services.rst b/running_services.rst index c6296a3..ab2193d 100644 --- a/running_services.rst +++ b/running_services.rst @@ -1,59 +1,60 @@ .. _running_services: -################## - Running Services -################## - -There are :ref:`different ways ` in which you can run -{Singularity} containers. If you use commands like ``run``, ``exec`` and -``shell`` to interact with processes in the container, you are running -{Singularity} containers in the foreground. {Singularity}, also lets you -run containers in a "detached" or "daemon" mode which can run different -services in the background. A "service" is essentially a process running -in the background that multiple different clients can use. For example, -a web server or a database. To run services in a {Singularity} container -one should use *instances*. A container instance is a persistent and -isolated version of the container image that runs in the background. - -********** - Overview -********** +############################ +Instances - Running Services +############################ + +{Singularity} is most commonly used to run containers interactively, or in a +batch job, where the container runs in the foreground, performs some work, and +then exits. There are :ref:`different ways ` in which you can run +{Singularity} containers in the foreground. If you use ``run``, ``exec`` and +``shell`` to interact with processes in the container, then you are running +{Singularity} containers in the foreground. + +{Singularity}, also allows you to run containers in a "detached" or "daemon" +mode where the container runs a service. A "service" is essentially a process +running in the background that multiple different clients can use. For example, +a web server or a database. + +A {Singularity} container running a service in the background is called an +*instance*, to distinguish it from the default mode which runs containers in the +foreground. + +******** +Overview +******** .. _sec:instances: -{Singularity} v2.4 introduced the concept of *instances* allowing users -to run services in {Singularity}. This page will help you understand -instances using an elementary example followed by a more useful example -running an NGINX web server using instances. In the end, you will find a -more detailed example of running an instance of an API that converts URL -to PDFs. +This page will help you understand instances using an elementary example +followed by a more useful example running an NGINX web server using instances. +At the end, you will find a more detailed example of running an instance of a +service exposing an API that converts URL to PDFs. -To begin with, suppose you want to run an NGINX web server outside of a -container. On Ubuntu, you can simply install NGINX and start the service -by: +To run a service, such as a web server, outside of a container you would +typically install the package for the web server and then instruct systemd +(which manages system services on most Linux distributions) to start it. E.g. .. code:: - $ sudo apt-get update && sudo apt-get install -y nginx + $ sudo yum install nginx + $ sudo systemctl enable --now nginx - $ sudo service nginx start +If you were to attempt this in a container, it's likely that it will not work as +expected. You cannot use systemd to run services in a container, by default. It +expects various conditions that are satisfied on the host system, but not inside +containers. -If you were to do something like this from within a container you would -also see the service start, and the web server running. But then if you -were to exit the container, the process would continue to run within an -unreachable mount namespace. The process would still be running, but you -couldn't easily kill or interface with it. This is a called an orphan -process. {Singularity} instances give you the ability to handle services -properly. +Instead, you should run NGINX in the foreground *inside* the container, but then +run the container in the background, as an *instance*. -************************************** - Container Instances in {Singularity} -************************************** +************************************ +Container Instances in {Singularity} +************************************ -For demonstration, let's use an easy (though somewhat useless) example -of `alpine_latest.sif -`_ -image from the `container library `_: +To demonstrate the basices of instances, let's use an easy (though somewhat +useless) example, using an Alpine Linux image from the `container library +`_: .. code:: @@ -62,6 +63,9 @@ image from the `container library `_: The above command will save the alpine image from the Container Library as ``alpine_latest.sif``. +Starting Instances +================== + To start an instance, you should follow this procedure : .. code:: @@ -71,8 +75,8 @@ To start an instance, you should follow this procedure : $ singularity instance start alpine_latest.sif instance1 This command causes {Singularity} to create an isolated environment for -the container services to live inside. One can confirm that an instance -is running by using the ``instance list`` command like so: +the container services to live inside. You can confirm that an instance +is running by using the ``instance list`` command: .. code:: @@ -83,11 +87,10 @@ is running by using the ``instance list`` command like so: .. note:: - The instances are linked with your user account. So make sure to run - *all* instance commands either with or without the ``sudo`` - privilege. If you ``start`` an instance with sudo then you must - ``list`` it with sudo as well, or you will not be able to locate the - instance. + Instances are linked to the user account that started them. This means that + if you use ``sudo`` to start an instance as ``root``, you will need to use + ``sudo`` for all commands managing that instance. ``instance list`` will not + show instances started by other users. If you want to run multiple instances from the same image, it’s as simple as running the command multiple times with different instance @@ -100,7 +103,7 @@ repeated. $ singularity instance start alpine_latest.sif instance3 -And again to confirm that the instances are running as we expected: +We now have 3 instances, all using the same image: .. code:: @@ -111,7 +114,7 @@ And again to confirm that the instances are running as we expected: instance2 22443 /home/dave/instances/alpine_latest.sif instance3 22493 /home/dave/instances/alpine_latest.sif -You can also filter the instance list by supplying a pattern: +You can filter the instance list by supplying a pattern: .. code:: @@ -120,19 +123,33 @@ You can also filter the instance list by supplying a pattern: INSTANCE NAME PID IP IMAGE instance2 22443 /home/dave/instances/alpine_latest.s -You can use the ``singularity run/exec`` commands on instances: +When an instance is started, it will begin to run the ``%startscript`` from the +container's :ref:`definition file ` in the background. If +there is no ``%startscript`` the container will stay idle in the background. + +Interacting With Instances +========================== + +Although an instance runs its ``%startscript`` (if there is one) in the +background, you can also interact with it in the foreground, by referring to it +with an ``instance://`` URI, where ```` is replaced with the +instance name. + +To run a specific command against an instance, in the foreground, use +``singularity exec``: .. code:: - $ singularity run instance://instance1 + $ singularity exec instance://instance1 cat /etc/os-release - $ singularity exec instance://instance2 cat /etc/os-release +Similarly, you can use ``singularity run`` to run the ``%runscript`` for the +container, against a running instance: -When using ``run`` with an instance URI, the ``runscript`` will be -executed inside of the instance. Similarly with ``exec``, it will -execute the given command in the instance. +.. code:: -If you want to poke around inside of your instance, you can do a normal + $ singularity run instance://instance2 + +If you want to poke around inside of your instance, you can use the normal ``singularity shell`` command, but give it the instance URI: .. code:: @@ -141,6 +158,9 @@ If you want to poke around inside of your instance, you can do a normal Singularity> +Stopping Instances +================== + When you are finished with your instance you can clean it up with the ``instance stop`` command as follows: @@ -150,7 +170,7 @@ When you are finished with your instance you can clean it up with the If you have multiple instances running and you want to stop all of them, you can do so with a wildcard or the --all flag. The following three -commands are all identical. +commands are identical. .. code:: @@ -162,17 +182,17 @@ commands are all identical. .. note:: - Note that you must escape the wildcard with a backslash like this - ``\*`` to pass it properly. + You must escape the wildcard with a backslash ``\*`` to pass it properly + through your shell. -************************************** - Nginx “Hello-world” in {Singularity} -************************************** +************************************ +Nginx “Hello-world” in {Singularity} +************************************ The above example, although not very useful, should serve as a fair introduction to the concept of {Singularity} instances and running -services in the background. The following illustrates a more useful -example of setting up a sample NGINX web server using instances. First +containers in the background. We will now look at a more useful +example of setting up an NGINX web server using instances. First we will create a basic :ref:`definition file ` (let's call it nginx.def): @@ -180,15 +200,15 @@ call it nginx.def): Bootstrap: docker From: nginx - Includecmd: no %startscript nginx This downloads the official NGINX Docker container, converts it to a -{Singularity} image, and tells it to run NGINX when you start the -instance. Since we’re running a web server, we’re going to run the -following commands as root. +{Singularity} image, and tells it to run the ``nginx`` command when you start +the instance. Because we are running a web server, which defaults to listening +on privileged port 80, we’re going to run the following instance commands as +root, using ``sudo``. .. code:: @@ -196,15 +216,14 @@ following commands as root. $ sudo singularity instance start --writable-tmpfs nginx.sif web -.. note:: - - The above ``start`` command requires ``sudo`` because we are running - a web server. Also, to let the instance write temporary files during - execution, you should use ``--writable-tmpfs`` while starting the - instance. +The ``--writable-tmpfs`` option is needed, because NGINX will attempt to write +some files when it starts up. ``--writable-tmpfs`` allows these to be written +to a temporary, in-memory location, that will be removed when the instance is +stopped. -Just like that we’ve downloaded, built, and run an NGINX {Singularity} -image. And to confirm that it’s correctly running: +Just like that we’ve downloaded, built, and run an NGINX {Singularity} image. We +can confirm it's running using the curl tool, to fetch the web page that is now +being hosted by NGINX. .. code:: @@ -236,43 +255,34 @@ image. And to confirm that it’s correctly running: -Visit localhost on your browser, you should see a Welcome message! +You could also visit ``http://localhost`` in a web browser, if you are running +the instance from a desktop session. -********************** - Putting all together -********************** +****************** +API Server Example +****************** -In this section, we will demonstrate an example of packaging a service -into a container and running it. The service we will be packaging is an -API server that converts a web page into a PDF, and can be found `here -`__. You can build the -image by following the steps described below or you can just download -the final image directly from Container Library, simply run: - -.. code:: - - $ singularity pull url-to-pdf.sif library://sylabs/doc-examples/url-to-pdf:latest +Let's now package a useful service into a SIF container, and run it as an +instance. The service we will be packaging is an API server that converts a web +page into a PDF, and can be found `here +`__. Building the image ================== -This section will describe the requirements for creating the definition -file (url-to-pdf.def) that will be used to build the container image. -``url-to-pdf-api`` is based on a Node 8 server that uses a headless -version of Chromium called `Puppeteer -`_. Let’s first choose a base -from which to build our container, in this case the docker image -``node:8`` which comes pre-installed with Node 8 has been used: +To package the Web to PDF service into a SIF container, we must create a +definition file. Let’s first choose a base from which to build our container. In +this case the docker image ``node:8`` which comes pre-installed with Node 8 has +been used: .. code:: singularity Bootstrap: docker From: node:8 - Includecmd: no -Puppeteer also requires a slew of dependencies to be manually installed +The service also requires a slew of dependencies to be manually installed in addition to Node 8, so we can add those into the ``post`` section as -well as the installation script for the ``url-to-pdf``: +well as calling the installation script for the ``url-to-pdf``: .. code:: singularity @@ -290,18 +300,18 @@ well as the installation script for the ``url-to-pdf``: git clone https://github.com/alvarcarto/url-to-pdf-api.git pdf_server cd pdf_server npm install + touch .env chmod -R 0755 . -And now we need to define what happens when we start an instance of the -container. In this situation, we want to run the commands that starts up -the url-to-pdf service: +We need to define what happens when we start an instance of the container by +writing a ``%startscript``. In this situation, we want to run the commands that +start up the url-to-pdf service: .. code:: singularity %startscript cd /pdf_server - # Use nohup and /dev/null to completely detach server process from terminal - nohup npm start > /dev/null 2>&1 < /dev/null & + npm start Also, the ``url-to-pdf`` service requires some environment variables to be set, which we can do in the environment section: @@ -321,7 +331,6 @@ The complete definition file will look like this: Bootstrap: docker From: node:8 - Includecmd: no %post @@ -337,12 +346,12 @@ The complete definition file will look like this: git clone https://github.com/alvarcarto/url-to-pdf-api.git pdf_server cd pdf_server npm install + touch .env chmod -R 0755 . %startscript cd /pdf_server - # Use nohup and /dev/null to completely detach server process from terminal - nohup npm start > /dev/null 2>&1 < /dev/null & + npm start %environment NODE_ENV=development @@ -351,7 +360,7 @@ The complete definition file will look like this: URL=localhost export NODE_ENV PORT ALLOW_HTTP URL -The container can be built like so: +We can now build the container image from the definition file: .. code:: @@ -360,18 +369,14 @@ The container can be built like so: Running the Service =================== -We can now start an instance and run the service: +We can now start an instance to run the service: .. code:: - $ sudo singularity instance start url-to-pdf.sif pdf - -.. note:: + $ singularity instance start url-to-pdf.sif pdf - If there occurs an error related to port connection being refused - while starting the instance or while using it later, you can try - specifying different port numbers in the ``%environment`` section of - the definition file above. +Because the service listens on port 9000, which is not a privileged port, we +don't need to run it with ``sudo`` this time. We can confirm it’s working by sending the server an http request using curl: @@ -390,189 +395,107 @@ You should see a PDF file being generated like the one shown below: .. image:: docpage.png :alt: Screenshot of the PDF generated! -If you shell into the instance, you can see the running processes: +If you shell into the instance, you can see the processes that are running, to +provide the service: .. code:: - $ sudo singularity shell instance://pdf - {Singularity}: Invoking an interactive shell within container... - - {Singularity} final.sif:/home/ysub> ps auxf - USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND - root 461 0.0 0.0 18204 3188 pts/1 S 17:58 0:00 /bin/bash --norc - root 468 0.0 0.0 36640 2880 pts/1 R+ 17:59 0:00 \_ ps auxf - root 1 0.0 0.1 565392 12144 ? Sl 15:10 0:00 sinit - root 16 0.0 0.4 1113904 39492 ? Sl 15:10 0:00 npm - root 26 0.0 0.0 4296 752 ? S 15:10 0:00 \_ sh -c nodemon --watch ./src -e js src/index.js - root 27 0.0 0.5 1179476 40312 ? Sl 15:10 0:00 \_ node /pdf_server/node_modules/.bin/nodemon --watch ./src -e js src/index.js - root 39 0.0 0.7 936444 61220 ? Sl 15:10 0:02 \_ /usr/local/bin/node src/index.js - - {Singularity} final.sif:/home/ysub> exit - -Making it Fancy -=============== - -Now that we have confirmation that the server is working, let’s make it -a little cleaner. It’s difficult to remember the exact ``curl`` command -and URL syntax each time you want to request a PDF, so let’s automate -it. Instead of creating completely separate containers for the server -and our streamlined client, it'd be nice to have them both available in -the same SIF file. To do that, we can use Scientific Filesystem (SCIF) -apps. - -.. note:: - - SCIF is a standard for encapsulating multiple apps into a container. - A container with SCIF apps has multiple entry points, and you can - choose which to run easily. Each entry point can carry out a - different task with it's own environment, metadata etc., without the - need for a collection of different containers. - - {Singularity} implements SCIF, and you can read more about how to use - it :ref:`apps `. - - SCIF is not specific to {Singularity}. You can learn more about it at - the project site: `_. - -First off, we’re going to move the installation of the url-to-pdf into -an app, so that there is a designated spot to place output files. To do -that, we want to add a section to our definition file to build the -server: - -.. code:: singularity - - %appinstall pdf_server - git clone https://github.com/alvarcarto/url-to-pdf-api.git pdf_server - cd pdf_server - npm install - chmod -R 0755 . - -And update our ``startscript`` to point to the app location: - -.. code:: singularity - - %startscript - cd /scif/apps/pdf_server/scif/pdf_server - # Use nohup and /dev/null to completely detach server process from terminal - nohup npm start > /dev/null 2>&1 < /dev/null & - -Now we want to define the pdf_client app, which we will run to send the -requests to the server: - -.. code:: singularity - - %apprun pdf_client - if [ -z "${1:-}" ]; then - echo "Usage: singularity run --app pdf [output file]" - exit 1 - fi - curl -o "${SINGULARITY_APPDATA}/output/${2:-output.pdf}" "${URL}:${PORT}/api/render?url=${1}" + $ singularity shell instance://pdf + Singularity> ps aux + USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND + dtrudg-+ 1 0.0 0.0 1178984 20700 ? Sl 11:40 0:00 sinit + dtrudg-+ 13 0.0 0.0 4284 696 ? S 11:40 0:00 /bin/sh /.singularity.d/startscript + dtrudg-+ 15 1.0 0.0 984908 41508 ? Sl 11:40 0:00 npm + dtrudg-+ 25 0.0 0.0 4292 716 ? S 11:40 0:00 sh -c env-cmd nodemon --watch ./src -e js src/index.js + dtrudg-+ 26 0.1 0.0 876908 31084 ? Sl 11:40 0:00 node /pdf_server/node_modules/.bin/env-cmd nodemon --watch ./src -e js src/index + dtrudg-+ 32 0.7 0.0 1113984 39976 ? Sl 11:40 0:00 node /pdf_server/node_modules/.bin/nodemon --watch ./src -e js src/index.js + dtrudg-+ 44 1.7 0.0 941556 53804 ? Sl 11:40 0:00 /usr/local/bin/node src/index.js + dtrudg-+ 124 0.0 0.0 18372 3592 pts/1 S 11:41 0:00 /bin/bash --norc + dtrudg-+ 130 0.0 0.0 36640 2836 pts/1 R+ 11:41 0:00 ps aux -As you can see, the ``pdf_client`` app checks to make sure that the user -provides at least one argument. -The full def file will look like this: +************* +Instance Logs +************* -.. code:: singularity - - Bootstrap: docker - From: node:8 - Includecmd: no - - %post - - apt-get update && apt-get install -yq gconf-service libasound2 \ - libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 \ - libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 \ - libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 \ - libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \ - libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 \ - libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates \ - fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils \ - wget curl && rm -r /var/lib/apt/lists/* - - %appinstall pdf_server - git clone https://github.com/alvarcarto/url-to-pdf-api.git pdf_server - cd pdf_server - npm install - chmod -R 0755 . +Generally, when running services using instances, we write the ``%startscript`` +so that the service will run in the foreground, and would write any log messages +to the terminal. When an instance container is started there is no terminal. +{Singularity} moves the container into the background, and collects output and +error messages into log files. - %startscript - cd /scif/apps/pdf_server/scif/pdf_server - # Use nohup and /dev/null to completely detach server process from terminal - nohup npm start > /dev/null 2>&1 < /dev/null & +You can view the location of log files for running instances using the ``--log`` +option of the ``instance list`` command: - %environment - NODE_ENV=development - PORT=9000 - ALLOW_HTTP=true - URL=localhost - export NODE_ENV PORT ALLOW_HTTP URL +.. code:: - %apprun pdf_client - if [ -z "${1:-}" ]; then - echo "Usage: singularity run --app pdf [output file]" - exit 1 - fi - curl -o "${SINGULARITY_APPDATA}/output/${2:-output.pdf}" "${URL}:${PORT}/api/render?url=${1}" + $ singularity instance list --logs + INSTANCE NAME PID LOGS + pdf 935864 /home/dtrudg-sylabs/.singularity/instances/logs/mini/dtrudg-sylabs/pdf.err + /home/dtrudg-sylabs/.singularity/instances/logs/mini/dtrudg-sylabs/pdf.out -Create the container as before. The ``--force`` option will overwrite -the old container: +Note that the log files are located under ``.singularity/instances`` in the +user's home directory, and are grouped by the hostname, and instance name. -.. code:: +The ``.out`` log collects standard output. The ``.err`` log collects standard +error. You can look at the content of the log files to check how your service is +running: - $ sudo singularity build --force url-to-pdf.sif url-to-pdf.def +.. code:: -Now that we have an output directory in the container, we need to expose -it to the host using a bind mount. Once we’ve rebuilt the container, -make a new directory called ``/tmp/out`` for the generated PDFs to go. + $ cat /home/dtrudg_sylabs/.singularity/instances/logs/mini/dtrudg-sylabs/pdf.out -.. code:: + > url-to-pdf-api@1.0.0 start /pdf_server + > env-cmd nodemon --watch ./src -e js src/index.js - $ mkdir /tmp/out + [nodemon] 1.19.0 + [nodemon] to restart at any time, enter `rs` + [nodemon] watching: /pdf_server/src/**/* + [nodemon] starting `node src/index.js` + 2023-02-01T11:14:58.185Z - info: [app.js] ALLOW_HTTP=true, unsafe requests are allowed. Don't use this in production. + 2023-02-01T11:14:58.187Z - info: [app.js] ALLOW_URLS set! Allowed urls patterns are: + 2023-02-01T11:14:58.187Z - info: [app.js] Using CORS options: origin=*, methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD, PATCH] + 2023-02-01T11:14:58.206Z - warn: [router.js] Warning: no authentication required to use the API + 2023-02-01T11:14:58.209Z - info: [index.js] Express server listening on http://localhost:9000/ in development mode + 2023-02-01T11:15:17.269Z - info: [render-core.js] Rendering with opts: { + ... -After building the image from the edited definition file we simply start -the instance: +*********************** +Resource Usage / Limits +*********************** -.. code:: +If you are running a container as the ``root`` user, or your system supports +cgroups v2, then all instances will be started inside a cgroup. A cgroup allows +the resources used by the instance to be monitored, and limited. - $ singularity instance start --bind /tmp/out/:/output url-to-pdf.sif pdf +To monitor the resource usage of an instance, use the ``instance stats`` command: -To request a pdf simply do: +.. code:: -.. code:: + $ singularity instance stats pdf + INSTANCE NAME CPU USAGE MEM USAGE / LIMIT MEM % BLOCK I/O PIDS + pdf 0.00% 479.8MiB / 62.2GiB 0.75% 470MiB / 131.6MiB 45 - $ singularity run --app pdf_client instance://pdf http://sylabs.io/docs sylabs.pdf +We can see that the instance is currently idle (0.00% CPU), and is using +479.8MiB of RAM. No limits have been applied, so the total RAM size of the +machine is shown. -To confirm that it worked: - -.. code:: - - $ ls /tmp/out/ - sylabs.pdf - -When you are finished, use the instance stop command to close all -running instances. - -.. code:: - - $ singularity instance stop --all - -.. note:: +By default, ``instance stats`` is interactive when run from a terminal, and will +update every second. To obtain point-in-time usage details use the +``--no-stream`` or ``--json`` options. - If the service you want to run in your instance requires a bind - mount, then you must pass the ``--bind`` option when calling - ``instance start``. For example, if you wish to capture the output of - the ``web`` container instance which is placed at ``/output/`` inside - the container you could do: +Where supported by the system's cgroups configuration, resource limits can be +applied to instances using the same :ref:`command line flags ` +that are available for interactive containers. E.g. to limit memory usage to +1GiB, we can use the ``--memory`` flag: - .. code:: +.. code:: - $ singularity instance start --bind output/dir/outside/:/output/ nginx.sif web + singularity instance start --memory 1G url-to-pdf.sif pdf -******************************** - System integration / PID files -******************************** +****************************** +System integration / PID files +****************************** If you are running services in containers you may want them to be started on boot, and shutdown gracefully automatically. This is usually @@ -580,7 +503,7 @@ performed by an init process, or another supervisor daemon installed on your host. Many init and supervisor daemons support managing processes via pid files. -You can specify a `--pid-file` option to `singularity instance start` to +You can specify a ``--pid-file`` option to ``singularity instance start`` to write the PID for an instance to the specified file, e.g. .. code::