Skip to main content Skip to navigation
Kamiak Programmer's Guide

Singularity

Overview

Containers allow users to bundle their application in a lightweight package along with everything required to run it, including code, software dependencies, system libraries, and configuration files. Docker has long been the most popular container framework, but it is not suitable for HPC platforms, primarily due to security concerns arising from giving users root privileges inside the container. Singularity is a container framework specifically designed for HPC platforms. Singularity runs the container in user space, without any possibility of privilege escalation inside the container. Singularity also provides us the ability to convert existing Docker containers to Singularity image format, or to run containers directly from public repositories such as Docker Hub.

When running a compute job that does not use containers, you are running in what we’ll refer to as the host environment. This includes the operating system, libraries, devices, and network for the compute node. When running a compute job within a container, you are instead running in the container’s environment. However, some aspects of the host environment are reflected into every container. This includes:

  • All mount points to HPC storage (/home, /data, /opt/apps, /scratch)
  • /dev (including Infiniband and GPU devices)

Quick Start

Request a compute node using idev. singularity pull command can be used to download pre-built images from resources like the Docker Hub or Singularity Hub.
Set umask u=rwx,g=rx,o=rx before pulling an image to allow “other” read access.

$ idev -N 1 --ntasks=1 -t 90
$ module load singularity/3.0.0 
$ umask u=rwx,g=rx,o=rx
$ singularity pull shub://vsoch/hello-world
$ singularity pull docker://ubuntu:latest 

Run the container’s runscript using run command or directly as an executable file.

$ singularity run hello-world_latest.sif
$ ./hello-world_latest.sif

We can also execute a specific command within a container using exec command.

$ singularity exec hello_world_latest.sif cat /etc/os-release

To run an interactive shell in a container, use the shell command.

$ singularity shell hello_world_latest.sif

To run GPU-enabled containers, add --nv option to exec or run commands.

$ idev -N 1 --ntasks=1 --gres=gpu:1 -t 90
$ git clone https://github.com/tensorflow/models.git
$ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu python ./models/tutorials/image/mnist/convolutional.py

Running containers with MPI support using Singularity is more involved. See MPI support section below for more discussion.
Singularity commands can also be run using a batch script. See batch mode section below for a sample job script.

Singularity Basics

This section contains all the necessary information in order to get started with Singularity. User input is shown in bold.

Commonly used commands

Below are the most commonly used Singularity commands. See Singularity documentation for all available commands.
help Help about any command
build Build a new Singularity container
exec Execute a command within container
inspect Display metadata for container if available
pull Pull a container from a URI
run Launch a runscript within container
run-help Display help for container if available
shell Run a Bourne shell within container
test Run defined tests for this particular container

Examples:

$ singularity help
$ singularity help <command>
$ singularity shell docker://ubuntu:latest
$ singularity exec docker://ubuntu:latest cat /etc/os-release

Obtaining container images

pull and build commands can be used to download pre-built images from resources like the Docker Hub or Singularity Container Library. For Singularity v3.0 the default image format is Singularity Image File (SIF). build requires specifying a name for the container, whereas, it is optional while downloading a container using pull. Note that all the examples in this guide are run on a compute node within an interactive job.
To start with, let’s look around the host environment for a moment. We can see compute nodes (as of this writing) run CentOS 7 on a 3.10.0-693 kernel.

$ uname -r
3.10.0-693.11.6.el7.x86_64

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

We’ll also create a scratch workspace to work within, but you can use any storage you have access to.

$ mkworkspace 
Successfully created workspace.  Details:
    Workspace: /path/to/workspace
    User: user.name
    Group: its_p_sys_ur_kam-mygroup
    Expiration: 2018-12-28 14:55:28.38494

$ cd /path/to/workspace

Now we can load the singularity module and download pre-built images using either pull or build.
NOTE: Set umask u=rwx,g=rx,o=rx before pulling or building an image to allow “other” read access, otherwise you will get an error No valid /bin/sh in container

$ module load singularity/3.0.0
$ umask u=rwx,g=rx,o=rx
$ singularity pull docker://ubuntu:latest # with default ubuntu_latest.sif name
$ singularity pull ubuntu.sif docker://ubuntu:latest # with custom ubuntu.sif name
$ singularity build ubuntu.sif library://ubuntu

Running containers

There are two ways in which we can run Singularity containers. First, interactive mode, where we run a container interactively on a compute node; and second, batch mode, where we run a container as a part of a batch script.

Interactive mode
$ module load singularity/3.0.0
$ singularity pull docker://ubuntu:latest
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f
 30.62 MiB / 30.62 MiB [====================================================] 0s
...
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: ubuntu_latest.sif 

$ singularity shell ubuntu_latest.sif 
Singularity ubuntu_latest.sif:/path/to/workspace>

Notice the change in the command prompt, indicating we are now running within the container. Let’s compare it to the host environment and verify we’re really in an Ubuntu container.

Singularity ubuntu-latest.sif:/path/to/workspace> uname -r
3.10.0-693.11.6.el7.x86_64

Singularity ubuntu-latest.sif:/path/to/workspace> cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Exit the shell with exit to return to the host environment. Notice that while in the container we appear to be running Ubuntu – because we are – but we are still running on the host’s kernel since a container is not a virtual machine.

You can also directly interact with image URIs. This creates an ephemeral container that disappears when the shell is exited.

$ singularity shell docker://ubuntu:latest
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f
...
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: /home/user.name/.singularity/cache/oci-tmp/6d0e0c26489e33f5a6f0020edface2727db9489744ecc9b4f50c7fa671f23c49/ubuntu_latest.sif
INFO:    Image cached as SIF at /home/user.name/.singularity/cache/oci-tmp/6d0e0c26489e33f5a6f0020edface2727db9489744ecc9b4f50c7fa671f23c49/ubuntu_latest.sif

Singularity ubuntu_latest.sif:/path/to/workspace> cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Singularity ubuntu_latest.sif:/path/to/workspace> exit
exit
$

We can also execute a specific command within a container using exec command:

$ singularity exec ubuntu_latest.sif cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Batch mode

Let’s see how we can use the downloaded Ubuntu image in a compute job. Create a file with the following submission script and submit it with sbatch.

#!/bin/bash
#SBATCH -n 1        # Number of cores
#SBATCH -t 0-00:05  # Runtime in D-HH:MM
#SBATCH --job-name=singularity

echo "Starting singularity on host $HOSTNAME"

image=/path/to/workspace/ubuntu_latest.sif

module load singularity/3.0.0

singularity exec $image ls $HOME

echo "Completed singularity on host $HOSTNAME"

The job’s output file should contain a listing of the contents of your home directory.

Binding Directories

Singularity allows you to access the directories on the host system from within the container using bind mounts. On the HPC cluster, several directories such as /home, /scratch, /data, /local, /scidas, /tmp are included inside each container by default. You can also bind other directories into your container using the --bind/ -B command-line option or $SINGULARITY_BINDPATH environment variable. Mount options may be specified as ro (read-only) or rw (read/write, which is the default).

The following example shows how to bind the /opt on the host to /opt in the container in read-only mode and /data on the host to /mnt in the container using --bind command-line option:

$ singularity shell --bind /opt:/opt:ro,/data:/mnt container.sif

or using the environment variable:

$ export SINGULARITY_BINDPATH="/opt:/opt:ro,/data:/mnt"
$ singularity shell container.sif

Containers with MPI support

Singularity was developed to run containers on HPC platforms and supports different MPI implementations (such as Intel MPI, MVAPICH2, OpenMPI). It has been designed with out of the box compatibility with Open MPI (v2.1.x), but is known to work well with other MPI implementations as well. There are few points, however, that should be kept in mind before running a container with MPI:

  • MPI implementations on host and container necessarily need not to be the same, but should be ABI (or binary) compatible. Also, the MPI version inside the container should be the same or newer than the host.
  • Bind mount InfiniBand libraries into the container.

The example below shows how to obtain a Docker image and run the container using MPI by binding the host MPI and InfiniBand libraries.

$ module load singularity/3.0.0
$ singularity pull docker://underworldcode/underworld2:dev
$ export SINGULARITYENV_LD_LIBRARY_PATH=/opt/apps/mpich/gcc/6.1.0/3.2/lib:$LD_LIBRARY_PATH
$ export SINGULARITY_BINDPATH="/opt/apps/mpich/gcc/6.1.0/3.2:/opt/apps/mpich/gcc/6.1.0/3.2:ro,/opt/mellanox:/opt/mellanox:ro,/opt/ibutils:/opt/ibutils:ro"
$ srun --mpi=pmi2 -N 2 -n 4 singularity exec underworld2_dev.sif python path/to/script

Containers with GPU support

Singularity allows you to run GPU-enabled container by simply adding --nv option to exec or run commands. The following example shows how to interactively run a GPU-enabled container on the HPC cluster.

$ idev -N 1 --ntasks=1 --gres=gpu:1 -t 90
...
salloc: Granted job allocation 5911427
Allocated nodes: sn12
[user.name@sn12 /path/to/workspace]$ module load singularity/3.0.0 
[user.name@sn12 /path/to/workspace]$ git clone https://github.com/tensorflow/models.git
Cloning into 'models'...
remote: Enumerating objects: 95, done.
remote: Counting objects: 100% (95/95), done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 23695 (delta 52), reused 55 (delta 28), pack-reused 23600
Receiving objects: 100% (23695/23695), 563.07 MiB | 43.51 MiB/s, done.
Resolving deltas: 100% (13941/13941), done.
Checking out files: 100% (2798/2798), done.

$ singularity exec --nv docker://tensorflow/tensorflow:latest-gpu python ./models/tutorials/image/mnist/convolutional.py
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:18d680d616571900d78ee1c8fff0310f2a2afe39c6ed0ba2651ff667af406c3e
 41.34 MiB / 41.34 MiB [====================================================] 0s
...
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: /home/user.name/.singularity/cache/oci-tmp/847690afb29977920dbdbcf64a8669a2aaa0a202844fe80ea5cb524ede9f0a0b/tensorflow_latest-gpu.sif
INFO:    Image cached as SIF at /home/user.name/.singularity/cache/oci-tmp/847690afb29977920dbdbcf64a8669a2aaa0a202844fe80ea5cb524ede9f0a0b/tensorflow_latest-gpu.sif
Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting data/train-images-idx3-ubyte.gz
Extracting data/train-labels-idx1-ubyte.gz
Extracting data/t10k-images-idx3-ubyte.gz
Extracting data/t10k-labels-idx1-ubyte.gz
2018-12-20 17:09:04.128519: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-12-20 17:09:04.540442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:04:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-12-20 17:09:04.790309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:05:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-12-20 17:09:05.059855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 2 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:84:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-12-20 17:09:05.320347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 3 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:85:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2018-12-20 17:09:05.321053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1, 2, 3
2018-12-20 17:09:13.020318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-12-20 17:09:13.020372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1 2 3 
2018-12-20 17:09:13.020382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y N N 
2018-12-20 17:09:13.020390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N N N 
2018-12-20 17:09:13.020396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2:   N N N Y 
2018-12-20 17:09:13.020403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3:   N N Y N 
2018-12-20 17:09:13.028882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10756 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:04:00.0, compute capability: 3.7)
2018-12-20 17:09:13.029360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10756 MB memory) -> physical GPU (device: 1, name: Tesla K80, pci bus id: 0000:05:00.0, compute capability: 3.7)
2018-12-20 17:09:13.029730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10756 MB memory) -> physical GPU (device: 2, name: Tesla K80, pci bus id: 0000:84:00.0, compute capability: 3.7)
2018-12-20 17:09:13.030092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10756 MB memory) -> physical GPU (device: 3, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7)
Initialized!
Step 0 (epoch 0.00), 162.1 ms
Minibatch loss: 8.334, learning rate: 0.010000
Minibatch error: 85.9%
Validation error: 84.6%
Step 100 (epoch 0.12), 9.6 ms
Minibatch loss: 3.243, learning rate: 0.010000
Minibatch error: 6.2%
Validation error: 7.5%
...
Step 8500 (epoch 9.89), 8.1 ms
Minibatch loss: 1.599, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.9%
Test error: 0.8%

Building Singularity images

At this point you will likely want to know how to install your application(s) into a container or otherwise change it to suit your needs. For that you need to read Singularity’s documentation, linked below. That will provide information on how to create and use your own container images or modify existing ones. Note that we do not support the use of sudo with Singularity. If you need sudo rights you can create a container that suits your needs using your own workstation or in a virtual machine running Linux where you have sudo/root access. After you create the container or Singularity image you can copy it to the HPC cluster and run it.

Further Reading

Singularity documentation