Singularity: Singularity TMPDIR and CACHEDIR specification

[olifre]

The documentation is missing information on how to specify the cache directory (where e.g. tarballs from docker layers are stored) and temporary directories (where, starting from 2.4 with SquashFS, enough space for the full container is required).
To setup an automated container build host, such information is crucial: The defaults should be known / documented, and the way to configure that needs to be documented as well, so the user can choose the appropriate filesystem / device to hold that information.
Best would be to document all user-exposed environment variables Singularity may use.

Concerning TMPDIR and CACHEDIR, here is what I learned from some basic testing.

Running:

SINGULARITY_TMPDIR=/scratch/tmp SINGULARITY_CACHEDIR=/scratch/cache singularity build  fos.img minimal.def

(with "minimal.def" boostrapping from Docker).

I observe the following:
/scratch/tmp/.singularity-build.NbsdAx
is created during build, holding the container root, as expected. This is temporary and removed afterwards, all fine :-).

Additionally:
/scratch/cache/docker
contains the tarballs from the Docker layers, and is kept persistent. Also perfectly fine :-).

However, the communication file:
/tmp/.singularity-layers.GMwGRqyD
is shortly present in /tmp. This location seems to be hardcoded (see https://github.com/singularityware/singularity/blob/2.4/libexec/handlers/image-docker.sh#L27 ),
but since that just seems to contain the list of layers, that's not a huge issue (would be nice though if SINGULARITY_TMPDIR would be honoured).

So all in all, it seems to really work as one might expect: SINGULARITY_CACHEDIR for persistent things, and SINGULARITY_TMPDIR for temporary things during build.

These environment variables (and maybe more) should be documented in the frame of "singularity build". One example as on why users would like to know that:
We will now put an extra partition with several GB in our build-VM to contain SINGULARITY_CACHEDIR to reduce the load on DockerHub.
If we would use the SquashFS-images, we would also set SINGULARITY_TMPDIR to some ramdisk location (e.g. tmpfs) to speed up the build tremendously (now we do that with --sandbox and some manual scripting in any case, since we build tarbarls instead of SquashFS-images for easier CVMFS deployment of the flat directory trees).

[soichih]

I can't get SINGULARITY_TMPDIR to work with currently released version of 2.4.1-dist. Is this very new and only available on the later build? Or maybe the other way around; deprecated?

[vsoch]

Could you give a toy example to replicate the issue? I can try to help.

[soichih]

@vsoch Following step should reprecate the problem

1) Install singularity on Ubuntu 16 (via neurodebian) - it should come with 2.4.1-dist
2) Set SINGULARITY_TMPDIR to some non default location.
export SINGULARITY_TMPDIR=/somewhere
3) Run singularity exec docker://busybox bash

After 3, you should see /tmp/.singularity used to create squashed image directory - instead of /somewhere.

[vsoch]

okay here it is testing with 2.4.2 master (from neurodebian)

First with nothing special, goes to default

singularity --debug pull docker://ubuntu
Enabling debugging
Ending argument loop
Singularity version: 2.4.2-master.g91881f7
Exec'ing: /usr/local/libexec/singularity/cli/pull.exec
Singularity Image: ./ubuntu.simg
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /home/vanessa/.singularity/docker
[5/5] |===================================| 100.0% 
Importing: base Singularity environment
Importing: /home/vanessa/.singularity/docker/sha256:1be7f2b886e89a58e59c4e685fcc5905a26ddef3201f290b96f1eff7d778e122.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:6fbc4a21b806838b63b774b338c6ad19d696a9e655f50b4e358cc4006c3baa79.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:c71a6f8e13782fed125f2247931c3eb20cc0e6428a5d79edb546f1f1405f0e49.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:4be3072e5a37392e32f632bb234c0b461ff5675ab7e362afad6359fbd36884af.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:06c6d2f5970057aef3aef6da60f0fde280db1c077f0cd88ca33ec1a70a9c7b58.tar.gz
Importing: /home/vanessa/.singularity/metadata/sha256:c6a9ef4b9995d615851d7786fbc2fe72f72321bee1a87d66919b881a0336525a.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ./ubuntu.simg
Cleaning up...
Done. Container is at: ./ubuntu.simg

and now with setting the SINGULARITY_CACHEDIR

vanessa@vanessa-ThinkPad-T450s:~$ SINGULARITY_CACHEDIR=/tmp singularity --debug pull --name ubuntu-tmpdir.simg docker://ubuntu
Enabling debugging
Ending argument loop
Singularity version: 2.4.2-master.g91881f7
Exec'ing: /usr/local/libexec/singularity/cli/pull.exec
Singularity Image: /tmp/ubuntu-tmpdir.simg
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /tmp/docker
[5/5] |===================================| 100.0% 
Importing: base Singularity environment
Importing: /tmp/docker/sha256:1be7f2b886e89a58e59c4e685fcc5905a26ddef3201f290b96f1eff7d778e122.tar.gz
Importing: /tmp/docker/sha256:6fbc4a21b806838b63b774b338c6ad19d696a9e655f50b4e358cc4006c3baa79.tar.gz
Importing: /tmp/docker/sha256:c71a6f8e13782fed125f2247931c3eb20cc0e6428a5d79edb546f1f1405f0e49.tar.gz
Importing: /tmp/docker/sha256:4be3072e5a37392e32f632bb234c0b461ff5675ab7e362afad6359fbd36884af.tar.gz
Importing: /tmp/docker/sha256:06c6d2f5970057aef3aef6da60f0fde280db1c077f0cd88ca33ec1a70a9c7b58.tar.gz
Importing: /tmp/metadata/sha256:c6a9ef4b9995d615851d7786fbc2fe72f72321bee1a87d66919b881a0336525a.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: /tmp/ubuntu-tmpdir.simg
Cleaning up...
Done. Container is at: /tmp/ubuntu-tmpdir.simg

SINGULARITY_TMPDIR also uses the default cache, because (I think) it's relevant for "on the fly" images like with shell / run of a docker uri, let's try that.

Okay, when I run with exec (and note that busybox doesn't have bash so I had to use sh) I see the temporary directory is created at Cleaning directory: /tmp/.singularity-runtime.1Bwh3aFI is that the one you are referring to? If you want to change this one, try SINGULARITY_LOCALCADHEDIR

 SINGULARITY_LOCALCACHEDIR=/home/vanessa/Desktop/testy singularity -v exec docker://busybox sh
Increasing verbosity level (2)
Singularity version: 2.4.2-master.g91881f7
Exec'ing: /usr/local/libexec/singularity/cli/exec.exec
Evaluating args: 'docker://busybox sh'
VERBOSE Docker image: busybox
VERBOSE Registry: index.docker.io
VERBOSE Namespace: library
VERBOSE Repo Name: busybox
VERBOSE Repo Tag: latest
VERBOSE Version: None
VERBOSE Obtaining tags: https://index.docker.io/v2/library/busybox/tags/list
Docker image path: index.docker.io/library/busybox:latest
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/sha256:4cee1979ba0bf7db9fc5d28fb7b798ca69ae95a47c5fecf46327720df4ff352d
Cache folder set to /home/vanessa/.singularity/docker
Creating container runtime...
Importing: base Singularity environment
Exploding layer: sha256:57310166fe88e0dc63a80ca5c219283a932db0f3969712e2f8a86ada143bf566.tar.gz
Exploding layer: sha256:29ade4f96a9909940ce0251b8182993c53f22a7e006a8c7fe868f1e0032d52b5.tar.gz
VERBOSE: Set messagelevel to: 2
VERBOSE: Initialize configuration file: /usr/local/etc/singularity/singularity.conf
VERBOSE: Initializing Singularity Registry
VERBOSE: Invoking the user namespace
VERBOSE: Not virtualizing USER namespace: running as SUID
VERBOSE: No autofs bug path in configuration, skipping
VERBOSE: Exec'ing cleanupd thread: /usr/local/libexec/singularity/bin/cleanupd
VERBOSE: Set messagelevel to: 2
VERBOSE: Daemonizing cleandir cleanup process
VERBOSE: Using session directory: /usr/local/var/singularity/mnt/session
VERBOSE: Mounting overlay with options: lowerdir=/usr/local/var/singularity/mnt/container,upperdir=/usr/local/var/singularity/mnt/overlay/upper,workdir=/usr/local/var/singularity/mnt/overlay/work
VERBOSE: Running all mount components
VERBOSE: Binding '/etc/localtime' to '/usr/local/var/singularity/mnt/final//etc/localtime'
VERBOSE: Binding '/etc/hosts' to '/usr/local/var/singularity/mnt/final//etc/hosts'
VERBOSE: Bind-mounting host /proc
VERBOSE: Mounting /sys
VERBOSE: Bind mounting /dev
VERBOSE: Mounting home directory source into session directory: /home/vanessa -> /usr/local/var/singularity/mnt/session/home/vanessa
VERBOSE: Mounting staged home directory to container: /usr/local/var/singularity/mnt/session/home/vanessa -> /usr/local/var/singularity/mnt/final/home/vanessa
VERBOSE: Mounting directory: /tmp
VERBOSE: Mounting directory: /var/tmp
VERBOSE: Not mounting current directory: location already available within container
VERBOSE: Running file components
VERBOSE: Creating template passwd file and appending user data: /usr/local/var/singularity/mnt/session/passwd
VERBOSE: Binding file '/usr/local/var/singularity/mnt/session/passwd' to '/usr/local/var/singularity/mnt/final/etc/passwd'
VERBOSE: Updating group file with user info
VERBOSE: Binding file '/usr/local/var/singularity/mnt/session/group' to '/usr/local/var/singularity/mnt/final/etc/group'
VERBOSE: Binding file '/etc/resolv.conf' to '/usr/local/var/singularity/mnt/final/etc/resolv.conf'
VERBOSE: Containing all rootfs components
VERBOSE: Entering container file system root: /usr/local/var/singularity/mnt/final
LOG    : USER=vanessa, IMAGE='busybox', COMMAND='exec'
VERBOSE: Exec'ing /.singularity.d/actions/exec
Singularity> exit
VERBOSE: Cleaning directory: /home/vanessa/Desktop/testy/.singularity-runtime.ot6ieZCP

[soichih]

I see the temporary directory is created at Cleaning directory: /tmp/.singularity-runtime.1Bwh3aFI is that the one you are referring to? If you want to change this one, try SINGULARITY_LOCALCADHEDIR

Yes! OK.. I just tried setting SINGULARITY_LOCALCADHEDIR to my alt. tmp directory, and it worked! Is this documented anywhere?

By the way, for some reason, singularity that comes with neurodebian repo on Ubuntu 16 is still 2.4.1

hayashis@dev1:~ $ sudo apt upgrade singularity-container
Reading package lists... Done
Building dependency tree       
Reading state information... Done
singularity-container is already the newest version (2.4.1-1~nd16.04+1).

I am curious how you are getting 2.4.2 from neurodebian.

Following page also shows that it should come with 2.4.1

http://neuro.debian.net/pkgs/singularity-container.html?highlight=singularity

[vsoch]

I was on an older Ubuntu, trusty, hehe :)

I can add this to be better found in the docs. Could you give me one or two sentences for what you were trying to do (e.g. I wanted to create a temporary image with shell and change the default?) and I'll give it a go!

[soichih]

How about something like..

/tmp directories are typically small on most VM hosted at Jetstream/OpenStack, Azure, and probably EC2. Our VMs were constantly running out of /tmp space. The issue was exacerbated by singularity crashing at startup before it was able to clean up /tmp/.singularity* (due to .whiteout file handling issue.) I needed a way to instruct singularity to not use /tmp, so that I will have time to fix the issue before the entire VM becomes instable

By the way, is there way to instruct singularity to re-use the same container image created in SINGULARITY_LOCALCADHEDIR? I believe it doesn't reuse because the content of the container image could change (and singularity can't use overlay / aufs type file system..) but if singularity can make the image read-only, then technically it should be able to just reuse already flattened container image (and save 30-60 seconds that it takes to generate the image everytime I run singularity?)

[vsoch]

okay, check out here! http://singularity.lbl.gov/build-environment#temporary-folders

I also added some additional links to get there, period. It was only linked from the BOTTOM of build and impossible to find.

[soichih]

Thanks.

By the way, I think SINGULARITY_LOCALCADHEDIR should be SINGULARITY_LOCALCACHEDIR instead (I think I misspelled it originally)

[vsoch]

Good catch! Should be fixed now...

[soichih]

I just found out that, I can't set SINGULARITY_LOCALCACHEDIR to lustre mounted volume..

$ export SINGULARITY_LOCALCACHEDIR=/N/dc2/scratch/brlife/singularity-tmp-carb
$ singularity exec -e docker://brainlife/mcr:neurodebian1604-r2017a ./compiled/dtiinit
Docker image path: index.docker.io/brainlife/mcr:neurodebian1604-r2017a
Cache folder set to /N/dc2/scratch/brlife/singularity-cache-carb/docker
Creating container runtime...
ERROR  : Could not mount Singularity overlay: Invalid argument
ABORT  : Retval = 255

Leaving it default (/tmp) doesn't work either.. because it would run of our /tmp on our cluster...

Why does singularity leave those /tmp/.singularity-runtime... directories? The only solution that I can think of it is to write a cron job which will find unused /tmp/.singularity-runtime.. directories and remove them, but I am not sure how to check which one is used and which ones are left behind.. Is there a singularity equivalent of "docker system prune"?

[jmstover]

@soichih

Why does singularity leave those /tmp/.singularity-runtime... directories?

It shouldn't. There is a race condition bug where it could try deleting the directories before everything is fully seen as unmounted. See PR #1265 and issue #1255

[soichih]

@jmstover Thanks. I've subscribed to those issues.

[dtrudg]

We now list env vars in docs at: https://sylabs.io/guides/3.5/user-guide/appendix.html

We also have: https://sylabs.io/guides/3.5/user-guide/build_env.html#cache-folders

and: https://sylabs.io/guides/3.5/user-guide/build_env.html#temporary-folders