Singularity: Dir in OverlayFS from Layer only 1MB

[ax3l]

Version of Singularity:

2.3.1-dist on Debian testing (buster/sid)

Expected behavior

Directories created from a layer (%post script or imported docker layer) should have a large OverlayFS when run.

Actual behavior

For directories created during the bootstrap of a singularity image, only 1MB OverlayFS will be available for modifications at runtime.

This is sub-ideal if I want to run JIT-ed projects or frameworks on which at runtime the users compile additional software (for which I want to assist them by already having example files and temporary build directories set up as a FS structure they know)

I also might just be using Singularity wrongly, in that case: is there a way to mount/bind an empty tmpfs from the host to the container path below when the container is started (run)?

Steps to reproduce behavior

Singularity file

Bootstrap: docker
From: ubuntu:16.04

%post
mkdir -p /home/something
chmod a+rwx  /home/something

# same behaviour with /opt/something or /code/something

%runscript
exec /bin/bash -c 'cd /home/something && /bin/bash -l'

Then execute:

singularity create ubuntu.img
sudo singularity bootstrap ubuntu.img Singularity
singularity run ubuntu.img

And inside the container:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
OverlayFS       1.0M     0  1.0M   0% /
[...]

$ dd if=/dev/zero bs=1M count=1024 of=aa 
dd: error writing 'aa': No space left on device
2+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00141735 s, 740 MB/s

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
OverlayFS       1.0M  1.0M     0 100% /
/dev/sda3       854G   34G  812G   4% /tmp
udev            3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs            16M  8.0K   16M   1% /etc/group
tmpfs           799M  9.4M  790M   2% /etc/resolv.conf

[neiltwist]

I've had the same issue with this version (2.3.1-dist). I don't want to copy to the users home directory, but I do need to copy and modify a file, before running OpenFOAM within the container, which writes to the directory it's running in.

[gmkurtzer]

Is this data supposed to be persistent? If so, you should be mounting the container as --writable. If not, then why not just use /tmp, or bind mount a directory from the host into /tmp?

[ax3l]

Doesn't mount require root?

Adding it to the cmd line would work but blows up the interface my users see to a longer magic string.

Just linking or mktmp -d does not cut it either if one expects certain links with relative paths within those.

[neiltwist]

Doesn't tmp get mounted to the host system? I don't want the data to be persistent, and in fact I actively would prefer to have the data not accessible outside the container. Once the intermediate results are processed, I then copy the final results to a mounted directory for persistence. I think that I could achieve what I want in a different way, but it wasn't really well documented that I couldn't write inside the container while running it. It was clear that the changes would not be persisted, but not that I couldn't write at all.

I have a workaround by binding a directory inside the container, but with Docker, I could do some work inside the container before copying to the outside (not that Singularity should follow Docker necessarily).

[gmkurtzer]

@ax3l, mount does not require root, but in this case I was referring to the mounting of the container image at runtime as writable. I can't speak directly to your need regarding the necessity to relative paths or use of mktemp as that is highly dependent of the use case.

@neiltwist, I think your workaround is the best practice, but you could also use /dev/shm, if you needed something fast. We could easily change the overlay file system size or make it configurable if that would help.

Lastly, Docker does allow this sorta thing, but that is because it doesn't use image files (that are mounted as read-only by default) and processes inside the container run as root. It makes it very easy to rampage inside a container. Lol

[ax3l]

@gmkurtzer maybe I just did not get the workflow yet how to do this properly in Singularity.

as @neiltwist 's use case, I want to generate temporary data that can be purged when the container is shut down. Ideally, I would like to express: "mount me an empty tmpfs at mountpoint XY" when the container is started.

Can I somehow express the equivalent of the following line in an /etc/fstab

tmpfs   /home/something/tmp         tmpfs   defaults,size=1G          0  0

in the Singularity file?

[sneumann]

Any solution here ? I need a temporary, user-owned WINEPREFIX inside the container.
UPDATE: seems I can use -S /mywineprefix/ for that. Yours, Steffen

[ajyounge]

Forgive me for digging up an old thread here, but I actually stumbled upon a use case which has the same issue and perhaps its good to share. Granted, it's a bit of a corner case, but may also be relevant for others.

My developer is building an HPC application, lets call it Blah, originally in Docker, that includes a "make test" via cmake which runs a series of small serial and MPI tests to validate the app is working as expected. The actual apps and libraries are installed in the container as /opt, the app is built in /build/, with open permissions 777 for non-root to leverage. Our developer wants to run "make test" interactively after the image is converted into singularity "singularity build blah.simg docker://myprivaterepo.co/ajy/blah:latest" to ensure everything works in the singularity container on the HPC resources, before actually running real apps.

The issue is "make test" creates several log files of the various tests, which quickly fill up the 1M overlay and all subsequent tests fail. Developer is not happy haha. Now, my quick solution is to just copy things to /tmp (mount tmp = yes in singularity.conf) and run there, eg: "cp /build/blah /tmp/ && cd /tmp/blah/build && make test". However, this seems really hacky. Furthermore, we don't really care about the results or logs past container exit, other than the developer was able to see his tests pass/fail and, I guess quickly inspect those logs while still inside the container if they did indeed fail for some reason.

So, what's the right process and model here when I actually want my container to stay ephemeral but still usable beyond 1M? Obviously, we'll have to be mindful of limits to where the overlay upperdir is coming from (tmpfs/ramfs?). Is there a setting to increase the overlay size, of course noting the discarding my data at container exit? Should I be telling the developers to come up with a better cmake test version that doesn't depend on particular directory structure? Or should I just get used to manually copying into /tmp for things truly ephemeral? I'm running 2.6 and perhaps all this has changed with 3.X and I need to upgrade? :P

Does anybody have thoughts or more experience than I do on the best solution and/or practice here?

[jmstover]

Hi @ajyounge ,

In 2.x this is a hard coded valued at 1MB. The plan is for 3.1.0 to implement a tmpfs that will allow a user to customize the size.

Does the make test make use of the TMPDIR value? Meaning, can you do:

  mkdir /tmp/blah
  TMPDIR=/tmp/blah make test

And have it write out to /tmp/blah instead of the working directory?

[dtrudg]

Closing this as I do not believe it to be an issue on 3.x with the --writable-tmpfs approach now in place. Please feel free to re-open or create a new issue if you have a problem on the latest version of Singularity. Thanks!

[danielballan]

The plan is for 3.1.0 to implement a tmpfs that will allow a user to customize the size.

Can anyone offer some guidance or documentation on how to customize the size of --writable-tmpfs?

[danielballan]

To answer my own question, in case someone else ends up here with the same question:

Set sessiondir max size in the configuration file (singularity.conf).

[tashrifbillah]

To make Dan's answer more specific, locate the /etc/singularity/singularity.conf file and adjust the following number:

# SESSIONDIR MAXSIZE: [STRING]
# DEFAULT: 16
# This specifies how large the default sessiondir should be (in MB) and it will
# only affect users who use the "--contain" options and don't also specify a
# location to do default read/writes to (e.g. "--workdir" or "--home").
sessiondir max size = 16

I am on a CentOS 7 machine and using singularity version 3.7.0-1.el7.

[tashrifbillah]

However, this comment does not appear to hold true:

it will only affect users who use the "--contain" options and don't also specify a location to do default read/writes

I have tried --contain --workdir /my/dir, yet sessiondir max size remained limited to 16 M.