Singularity: Could not mount Singularity overlay: No such device

[yarikoptic]

Version of Singularity:

NeuroDebian backport build of 2.3 for jessie

One of the students reported that image fails to run using my (Neuro)Debian build but works using "upstream" build (installed under /usr/local)... local installation is N/A ATM so can't verify but here is the trail of output from singularity. I wonder if any clue what could lead to such failure?

mvdoc@euclid ~/singularity/hauntedhouse $ singularity -d run hauntedhouse.img
...
DEBUG   [U=1000,P=34675]   singularity_registry_get()                Returning NULL on 'DISABLE_OVERLAYFS'
DEBUG   [U=1000,P=34675]   singularity_registry_get()                Returning NULL on 'WRITABLE'
DEBUG   [U=1000,P=34675]   singularity_runtime_overlayfs()           OverlayFS enabled by host build
DEBUG   [U=1000,P=34675]   singularity_runtime_overlayfs()           Setting up overlay mount options
DEBUG   [U=1000,P=34675]   singularity_runtime_overlayfs()           Checking for existance of overlay directory: /var/lib/singularity/mnt/overlay
DEBUG   [U=1000,P=34675]   singularity_priv_escalate()               Temporarily escalating privileges (U=1000)
DEBUG   [U=0,P=34675]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=34675]      singularity_runtime_overlayfs()           Mounting overlay tmpfs: /var/lib/singularity/mnt/overlay
DEBUG   [U=0,P=34675]      singularity_runtime_overlayfs()           Creating upper overlay directory: /var/lib/singularity/mnt/overlay/upper
DEBUG   [U=0,P=34675]      s_mkpath()                                Creating directory: /var/lib/singularity/mnt/overlay/upper
DEBUG   [U=0,P=34675]      singularity_runtime_overlayfs()           Creating overlay work directory: /var/lib/singularity/mnt/overlay/work
DEBUG   [U=0,P=34675]      s_mkpath()                                Creating directory: /var/lib/singularity/mnt/overlay/work
DEBUG   [U=0,P=34675]      singularity_runtime_overlayfs()           Creating overlay_final directory: /var/lib/singularity/mnt/overlay/final
DEBUG   [U=0,P=34675]      s_mkpath()                                Creating directory: /var/lib/singularity/mnt/overlay/final
VERBOSE [U=0,P=34675]      singularity_runtime_overlayfs()           Mounting overlay with options: lowerdir=/var/lib/singularity/mnt/container,upperdir=/var/lib/singularity/mnt/overlay/upper,workdir=/var/lib/singularity/mnt/overlay/work
ERROR   [U=0,P=34675]      singularity_runtime_overlayfs()           Could not mount Singularity overlay: No such device
ABORT   [U=0,P=34675]      singularity_runtime_overlayfs()           Retval = 255

overlay is enabled:
` mvdoc@euclid ~/singularity/hauntedhouse $ grep overlay /etc/singularity/singularity.conf enable overlay = yes

[vsoch]

is there any chance of old libraries still hanging around?

[mvdoc]

I'm the student, hello! :-)
so there were some old libraries hanging around. I removed them, and performed a clean install from neurodebian package. Then I re-created the image, just to be sure, but it still gives the same error. If I build from source it works fine.

debian:

mvdoc@euclid ~/singularity/hauntedhouse $ /usr/bin/singularity exec hauntedhouse2.img echo "Hello world\!"
ERROR  : Could not mount Singularity overlay: No such device
ABORT  : Retval = 255

Compiled:

mvdoc@euclid ~/singularity/hauntedhouse $ /usr/local/bin/singularity exec hauntedhouse2.img echo "Hello world\!"
Hello world!

[yarikoptic]

grr.. the beast works ok on my box with debian version of the package:

smaug:~/repronim/singularity
$> /usr/bin/singularity exec hauntedhouse.img echo "Hello world\!" 
Hello world!
1 11554.....................................:Sun 04 Jun 2017 08:35:40 PM EDT:.
smaug:~/repronim/singularity
$> singularity --version                                             
2.3-dist

[yarikoptic]

ok -- at least the mystery of diverging behavior is now clear:

mvdoc@euclid ~/singularity/hauntedhouse $ grep 'OverlayFS.*host.*build' debian-d.log compiled-d.log 
debian-d.log:DEBUG   [U=1000,P=71887]   singularity_runtime_overlayfs()           OverlayFS enabled by host build
compiled-d.log:VERBOSE [U=1000,P=71877]   singularity_runtime_overlayfs()           OverlayFS not supported by host build

so the pre-compiled one has no OverlayFS, thus it didn't "fail" since it didn't try (silently, even though .conf was instructing to "use" it).

[yarikoptic]

and the 2nd part of the mystery is that on that stretch box, with 3.16.0 kernel there is no overlayfs.ko, i.e. there is no overlay support

mvdoc@euclid ~/singularity/hauntedhouse $ find /lib/modules -iname '*overlay*'
mvdoc@euclid ~/singularity/hauntedhouse $ lsmod | grep overlay

so you would either need to install some backported kernel or just tune singularity.conf to not use overlays on that box...
in my case on smaug server I have newer kernel and overlayfs.ko is available and loaded, that is why it worked just fine.

on singularity side it would be nice if error message in this case was a bit more descriptive -- e.g. if it could say "failed to mount OverlayFS file system. Your kernel seems to lack support for it" or alike

edit 1: and it would be nice if singularity issued a warning whenever overlays are enabled in config but cannot be used. May be even unify behavior between two cases: built without overlays support, and/or overlayfs is not available. And then a uniform behavior -- either error + crash, or warning + continue (without overlay)

edit 2: not sure if I should patch backports for older systems to ship with default config setting to disable overlays... people are likely to be running backported kernels -- after all it is a repository of backports

[mvdoc]

Yeah, that was it—disabling overlay in /etc/singularity/singularity.conf for debian build makes it work again.

[gmkurtzer]

Is the lack of overlay support "normal", or is that a custom build?

[yarikoptic]

yes -- normal on older kernels eg 3.16 on stable debian (jessie) without kernel backports

[gmkurtzer]

Ahhhh, I keep getting tripped up by RH's backports. We could silence this error on systems that don't have overlay support at the time of compile.

[gmkurtzer]

Actually on second thought, this case is interesting. Why does the debian package support it, but the host doesn't? Different kernel?

[yarikoptic]

Yes. We provide a backport. People might have installed original or backports kernel. So it is impossible to know at build time since we've are dealing with backports here. Better be ready for both scenarios imho without requiring rebuild.

[gmkurtzer]

Is the configuration file change enough to cover both scenerios?

[sje30]

Hi @yarikoptic
I'm new to singularity, and just asked our sys admins to install it on our deptl unix machines.
(Ubuntu 14.04.5 LTS (trusty)).

Running the first example i get:

$ singularity run shub://vsoch/singularity-images
Progress |===================================| 100.0% 
ERROR  : Could not mount Singularity overlay: No such device
ABORT  : Retval = 255

Is the fix to edit /etc/singularity/singularity.conf to disable overlay?

# ENABLE OVERLAY: [BOOL]
# DEFAULT: yes
# Enabling this option will make it possible to specify bind paths to locations
# that do not currently exist within the container.
enable overlay = yes

singularity version: 2.3-dist

Thanks, stephen

[mvdoc]

Yes, that's how I fixed it in my case

On Jun 20, 2017 12:50, "Stephen Eglen" [e-mail] wrote:

Hi @yarikoptic https://github.com/yarikoptic
I'm new to singularity, and just asked our sys admins to install it on our
deptl unix machines.
(Ubuntu 14.04.5 LTS (trusty)).

Running the first example i get:

$ singularity run shub://vsoch/singularity-images
Progress |===================================| 100.0%
ERROR : Could not mount Singularity overlay: No such device
ABORT : Retval = 255

Is the fix to edit /etc/singularity/singularity.conf to disable overlay?

ENABLE OVERLAY: [BOOL]

DEFAULT: yes

Enabling this option will make it possible to specify bind paths to locations

that do not currently exist within the container.

enable overlay = yes

Thanks, stephen

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/singularityware/singularity/issues/721#issuecomment-309819937,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF3ZmgNguTizOj0fBkSQJYtP8rnJzIjdks5sF_hBgaJpZM4NvgOx
.

[gmkurtzer]

Hi @sje30,

What OS/kernel are you running?

Thanks!

[sje30]

thanks @gmkurtzer - see below. does that tell you all you need?

$ cat /etc/motd
=======================================================================
 Ubuntu 14.04.5 LTS (trusty) in DAMTP
 Host: mea, Group: CBI, Kernel: Linux 3.13.0-117-generic
 Avail Memory: 24105M, Swap: 7628M, Disk: 5888G
 Arch: x86_64, Intel(R) Xeon(R) CPU E5645 @ 2.40GHz [24 CPUs] @2395MHz
 Maker: Dell, Model: Precision WorkStation T5500 Tower
=======================================================================

$ uname -a
Linux mea 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[gmkurtzer]

Yep, thanks @sje30! I'm just surprised that overlay is not supported on that distribution.

[yarikoptic]

sorry @sje30 -- I was too late to the game, glad to see the issue is mitigated ;)

[sje30]

@gmkurtzer : my sys admin has changed the config file so that it runs for me on my machine. She also told me that other machines with newer kernels worked fine.

[dtrudg]

Closing as it looks like this issue was resolved. Let us know if any outstanding problems. Thanks!

[yarikoptic]

well, it is "mitigated" I would say but I would not be surprised if others would run into it and would not know what is going on. I think it could have been addressed by a more informative message, e.g. if instead of

Could not mount Singularity overlay: No such device

it would report instead

Could not mount Singularity overlay: overlay device is not present. Verify support of overlayfs in your kernel

it would be much more gentle on the user ;)

[KirstieJane]

Hi! Sorry to ping in a closed issue, but I'm wondering if there's anything I can do to fix this or if the answer is to ask the sys admin to turn off overlays.

The current config file says:

# overlayfs will be tried but if it is unavailable it will be silently ignored.
enable overlay = yes

which makes me wonder if I may have a different error to the original poster? (It looks pretty similar though!)

Here's the error:

$ singularity run shub://vsoch/singularity-images
Progress |===================================| 100.0% 
ERROR  : Could not mount Singularity overlay: No such device
ABORT  : Retval = 255

And here's some information about the machine I'm working on (happy to provide more!)

$ uname -a
Linux wbic-gate-4 3.16.0-6-amd64 #1 SMP Debian 3.16.57-2 (2018-07-14) x86_64 GNU/Linux

$ singularity --version
2.5.2-dist

Thank you for any help! And let me know if you'd prefer a new issue!

[vsoch]

Are you sure there is overlayfs on your cluster? As a sanity check, stackoverflow is suggesting something like this:

if grep -qw aufs /proc/filesystems; then
  echo aufs is available
fi

And I think others on the thread can add comment about good ways to test if overlayfs is present and working. I do know that overlayfs isn't a thing for Centos earlier than 7, but it looks like your kernel is in the 7 family. Running the second container hints that it's something about your Singularity configuration, because it doesn't do anything special with overlay, it just prints a message:

$ singularity run shub://vsoch/singularity-images
Progress |===================================| 100.0% 
You say please, but all I see is pizza..

Could you please run that command with --debug and get additional output?

To debug this, I would first add --debug and see if you learn anything new. I would then try getting some confirmation that you indeed have overlayfs. Then maybe try setting it to "no" (or your admin doing this) and then seeing if this gets past this error and shows the next error that might come up with having a start script that (does look like it would warrant) overlayfs, depending on what exists on the host already.

But on a higher level, is there any reason you are trying to deploy mindboggle on HPC? This would be much better suited to something like Heroku, or the equivalent cheap instance on Google Cloud or AWS. Webby things + super computer clusters, unless it's more of a container cluster that a web server can easily plug into, tend to be pretty messy.

Anyhoo, the overlay / Centos experts here should be able to give more insight!

[KirstieJane]

Sorry for editing my comment so much. I took out the example with mindcontrol because I figured that was distracting. The reason we're trying to package it (@Shotgunosine is the master behind it) is that our data can't leave our HPC and our admins won't install meteor 😩

Unfortunately --debug doesn't give any additional output:

singularity run shub://vsoch/singularity-images --debug
Progress |===================================| 100.0% 
ERROR  : Could not mount Singularity overlay: No such device
ABORT  : Retval = 255

When I run:

if grep -qw aufs /proc/filesystems; then
    echo "aufs is available"
else 
    echo "Nope, doesn't exist"
fi

I get Nope, doesn't exist

So that looks a lot like overlayfs is not available and can be safely turned off, right? I'll ping an email to the HPC and see if we can fix it that way.

[vsoch]

We found the bug! :bug: Yes, I'd definitely ask the admin to take it off (and verify that it shouldn't actually be there). Then move forward with debugging your container, and importantly, finding a way to get it running with a version of the application that uses (already existing) filesystems on the host where you have write.

Ah meteor... meteor... :boom: :scream:

[KirstieJane]

Thank you so much for all the help! ✨ 👾 🚀

[DrDaveD]

@KirstieJane you left off an important bit of context in your singularity.conf info. You showed:

# overlayfs will be tried but if it is unavailable it will be silently ignored.
enable overlay = yes

But the previous line ends

If 'try' is chosen,

so it would be better for singularity.conf to contain the default value, which is

enable overlay = try

[KirstieJane]

Just a little update that our cluster is now running singularity with overlayfs turned off and everything's working smoothly! Thank you all 👾

[vsoch]

Woohoo!

And I wish I could say the same for Github!! Oufh. --> https://status.github.com/messages I'm amazed your comment here came though! h