Singularity: again, container creation failed: mount error: can't mount overlay filesystem

[bkmz12332]

Version of Singularity:

3.0.1-168.gcc17206

Expected behavior

launch container

Actual behavior

[practice@access-02 test]$ singularity shell ubuntu-desktop-lxde-vnc_latest.sif
FATAL: container creation failed: mount error: can't mount overlay filesystem to /opt/ccoe/singularity/var/singularity/mnt/session/final: invalid argument
[practice@access-02 test]$ singularity -d shell ubuntu-desktop-lxde-vnc_latest.sif
VERBOSE [U=0,P=49827] print() Set messagelevel to: 5
DEBUG [U=0,P=49827] init() PIPE_EXEC_FD value: 9
VERBOSE [U=0,P=49827] init() Container runtime
VERBOSE [U=0,P=49827] is_suid() Check if we are running as setuid
DEBUG [U=0,P=49827] init() Overlay seems supported by kernel
DEBUG [U=0,P=49827] init() Drop privileges
DEBUG [U=10267,P=49827] init() Read json configuration from pipe
DEBUG [U=10267,P=49827] init() Set child signal mask
DEBUG [U=10267,P=49827] init() Create socketpair for smaster communication channel
DEBUG [U=10267,P=49827] init() Wait C and JSON runtime configuration from scontainer stage 1
DEBUG [U=10267,P=49841] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=10267,P=49841] init() Spawn scontainer stage 1
VERBOSE [U=10267,P=49841] priv_escalate() Get root privileges
DEBUG [U=0,P=49841] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=0,P=49841] prepare_scontainer_stage() Entering in scontainer stage 1
DEBUG [U=10267,P=49841] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=10267,P=49841] startup() Execute scontainer stage 1
DEBUG [U=10267,P=49841] Stage() Entering scontainer stage 1
DEBUG [U=10267,P=49841] Init() Entering image format intializer
DEBUG [U=10267,P=49841] Init() Check for image format sif
DEBUG [U=10267,P=49827] init() Receiving configuration from scontainer stage 1
DEBUG [U=10267,P=49827] init() Wait completion of scontainer stage1
VERBOSE [U=10267,P=49827] priv_escalate() Get root privileges
VERBOSE [U=0,P=49827] create_namespace() Create mount namespace
DEBUG [U=0,P=49827] init() Create RPC socketpair for communication between scontainer and RPC server
DEBUG [U=0,P=49827] init() Pass terminal control to child
VERBOSE [U=0,P=49827] init() Spawn smaster process
DEBUG [U=0,P=49852] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=49852] init() Spawn scontainer stage 2
VERBOSE [U=0,P=49852] create_namespace() Create mount namespace
VERBOSE [U=0,P=49853] init() Spawn RPC server
VERBOSE [U=10267,P=49827] startup() Execute smaster process
VERBOSE [U=0,P=49853] startup() Serve RPC requests
DEBUG [U=10267,P=49827] setupSessionLayout() Attempting to use overlayfs (enable overlay = try)
DEBUG [U=10267,P=49827] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=10267,P=49827] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=10267,P=49827] addRootfsMount() Mounting block [squashfs] image: /home/practice/test/ubuntu-desktop-lxde-vnc_latest.sif
DEBUG [U=10267,P=49827] addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=10267,P=49827] addKernelMount() Adding proc to mount list
DEBUG [U=10267,P=49827] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=10267,P=49827] addKernelMount() Adding sysfs to mount list
DEBUG [U=10267,P=49827] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=10267,P=49827] addDevMount() Adding dev to mount list
DEBUG [U=10267,P=49827] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=10267,P=49827] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=10267,P=49827] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=10267,P=49827] addHomeStagingDir() Staging home directory (/home/practice) at /opt/ccoe/singularity/var/singularity/mnt/session/home/practice
DEBUG [U=10267,P=49827] addHomeMount() Adding home directory mount [/opt/ccoe/singularity/var/singularity/mnt/session/home/practice:/home/practice] to list using layer: overlay
DEBUG [U=10267,P=49827] isLayerEnabled() Using Layer system: overlay
DEBUG [U=10267,P=49827] addTmpMount() Checking for 'mount tmp' in configuration file
DEBUG [U=10267,P=49827] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=10267,P=49827] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=10267,P=49827] addResolvConfMount() Adding /etc/resolv.conf to mount list
DEBUG [U=10267,P=49827] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=10267,P=49827] create() Mount all
DEBUG [U=10267,P=49827] mountGeneric() Mounting tmpfs to /opt/ccoe/singularity/var/singularity/mnt/session
DEBUG [U=10267,P=49827] mountImage() Mounting loop device /dev/loop0 to /opt/ccoe/singularity/var/singularity/mnt/session/rootfs
DEBUG [U=10267,P=49827] mountGeneric() Mounting overlay to /opt/ccoe/singularity/var/singularity/mnt/session/final
DEBUG [U=10267,P=49827] CleanupContainer() Cleanup container
FATAL [U=10267,P=49827] Master() container creation failed: mount error: can't mount overlay filesystem to /opt/ccoe/singularity/var/singularity/mnt/session/final: invalid argument

About system

uname -r
3.10.0-229.el7.x86_64
cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

help me plz

[omenos]

Possibly unrelated, but I can't help notice your 'about system' section. You might want to consider rebooting. Your release version is 7.5.1804, however, your kernel version is the very first release of 7.1 from March 2015. The latest kernel for 7.5 is 3.10.0-862.14.4. I don't know how you're doing that while still having a stable system. There's the possibility that there is some functionality missing/buggy by using that older kernel with newer system packages. Shouldn't be, but possible.

Can you share your /etc/singularity/singularity.conf file? If you built from source rather than RPM you may find it in /usr/local/etc/....

Cheers,
Mike

[bkmz12332]

singularyti.conf

singularity.conf.TXT

I don't know how you're doing that while still having a stable system

the system is work stable :-)
This is a node of HPC, the kernel version on all nodes is identical :-)

plan to use singularity on HPC

C Уважением,
Фатеев Илья

[cclerget]

Hi @bkmz12332 , as @omento mentioned you use an early kernel from CentOS 7 where overlay is (was ?) a tech preview, this is a bug in kernel not Singularity, if you update to the kernel provided with your corresponding CentOS release 7.5.1804 there is high chance it will work

EDIT:
If you can't, your only option is to set in singularity.conf :

• enable overlay = no
• enable underlay = yes

[bkmz12332]

Hi @bkmz12332 , as @omento mentioned you use an early kernel from CentOS 7 where overlay is (was ?) a tech preview, this is a bug in kernel not Singularity, if you update to the kernel provided with your corresponding CentOS release 7.5.1804 there is high chance it will work

EDIT:
If you can't, your only option is to set in singularity.conf :

• enable overlay = no
• enable underlay = yes

did not help :-(

#

Need to update the system kernel?

#

What version singularity will fit the pulling core?

[omenos]

OverlayFS support was introduced in RHEL/CentOS 7.4. Run rpm -qa kernel and check the results. If you have the any version of the kernel I listed above (the one for 7.5: 3.10.0-862), all you need to do is reboot your systems and it should take effect by default.

Cheers,
Mike

[bkmz12332]

rpm -qa kernel

rpm -qa kernel
kernel-3.10.0-229.el7.x86_64

:-(
seems to need to update the kernel

[cclerget]

@bkmz12332 It's weird that underlay didn't work for you, what's the debug output when you enable it ? Otherwise yes you will need to update the kernel

[bkmz12332]

@bkmz12332 It's weird that underlay didn't work for you, what's the debug output when you enable it ? Otherwise yes you will need to update the kernel

made a mistake in the config

Work for me! :-)

and work --nv to!

[cclerget]

Great !