Singularity: Singularity fails to run on HPC compute nodes

[bjlittle]

Version of Singularity:

2.2.1

Expected behavior

A simple test to see Singularity running on a HPC shared node and a compute node.

Submitting a simple job qsub test.sh, with two levels of scripts:

test.sh

#!/usr/bin/env bash
#PBS -N output 
#PBS -j oe
#PBS -l select=1:ncpus=1

echo "hello world!"
echo ">>>>"
singularity exec ${HOME}/containers/ubuntu.img cat /etc/os-release
echo "<<<<"
aprun -n 1 -N 1 ${HOME}/doit.sh

and doit.sh:

#!/usr/bin/env bash

echo "hello compute node world!"
echo ">>>>"
singularity -d exec /home/d03/itwl/containers/ubuntu.img cat /etc/os-release
echo "<<<<"

I'd expect the following output as a result of running qsub test.sh:

hello world!
>>>>
WARNING: Not mounting current directory: host does not support PR_SET_NO_NEW_PRIVS
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
<<<<
hello compute node world!
>>>>
WARNING: Not mounting current directory: host does not support PR_SET_NO_NEW_PRIVS
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
<<<<

I believe Singularity 2.3.1 should fix the PR_SET_NO_NEW_PRIVS warning that I'm seeing, but that's aside to this issue.

Actual behavior

hello world!
>>>>
WARNING: Not mounting current directory: host does not support PR_SET_NO_NEW_PRIVS
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
<<<<
hello compute node world!
>>>>
enabling debugging
ending argument loop
Exec'ing: /usr/lib64/singularity/cli/exec.exec /home/d03/itwl/containers/ubuntu.imgVERBOSE [U=6006,P=14495]   message.c:52:init()                        : Set messagelevel to: 5
DEBUG   [U=6006,P=14495]   privilege.c:66:singularity_priv_init()     : Called singularity_priv_init(void)
DEBUG   [U=6006,P=14495]   privilege.c:131:singularity_priv_init()    : Returning singularity_priv_init(void)
DEBUG   [U=6006,P=14495]   privilege.c:179:singularity_priv_drop()    : Dropping privileges to UID=6006, GID=1000
DEBUG   [U=6006,P=14495]   privilege.c:191:singularity_priv_drop()    : Confirming we have correct UID/GID
VERBOSE [U=6006,P=14495]   sexec.c:73:main()                          : Running NON-SUID program workflow
DEBUG   [U=6006,P=14495]   sexec.c:75:main()                          : Checking program has appropriate permissions
VERBOSE [U=6006,P=14495]   config_parser.c:43:singularity_config_open(): Opening configuration file: /etc/singularity/singularity.conf
DEBUG   [U=6006,P=14495]   config_parser.c:62:singularity_config_rewind(): Rewinding configuration file
VERBOSE [U=6006,P=14495]   sexec.c:85:main()                          : Checking that we are allowed to run as SUID
DEBUG   [U=6006,P=14495]   config_parser.c:104:singularity_config_get_bool(): Called singularity_config_get_bool(allow setuid, 1)
DEBUG   [U=6006,P=14495]   config_parser.c:80:singularity_config_get_value(): Called singularity_config_get_value(allow setuid)
VERBOSE [U=6006,P=14495]   config_parser.c:88:singularity_config_get_value(): Got config key allow setuid (= 'yes')
DEBUG   [U=6006,P=14495]   config_parser.c:110:singularity_config_get_bool(): Return singularity_config_get_bool(allow setuid, 1) = 1
VERBOSE [U=6006,P=14495]   sexec.c:87:main()                          : Checking if we were requested to run as NOSUID by user
DEBUG   [U=6006,P=14495]   util/util.c:92:envar_defined()             : Checking if environment variable is defined: SINGULARITY_NOSUID
VERBOSE [U=6006,P=14495]   util/util.c:94:envar_defined()             : Environment variable is undefined: SINGULARITY_NOSUID
VERBOSE [U=6006,P=14495]   sexec.c:91:main()                          : Checking for sexec-suid at /usr/lib64/singularity/sexec-suid
VERBOSE [U=6006,P=14495]   sexec.c:95:main()                          : Invoking SUID sexec: /usr/lib64/singularity/sexec-suid
VERBOSE [U=0,P=14495]      message.c:52:init()                        : Set messagelevel to: 5
DEBUG   [U=0,P=14495]      privilege.c:66:singularity_priv_init()     : Called singularity_priv_init(void)
DEBUG   [U=0,P=14495]      privilege.c:131:singularity_priv_init()    : Returning singularity_priv_init(void)
DEBUG   [U=0,P=14495]      privilege.c:179:singularity_priv_drop()    : Dropping privileges to UID=6006, GID=1000
DEBUG   [U=6006,P=14495]   privilege.c:191:singularity_priv_drop()    : Confirming we have correct UID/GID
VERBOSE [U=6006,P=14495]   sexec.c:46:main()                          : Running SUID program workflow
VERBOSE [U=6006,P=14495]   sexec.c:48:main()                          : Checking program has appropriate permissions
VERBOSE [U=6006,P=14495]   sexec.c:53:main()                          : Checking configuration file is properly owned by root
VERBOSE [U=6006,P=14495]   config_parser.c:43:singularity_config_open(): Opening configuration file: /etc/singularity/singularity.conf
DEBUG   [U=6006,P=14495]   config_parser.c:62:singularity_config_rewind(): Rewinding configuration file
VERBOSE [U=6006,P=14495]   sexec.c:62:main()                          : Checking that we are allowed to run as SUID
DEBUG   [U=6006,P=14495]   config_parser.c:104:singularity_config_get_bool(): Called singularity_config_get_bool(allow setuid, 1)
DEBUG   [U=6006,P=14495]   config_parser.c:80:singularity_config_get_value(): Called singularity_config_get_value(allow setuid)
VERBOSE [U=6006,P=14495]   config_parser.c:88:singularity_config_get_value(): Got config key allow setuid (= 'yes')
DEBUG   [U=6006,P=14495]   config_parser.c:110:singularity_config_get_bool(): Return singularity_config_get_bool(allow setuid, 1) = 1
VERBOSE [U=6006,P=14495]   sexec.c:67:main()                          : Checking if we were requested to run as NOSUID by user
DEBUG   [U=6006,P=14495]   util/util.c:92:envar_defined()             : Checking if environment variable is defined: SINGULARITY_NOSUID
VERBOSE [U=6006,P=14495]   util/util.c:94:envar_defined()             : Environment variable is undefined: SINGULARITY_NOSUID
DEBUG   [U=6006,P=14495]   util/util.c:102:envar_path()               : Checking environment variable is valid path: 'SINGULARITY_IMAGE'
VERBOSE [U=6006,P=14495]   util/util.c:50:envar()                     : Checking input from environment: 'SINGULARITY_IMAGE'
DEBUG   [U=6006,P=14495]   util/util.c:52:envar()                     : Checking environment variable is defined: SINGULARITY_IMAGE
DEBUG   [U=6006,P=14495]   util/util.c:58:envar()                     : Checking environment variable length (<= 4096): SINGULARITY_IMAGE
DEBUG   [U=6006,P=14495]   util/util.c:64:envar()                     : Checking environment variable has allowed characters: SINGULARITY_IMAGE
VERBOSE [U=6006,P=14495]   util/util.c:87:envar()                     : Obtained input from environment 'SINGULARITY_IMAGE' = '/home/d03/itwl/containers/ubuntu.img'
VERBOSE [U=6006,P=14495]   util/util.c:50:envar()                     : Checking input from environment: 'SINGULARITY_COMMAND'
DEBUG   [U=6006,P=14495]   util/util.c:52:envar()                     : Checking environment variable is defined: SINGULARITY_COMMAND
DEBUG   [U=6006,P=14495]   util/util.c:58:envar()                     : Checking environment variable length (<= 10): SINGULARITY_COMMAND
DEBUG   [U=6006,P=14495]   util/util.c:64:envar()                     : Checking environment variable has allowed characters: SINGULARITY_COMMAND
VERBOSE [U=6006,P=14495]   util/util.c:87:envar()                     : Obtained input from environment 'SINGULARITY_COMMAND' = 'exec'
DEBUG   [U=6006,P=14495]   action.c:55:singularity_action_init()      : Checking on action to run
DEBUG   [U=6006,P=14495]   action.c:67:singularity_action_init()      : Setting action to: exec
DEBUG   [U=6006,P=14495]   action.c:95:singularity_action_init()      : Getting current working directory path string
DEBUG   [U=6006,P=14495]   rootfs.c:71:singularity_rootfs_init()      : Checking on container source type
DEBUG   [U=6006,P=14495]   config_parser.c:62:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=6006,P=14495]   rootfs.c:80:singularity_rootfs_init()      : Figuring out where to mount Singularity container
DEBUG   [U=6006,P=14495]   config_parser.c:80:singularity_config_get_value(): Called singularity_config_get_value(container dir)
VERBOSE [U=6006,P=14495]   config_parser.c:88:singularity_config_get_value(): Got config key container dir (= '/var/singularity/mnt')
VERBOSE [U=6006,P=14495]   rootfs.c:86:singularity_rootfs_init()      : Set image mount path to: /var/singularity/mnt
DEBUG   [U=6006,P=14495]   image.c:53:rootfs_image_init()             : Inializing container rootfs image subsystem
DEBUG   [U=6006,P=14495]   util/util.c:92:envar_defined()             : Checking if environment variable is defined: SINGULARITY_WRITABLE
VERBOSE [U=6006,P=14495]   util/util.c:94:envar_defined()             : Environment variable is undefined: SINGULARITY_WRITABLE
VERBOSE [U=6006,P=14495]   image-util.c:48:singularity_image_check()  : Checking file is a Singularity image
DEBUG   [U=6006,P=14495]   image-util.c:59:singularity_image_check()  : Checking if first line matches key
VERBOSE [U=6006,P=14495]   image-util.c:62:singularity_image_check()  : File is a valid Singularity image
DEBUG   [U=6006,P=14495]   sessiondir.c:61:singularity_sessiondir_init(): Checking Singularity configuration for 'sessiondir prefix'
DEBUG   [U=6006,P=14495]   config_parser.c:62:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=6006,P=14495]   util/util.c:102:envar_path()               : Checking environment variable is valid path: 'SINGULARITY_SESSIONDIR'
VERBOSE [U=6006,P=14495]   util/util.c:50:envar()                     : Checking input from environment: 'SINGULARITY_SESSIONDIR'
DEBUG   [U=6006,P=14495]   util/util.c:52:envar()                     : Checking environment variable is defined: SINGULARITY_SESSIONDIR
VERBOSE [U=6006,P=14495]   util/util.c:54:envar()                     : Environment variable is NULL: SINGULARITY_SESSIONDIR
DEBUG   [U=6006,P=14495]   config_parser.c:80:singularity_config_get_value(): Called singularity_config_get_value(sessiondir prefix)
DEBUG   [U=6006,P=14495]   config_parser.c:96:singularity_config_get_value(): No configuration file entry found for 'sessiondir prefix'
DEBUG   [U=6006,P=14495]   sessiondir.c:82:singularity_sessiondir_init(): Set sessiondir to: /tmp/.singularity-session-6006.-2006279392.144116296076817365
DEBUG   [U=6006,P=14495]   util/file.c:245:s_mkpath()                 : Creating directory: /tmp/.singularity-session-6006.-2006279392.144116296076817365
DEBUG   [U=6006,P=14495]   sessiondir.c:98:singularity_sessiondir_init(): Opening sessiondir file descriptor
DEBUG   [U=6006,P=14495]   sessiondir.c:104:singularity_sessiondir_init(): Setting shared flock() on session directory
DEBUG   [U=6006,P=14495]   util/util.c:92:envar_defined()             : Checking if environment variable is defined: SINGULARITY_NOSESSIONCLEANUP
VERBOSE [U=6006,P=14495]   util/util.c:94:envar_defined()             : Environment variable is undefined: SINGULARITY_NOSESSIONCLEANUP
DEBUG   [U=6006,P=14495]   util/util.c:92:envar_defined()             : Checking if environment variable is defined: SINGULARITY_NOCLEANUP
VERBOSE [U=6006,P=14495]   util/util.c:94:envar_defined()             : Environment variable is undefined: SINGULARITY_NOCLEANUP
VERBOSE [U=6006,P=14495]   fork.c:74:singularity_fork()               : Forking child process
VERBOSE [U=6006,P=14495]   fork.c:90:singularity_fork()               : Hello from parent process
DEBUG   [U=6006,P=14495]   fork.c:109:singularity_fork()              : Assigning sigaction()s
DEBUG   [U=6006,P=14495]   fork.c:140:singularity_fork()              : Creating generic signal pipes
DEBUG   [U=6006,P=14495]   fork.c:148:singularity_fork()              : Creating sigcld signal pipes
DEBUG   [U=6006,P=14495]   fork.c:170:singularity_fork()              : Waiting on signal from watchdog
VERBOSE [U=6006,P=14496]   fork.c:78:singularity_fork()               : Hello from child process
DEBUG   [U=6006,P=14496]   fork.c:81:singularity_fork()               : Closing watchdog write pipe
DEBUG   [U=6006,P=14496]   fork.c:86:singularity_fork()               : Child process is returning control to process thread
DEBUG   [U=6006,P=14496]   ns.c:45:singularity_ns_unshare()           : Unsharing all namespaces
VERBOSE [U=6006,P=14496]   user.c:61:singularity_ns_user_unshare()    : Not virtualizing user namespace: running SUID root
DEBUG   [U=6006,P=14496]   config_parser.c:62:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=6006,P=14496]   config_parser.c:104:singularity_config_get_bool(): Called singularity_config_get_bool(allow pid ns, 1)
DEBUG   [U=6006,P=14496]   config_parser.c:80:singularity_config_get_value(): Called singularity_config_get_value(allow pid ns)
VERBOSE [U=6006,P=14496]   config_parser.c:88:singularity_config_get_value(): Got config key allow pid ns (= 'yes')
DEBUG   [U=6006,P=14496]   config_parser.c:110:singularity_config_get_bool(): Return singularity_config_get_bool(allow pid ns, 1) = 1
DEBUG   [U=6006,P=14496]   util/util.c:92:envar_defined()             : Checking if environment variable is defined: SINGULARITY_UNSHARE_PID
VERBOSE [U=6006,P=14496]   util/util.c:94:envar_defined()             : Environment variable is undefined: SINGULARITY_UNSHARE_PID
VERBOSE [U=6006,P=14496]   pid.c:59:singularity_ns_pid_unshare()      : Not virtualizing PID namespace on user request
DEBUG   [U=6006,P=14496]   config_parser.c:62:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=6006,P=14496]   config_parser.c:104:singularity_config_get_bool(): Called singularity_config_get_bool(mount slave, 1)
DEBUG   [U=6006,P=14496]   config_parser.c:80:singularity_config_get_value(): Called singularity_config_get_value(mount slave)
VERBOSE [U=6006,P=14496]   config_parser.c:88:singularity_config_get_value(): Got config key mount slave (= 'yes')
DEBUG   [U=6006,P=14496]   config_parser.c:110:singularity_config_get_bool(): Return singularity_config_get_bool(mount slave, 1) = 1
DEBUG   [U=6006,P=14496]   privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=6006)
DEBUG   [U=0,P=14496]      mnt.c:54:singularity_ns_mnt_unshare()      : Virtualizing FS namespace
DEBUG   [U=0,P=14496]      mnt.c:61:singularity_ns_mnt_unshare()      : Virtualizing mount namespace
DEBUG   [U=0,P=14496]      mnt.c:70:singularity_ns_mnt_unshare()      : Making mounts slave
DEBUG   [U=0,P=14496]      privilege.c:179:singularity_priv_drop()    : Dropping privileges to UID=6006, GID=1000
DEBUG   [U=6006,P=14496]   privilege.c:191:singularity_priv_drop()    : Confirming we have correct UID/GID
DEBUG   [U=6006,P=14496]   config_parser.c:62:singularity_config_rewind(): Rewinding configuration file
DEBUG   [U=6006,P=14496]   config_parser.c:104:singularity_config_get_bool(): Called singularity_config_get_bool(allow ipc ns, 1)
DEBUG   [U=6006,P=14496]   config_parser.c:80:singularity_config_get_value(): Called singularity_config_get_value(allow ipc ns)
DEBUG   [U=6006,P=14496]   config_parser.c:96:singularity_config_get_value(): No configuration file entry found for 'allow ipc ns'
DEBUG   [U=6006,P=14496]   config_parser.c:123:singularity_config_get_bool(): Undefined configuration for 'allow ipc ns', returning default: yes
DEBUG   [U=6006,P=14496]   util/util.c:92:envar_defined()             : Checking if environment variable is defined: SINGULARITY_UNSHARE_IPC
VERBOSE [U=6006,P=14496]   util/util.c:94:envar_defined()             : Environment variable is undefined: SINGULARITY_UNSHARE_IPC
VERBOSE [U=6006,P=14496]   ipc.c:59:singularity_ns_ipc_unshare()      : Not virtualizing IPC namespace on user request
DEBUG   [U=6006,P=14496]   rootfs.c:116:singularity_rootfs_mount()    : Checking 'container dir' mount location: /var/singularity/mnt
DEBUG   [U=6006,P=14496]   privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=6006)
VERBOSE [U=0,P=14496]      rootfs.c:119:singularity_rootfs_mount()    : Creating container dir: /var/singularity/mnt
DEBUG   [U=0,P=14496]      util/file.c:245:s_mkpath()                 : Creating directory: /var/singularity
DEBUG   [U=0,P=14496]      util/file.c:245:s_mkpath()                 : Creating directory: /var/singularity/mnt
DEBUG   [U=0,P=14496]      privilege.c:179:singularity_priv_drop()    : Dropping privileges to UID=6006, GID=1000
DEBUG   [U=6006,P=14496]   privilege.c:191:singularity_priv_drop()    : Confirming we have correct UID/GID
DEBUG   [U=6006,P=14496]   rootfs.c:127:singularity_rootfs_mount()    : Checking for rootfs_source directory: /var/singularity/mnt/source
DEBUG   [U=6006,P=14496]   privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=6006)
VERBOSE [U=0,P=14496]      rootfs.c:130:singularity_rootfs_mount()    : Creating container destination dir: /var/singularity/mnt/source
DEBUG   [U=0,P=14496]      util/file.c:245:s_mkpath()                 : Creating directory: /var/singularity/mnt/source
DEBUG   [U=0,P=14496]      privilege.c:179:singularity_priv_drop()    : Dropping privileges to UID=6006, GID=1000
DEBUG   [U=6006,P=14496]   privilege.c:191:singularity_priv_drop()    : Confirming we have correct UID/GID
DEBUG   [U=6006,P=14496]   rootfs.c:138:singularity_rootfs_mount()    : Checking for overlay_mount directory: /var/singularity/mnt/overlay
DEBUG   [U=6006,P=14496]   privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=6006)
VERBOSE [U=0,P=14496]      rootfs.c:141:singularity_rootfs_mount()    : Creating container mount dir: /var/singularity/mnt/overlay
DEBUG   [U=0,P=14496]      util/file.c:245:s_mkpath()                 : Creating directory: /var/singularity/mnt/overlay
DEBUG   [U=0,P=14496]      privilege.c:179:singularity_priv_drop()    : Dropping privileges to UID=6006, GID=1000
DEBUG   [U=6006,P=14496]   privilege.c:191:singularity_priv_drop()    : Confirming we have correct UID/GID
DEBUG   [U=6006,P=14496]   rootfs.c:149:singularity_rootfs_mount()    : Checking for overlay_final directory: /var/singularity/mnt/final
DEBUG   [U=6006,P=14496]   privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=6006)
VERBOSE [U=0,P=14496]      rootfs.c:152:singularity_rootfs_mount()    : Creating overlay final dir: /var/singularity/mnt/final
DEBUG   [U=0,P=14496]      util/file.c:245:s_mkpath()                 : Creating directory: /var/singularity/mnt/final
DEBUG   [U=0,P=14496]      privilege.c:179:singularity_priv_drop()    : Dropping privileges to UID=6006, GID=1000
DEBUG   [U=6006,P=14496]   privilege.c:191:singularity_priv_drop()    : Confirming we have correct UID/GID
DEBUG   [U=6006,P=14496]   image.c:121:rootfs_image_mount()           : Binding image to loop device
DEBUG   [U=6006,P=14496]   sessiondir.c:156:singularity_sessiondir_get(): Returning: /tmp/.singularity-session-6006.-2006279392.144116296076817365
DEBUG   [U=6006,P=14496]   loop-control.c:65:singularity_loop_bind()  : Opening image loop device file: /tmp/.singularity-session-6006.-2006279392.144116296076817365/image_loop_dev
DEBUG   [U=6006,P=14496]   loop-control.c:71:singularity_loop_bind()  : Requesting exclusive flock() on loop_dev lockfile
DEBUG   [U=6006,P=14496]   loop-control.c:94:singularity_loop_bind()  : Calculating image offset
VERBOSE [U=6006,P=14496]   image-util.c:48:singularity_image_check()  : Checking file is a Singularity image
DEBUG   [U=6006,P=14496]   image-util.c:59:singularity_image_check()  : Checking if first line matches key
VERBOSE [U=6006,P=14496]   image-util.c:62:singularity_image_check()  : File is a valid Singularity image
VERBOSE [U=6006,P=14496]   image-util.c:82:singularity_image_offset() : Calculating image offset
VERBOSE [U=6006,P=14496]   image-util.c:91:singularity_image_offset() : Found image at an offset of 31 bytes
DEBUG   [U=6006,P=14496]   image-util.c:96:singularity_image_offset() : Returning image_offset(image_fp) = 31
DEBUG   [U=6006,P=14496]   privilege.c:152:singularity_priv_escalate(): Temporarily escalating privileges (U=6006)
DEBUG   [U=0,P=14496]      loop-control.c:101:singularity_loop_bind() : Finding next available loop device...
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop0: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop1: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop2: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop3: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop4: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop5: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop6: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop7: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop8: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop9: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop10: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop11: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop12: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop13: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop14: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop15: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop16: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop17: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop18: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop19: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop20: No such device or address

snip

VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop121: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop122: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop123: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop124: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop125: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop126: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:113:singularity_loop_bind() : Could not open loop device /dev/loop127: No such device or address
VERBOSE [U=0,P=14496]      loop-control.c:133:singularity_loop_bind() : Found avaialble loop device: (null)
DEBUG   [U=0,P=14496]      loop-control.c:135:singularity_loop_bind() : Setting loop device flags
DEBUG   [U=6006,P=14495]   fork.c:52:handle_sigchld()                 : Checking child pids: 14496 14496
DEBUG   [U=6006,P=14495]   fork.c:54:handle_sigchld()                 : Forwarding signal through sigchld_signal_wpipe
DEBUG   [U=6006,P=14495]   fork.c:196:singularity_fork()              : Parent process is exiting
DEBUG   [U=6006,P=14495]   util/util.c:102:envar_path()               : Checking environment variable is valid path: 'SINGULARITY_RUNDIR'
VERBOSE [U=6006,P=14495]   util/util.c:50:envar()                     : Checking input from environment: 'SINGULARITY_RUNDIR'
DEBUG   [U=6006,P=14495]   util/util.c:52:envar()                     : Checking environment variable is defined: SINGULARITY_RUNDIR
VERBOSE [U=6006,P=14495]   util/util.c:54:envar()                     : Environment variable is NULL: SINGULARITY_RUNDIR
DEBUG   [U=6006,P=14495]   sessiondir.c:118:singularity_sessiondir_init(): Cleanup thread waiting on child...
DEBUG   [U=6006,P=14495]   sessiondir.c:123:singularity_sessiondir_init(): Checking to see if we are the last process running in this sessiondir
VERBOSE [U=6006,P=14495]   sessiondir.c:125:singularity_sessiondir_init(): Cleaning sessiondir: /tmp/.singularity-session-6006.-2006279392.144116296076817365
DEBUG   [U=6006,P=14495]   util/file.c:267:s_rmdir()                  : Removing directory: /tmp/.singularity-session-6006.-2006279392.144116296076817365
<<<<
Application 1806620 resources: utime ~0s, stime ~1s, Rss ~5120, inblocks ~8363, outblocks ~74

Steps to reproduce behavior

As described above.

From what I understand, our HPC compute nodes have the busybox version 1.20.2 - BusyBox v1.20.2 (2015-04-16 20:27:13 UTC) multi-call binary.

The image that I'm testing with is the vanilla ubuntu:latest.

If you need further information, then please ask and I'll do my best to help.

Thanks!

[AdamSimpson]

On our Cray XK7 compute nodes the admins had to load the loop.ko, ext3.ko, mbcache.ko, and jbd.ko kernel modules to enable the mounting of the loop device.

[bjlittle]

@AdamSimpson Thanks, that's really good to know.

I'll speak to our sys admins. I know that they will be shortly patching our compute nodes to allow Shifter to work on our test Cray XC40 ... perhaps those patches required for Shifter will also support the operation of Singularity or resolve this particular issue ...

[vsoch]

This would be good to have in docs for other Cray users - is this issue and fix universal for all or most systems or specific to your cluster?

[bjlittle]

Our sys admin went grubbing around on the compute nodes and neither of the kernel objects mentioned by @AdamSimpson were registered within the installed BusyBox.

@vsoch Do you know the ko requirements of singularity? or is that way too vendor specific?

Still waiting for our compute nodes to be patched by Cray on our test cluster for Shifter. When that happens then I'll report back as to whether that fixes singularity, and if the patch includes the asforementioned ko's.

[vsoch]

I'm definitely not the one to have expertise with this, but I can attest we've had issues with busybox in the past because there are a bunch of symbolic links / other. Just reading this post it does seem like there is some extra loading that needs to be done before use:

PS: the support for loop devices have to be compiled also in kernel.
Check for 
# Additional Block Devices
#
CONFIG_BLK_DEV_LOOP=m  (or Y)

If you don't have the modules that @AdamSimpson mentioned, perhaps they need to be registered and loaded? Are you able to try/test that?

[bjlittle]

@vsoch Thanks. That's really helpful. We should be able to work that out, or poke a knowledgeable resident Cray engineer. Although, I may need to buy some doughnuts first...

[vsoch]

[dtrudg]

Hi - closing this as not updated since July, and new versions of Singularity may have fixed your issues. Let us know if you still have problems. Thanks!