Singularity: Failed invoking the NEWUSER namespace runtime: Invalid argument
I'm trying to test out Singularity on our cluster - I am not an administrator of the cluster, and I don't have root.
I created a few tar images on my laptop (Mac OSX, via vagrant), and sftp'ed them over to the cluster.
On one of the cluster nodes, I compiled and installed singularity with:
./autogen.sh
./configure --with-userns --with-slurm --prefix=$HOME
make && make install
So far so good. However, when I try to run the image, I get the error mentioned in the title:
$ singularity shell ~/tmp/tensorflow.tar
Opening tar archive, stand by...
ERROR : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT : Retval = 255
Running it with --debug results in the following output:
$ singularity --debug shell ~/tmp/tensorflow.tar
enabling debugging
ending argument loop
Exec'ing: /home/nelsonjs/libexec/singularity/cli/shell.exec /home/nelsonjs/tmp/tensorflow.tar
Opening tar archive, stand by...
VERBOSE [U=677,P=91936] message.c:52:message_init() : Set messagelevel to: 5
DEBUG [U=677,P=91936] privilege.c:73:singularity_priv_init() : Called singularity_priv_init(void)
DEBUG [U=677,P=91936] privilege.c:138:singularity_priv_init() : Returning singularity_priv_init(void)
VERBOSE [U=677,P=91936] privilege.c:264:singularity_priv_drop() : Could not restore EUID to 0: Operation not permitted (errno=1).
DEBUG [U=677,P=91936] privilege.c:267:singularity_priv_drop() : Dropping privileges to UID=677, GID=506 (1 supplementary GIDs)
DEBUG [U=677,P=91936] privilege.c:269:singularity_priv_drop() : Restoring supplementary groups
DEBUG [U=677,P=91936] privilege.c:286:singularity_priv_drop() : Confirming we have correct UID/GID
VERBOSE [U=677,P=91936] sexec.c:72:main() : Running NON-SUID program workflow
DEBUG [U=677,P=91936] sexec.c:74:main() : Checking program has appropriate permissions
VERBOSE [U=677,P=91936] config_parser.c:112:singularity_config_parse(): Initialize configuration file: /home/nelsonjs/etc/singularity/singularity.conf
DEBUG [U=677,P=91936] config_parser.c:124:singularity_config_parse(): Starting parse of configuration file /home/nelsonjs/etc/singularity/singularity.conf
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key allow setuid = 'no'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key allow pid ns = 'no'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key allow user image = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key protected image mode = 'none'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key enable overlay = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key config passwd = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key config group = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key config resolv_conf = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key mount proc = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key mount sys = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key mount dev = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key mount home = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key mount tmp = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key mount hostfs = 'no'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key bind path = '/etc/hosts'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key user bind control = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key mount slave = 'yes'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key container dir = '/var/lib/singularity/mnt'
VERBOSE [U=677,P=91936] config_parser.c:175:singularity_config_parse(): Got config key singularity user = 'singularity'
DEBUG [U=677,P=91936] config_parser.c:183:singularity_config_parse(): Finished parsing configuration file '/home/nelsonjs/etc/singularity/singularity.conf'
VERBOSE [U=677,P=91936] sexec.c:82:main() : Checking that we are allowed to run as SUID
DEBUG [U=677,P=91936] config_parser.c:293:_singularity_config_get_bool_chaCalled singularity_config_get_bool(allow setuid, yes)
DEBUG [U=677,P=91936] config_parser.c:238:_singularity_config_get_value_imReturning configuration value allow setuid='no'
DEBUG [U=677,P=91936] config_parser.c:304:_singularity_config_get_bool_chaReturn singularity_config_get_bool(allow setuid, yes) = 0
VERBOSE [U=677,P=91936] sexec.c:107:main() : Not invoking SUID mode: disallowed by the system administrator
DEBUG [U=677,P=91936] util/util.c:104:envar_path() : Checking environment variable is valid path: 'SINGULARITY_IMAGE'
VERBOSE [U=677,P=91936] util/util.c:52:envar() : Checking input from environment: 'SINGULARITY_IMAGE'
DEBUG [U=677,P=91936] util/util.c:54:envar() : Checking environment variable is defined: SINGULARITY_IMAGE
DEBUG [U=677,P=91936] util/util.c:60:envar() : Checking environment variable length (<= 4096): SINGULARITY_IMAGE
DEBUG [U=677,P=91936] util/util.c:66:envar() : Checking environment variable has allowed characters: SINGULARITY_IMAGE
VERBOSE [U=677,P=91936] util/util.c:89:envar() : Obtained input from environment 'SINGULARITY_IMAGE' = '/tmp/singularity-rundir.ilKVyC62/tensorflow.tar'
VERBOSE [U=677,P=91936] util/util.c:52:envar() : Checking input from environment: 'SINGULARITY_COMMAND'
DEBUG [U=677,P=91936] util/util.c:54:envar() : Checking environment variable is defined: SINGULARITY_COMMAND
DEBUG [U=677,P=91936] util/util.c:60:envar() : Checking environment variable length (<= 10): SINGULARITY_COMMAND
DEBUG [U=677,P=91936] util/util.c:66:envar() : Checking environment variable has allowed characters: SINGULARITY_COMMAND
VERBOSE [U=677,P=91936] util/util.c:89:envar() : Obtained input from environment 'SINGULARITY_COMMAND' = 'shell'
DEBUG [U=677,P=91936] action.c:54:singularity_action_init() : Checking on action to run
DEBUG [U=677,P=91936] action.c:62:singularity_action_init() : Setting action to: shell
DEBUG [U=677,P=91936] action.c:94:singularity_action_init() : Getting current working directory path string
DEBUG [U=677,P=91936] rootfs.c:71:singularity_rootfs_init() : Checking on container source type
DEBUG [U=677,P=91936] rootfs.c:79:singularity_rootfs_init() : Figuring out where to mount Singularity container
DEBUG [U=677,P=91936] config_parser.c:238:_singularity_config_get_value_imReturning configuration value container dir='/var/lib/singularity/mnt'
VERBOSE [U=677,P=91936] rootfs.c:82:singularity_rootfs_init() : Set image mount path to: /var/lib/singularity/mnt
DEBUG [U=677,P=91936] dir.c:44:rootfs_dir_init() : Inializing container rootfs dir subsystem
DEBUG [U=677,P=91936] util/util.c:94:envar_defined() : Checking if environment variable is defined: SINGULARITY_WRITABLE
VERBOSE [U=677,P=91936] util/util.c:96:envar_defined() : Environment variable is undefined: SINGULARITY_WRITABLE
DEBUG [U=677,P=91936] sessiondir.c:60:singularity_sessiondir_init(): Checking Singularity configuration for 'sessiondir prefix'
DEBUG [U=677,P=91936] util/util.c:104:envar_path() : Checking environment variable is valid path: 'SINGULARITY_SESSIONDIR'
VERBOSE [U=677,P=91936] util/util.c:52:envar() : Checking input from environment: 'SINGULARITY_SESSIONDIR'
DEBUG [U=677,P=91936] util/util.c:54:envar() : Checking environment variable is defined: SINGULARITY_SESSIONDIR
VERBOSE [U=677,P=91936] util/util.c:56:envar() : Environment variable is NULL: SINGULARITY_SESSIONDIR
DEBUG [U=677,P=91936] config_parser.c:226:_singularity_config_get_value_imNo configuration entry found for 'sessiondir prefix'; returning default value '/tmp/.singularity-session-'
DEBUG [U=677,P=91936] sessiondir.c:75:singularity_sessiondir_init(): Set sessiondir to: /tmp/.singularity-session-677.64770.30406
DEBUG [U=677,P=91936] util/file.c:263:s_mkpath() : Creating directory: /tmp/.singularity-session-677.64770.30406
DEBUG [U=677,P=91936] sessiondir.c:91:singularity_sessiondir_init(): Opening sessiondir file descriptor
DEBUG [U=677,P=91936] sessiondir.c:97:singularity_sessiondir_init(): Setting shared flock() on session directory
DEBUG [U=677,P=91936] util/util.c:94:envar_defined() : Checking if environment variable is defined: SINGULARITY_NOSESSIONCLEANUP
VERBOSE [U=677,P=91936] util/util.c:96:envar_defined() : Environment variable is undefined: SINGULARITY_NOSESSIONCLEANUP
DEBUG [U=677,P=91936] util/util.c:94:envar_defined() : Checking if environment variable is defined: SINGULARITY_NOCLEANUP
VERBOSE [U=677,P=91936] util/util.c:96:envar_defined() : Environment variable is undefined: SINGULARITY_NOCLEANUP
DEBUG [U=677,P=91936] fork.c:77:prepare_fork() : Creating parent/child coordination pipes.
VERBOSE [U=677,P=91936] fork.c:153:singularity_fork() : Forking child process
VERBOSE [U=677,P=91936] fork.c:171:singularity_fork() : Hello from parent process
DEBUG [U=677,P=91936] fork.c:190:singularity_fork() : Assigning sigaction()s
DEBUG [U=677,P=91936] fork.c:221:singularity_fork() : Creating generic signal pipes
DEBUG [U=677,P=91936] fork.c:229:singularity_fork() : Creating sigchld signal pipes
VERBOSE [U=677,P=91993] fork.c:157:singularity_fork() : Hello from child process
DEBUG [U=677,P=91936] fork.c:260:singularity_fork() : Waiting on signal from watchdog
DEBUG [U=677,P=91993] fork.c:160:singularity_fork() : Closing watchdog write pipe
DEBUG [U=677,P=91993] fork.c:167:singularity_fork() : Child process is returning control to process thread
DEBUG [U=677,P=91993] ns.c:45:singularity_ns_unshare() : Unsharing all namespaces
DEBUG [U=677,P=91993] user.c:47:singularity_ns_user_configured() : Checking if user namespaces are configured.
DEBUG [U=677,P=91993] config_parser.c:293:_singularity_config_get_bool_chaCalled singularity_config_get_bool(allow user ns, yes)
DEBUG [U=677,P=91993] config_parser.c:226:_singularity_config_get_value_imNo configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG [U=677,P=91993] config_parser.c:299:_singularity_config_get_bool_chaReturn singularity_config_get_bool(allow user ns, yes) = 1
DEBUG [U=677,P=91993] user.c:91:singularity_ns_user_unshare() : Attempting to virtualize the USER namespace
ERROR [U=677,P=91993] user.c:93:singularity_ns_user_unshare() : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT [U=677,P=91993] user.c:94:singularity_ns_user_unshare() : Retval = 255
DEBUG [U=677,P=91936] fork.c:287:singularity_fork() : Parent process is exiting
DEBUG [U=677,P=91936] util/util.c:104:envar_path() : Checking environment variable is valid path: 'SINGULARITY_RUNDIR'
VERBOSE [U=677,P=91936] util/util.c:52:envar() : Checking input from environment: 'SINGULARITY_RUNDIR'
DEBUG [U=677,P=91936] util/util.c:54:envar() : Checking environment variable is defined: SINGULARITY_RUNDIR
DEBUG [U=677,P=91936] util/util.c:60:envar() : Checking environment variable length (<= 4096): SINGULARITY_RUNDIR
DEBUG [U=677,P=91936] util/util.c:66:envar() : Checking environment variable has allowed characters: SINGULARITY_RUNDIR
VERBOSE [U=677,P=91936] util/util.c:89:envar() : Obtained input from environment 'SINGULARITY_RUNDIR' = '/tmp/singularity-rundir.ilKVyC62'
DEBUG [U=677,P=91936] sessiondir.c:111:singularity_sessiondir_init(): Cleanup thread waiting on child...
DEBUG [U=677,P=91936] sessiondir.c:116:singularity_sessiondir_init(): Checking to see if we are the last process running in this sessiondir
VERBOSE [U=677,P=91936] sessiondir.c:118:singularity_sessiondir_init(): Cleaning sessiondir: /tmp/.singularity-session-677.64770.30406
DEBUG [U=677,P=91936] util/file.c:285:s_rmdir() : Removing directory: /tmp/.singularity-session-677.64770.30406
VERBOSE [U=677,P=91936] sessiondir.c:126:singularity_sessiondir_init(): Cleaning run directory: /tmp/singularity-rundir.ilKVyC62
DEBUG [U=677,P=91936] util/file.c:285:s_rmdir() : Removing directory: /tmp/singularity-rundir.ilKVyC62
$
I've tried a variety of settings in ~/etc/singularity/singularity.conf based on http://singularity.lbl.gov/docs-config, but I haven't hit upon a config that works. I started with the default config, of course.
I believe that the kernel does support user namespaces, but I could be wrong. Here's some info in case it's useful:
$ cat /boot/config-$(uname -a | awk '{print $3}') | grep CONFIG_USER_NS
CONFIG_USER_NS=y
$ uname -a
Linux [hostname] 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)
fwip
All 31 comments
RedHat's implementation of user namespaces is very misleading.... While the kernel reports to support it, and the user space appears to be present, it is considered by Red Hat to be a "technology preview" and thus can only be enabled via a kernel boot argument (and even then, I'm not sure how functional it truly is).
To get proper support on the system, you will need to ask your system administrator to install Singularity to the system as root so it can leverage a set UID code path which does not require the user namespace.
Hope that helps!
gmkurtzer
on 14 Dec 2016
Ah, thank you for your help!
fwip
on 14 Dec 2016
My pleasure. I closed the ticket, but if you have any additional questions feel free to reopen it, create another or join our Slack or Google Group.
Thanks!
gmkurtzer
on 14 Dec 2016
Worked for me too - FWIW I simply sudo make install to a prefix that was sitting in my (non-root) homespace.
Thanks @gmkurtzer.
pelson
on 25 Jan 2017
I'm having the same problem on my university's HPC cluster even though the Admin installed Singularity 2.3.1 as root. I'm accessing Singularity via modules.
$ singularity exec ubuntu.img ls
ERROR : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT : Retval = 255
Here's some system info in case it's helpful:
$ uname --all
Linux [hostname] 3.10.0-514.26.2.el7.x86_64 #1 SMP Fri Jun 30 05:26:04 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)
If anyone has any other ideas let me know. I'd like to get this working on my university's HPC cluster if at all possible.
stillill
on 8 Aug 2017
@deepdawg, try running singularity with the -vvv option to get debug information on what it is doing. Also make sure that allow setuid = yes in /etc/singularity/singularity.conf (which is the default).
DrDaveD
on 15 Aug 2017
@DrDaveD I am having similar problems to @deepdawg on my university HPC cluster (also accessed using lmod). I do believe that singularity was installed as root but I got the following debug output:
Increasing verbosity level (5)
Ending argument loop
Singularity version: 2.3-master.gadf5259
Exec'ing: /cm/shared/apps/singularity/libexec/singularity/cli/shell.exec
Evaluating args: '-c -B /scratch -B /data --home /home-1/[email protected]:/home/[email protected] work/software/singularity_images/keras208_cuda80_cudnn6_medimg.img'
VERBOSE [U=2027,P=36583] message_init() Set messagelevel to: 5
VERBOSE [U=2027,P=36583] singularity_config_parse() Initialize configuration file: /cm/shared/apps/singularity/etc/singularity/singularity.conf
DEBUG [U=2027,P=36583] singularity_config_parse() Starting parse of configuration file /cm/shared/apps/singularity/etc/singularity/singularity.conf
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key allow setuid = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key max loop devices = '256'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key allow pid ns = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key config passwd = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key config group = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key config resolv_conf = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key mount proc = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key mount sys = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key mount dev = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key mount home = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key mount tmp = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key mount hostfs = 'no'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key bind path = '/etc/localtime'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key bind path = '/etc/hosts'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key bind path = '/cm/shared'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key bind path = '/work-zfs/marccadmin'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key user bind control = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key enable overlay = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key mount slave = 'yes'
VERBOSE [U=2027,P=36583] singularity_config_parse() Got config key sessiondir max size = '16'
DEBUG [U=2027,P=36583] singularity_config_parse() Finished parsing configuration file '/cm/shared/apps/singularity/etc/singularity/singularity.conf'
VERBOSE [U=2027,P=36583] singularity_registry_init() Initializing Singularity Registry
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'CONTAIN' = '1'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(CONTAIN, 1) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'LIBEXECDIR' = '/cm/shared/apps/singularity/libexec'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(libexecdir, /cm/shared/apps/singularity/libexec) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'COMMAND' = 'shell'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(COMMAND, shell) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'VERSION' = '2.3-master.gadf5259'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(version, 2.3-master.gadf5259) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'LOCALSTATEDIR' = '/cm/shared/apps/singularity/var'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(localstatedir, /cm/shared/apps/singularity/var) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'HOME' = '/home-1/[email protected]:/home/[email protected]'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(HOME, /home-1/[email protected]:/home/[email protected]) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'DIR' = '/cm/shared/apps/singularity'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(DIR, /cm/shared/apps/singularity) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'IMAGES' = '/scratch/groups/jprince1/software/singularity_images'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(IMAGES, /scratch/groups/jprince1/software/singularity_images) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'SYSCONFDIR' = '/cm/shared/apps/singularity/etc'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(sysconfdir, /cm/shared/apps/singularity/etc) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'BINDPATH' = '/data,/scratch,'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(BINDPATH, /data,/scratch,) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'BINDIR' = '/cm/shared/apps/singularity/bin'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(bindir, /cm/shared/apps/singularity/bin) = 0
VERBOSE [U=2027,P=36583] singularity_registry_set() Adding value to registry: 'IMAGE' = 'work/software/singularity_images/keras208_cuda80_cudnn6_medimg.img'
DEBUG [U=2027,P=36583] singularity_registry_set() Returning singularity_registry_set(IMAGE, work/software/singularity_images/keras208_cuda80_cudnn6_medimg.img) = 0
DEBUG [U=2027,P=36583] singularity_registry_get() Returning value from registry: 'HOME' = '/home-1/[email protected]:/home/[email protected]'
DEBUG [U=2027,P=36583] singularity_registry_get() Returning NULL on 'TARGET_UID'
DEBUG [U=2027,P=36583] singularity_registry_get() Returning NULL on 'TARGET_GID'
DEBUG [U=2027,P=36583] singularity_priv_init() Initializing user info
DEBUG [U=2027,P=36583] singularity_priv_init() Set the calling user's username to: [email protected]
DEBUG [U=2027,P=36583] singularity_priv_init() Marking uinfo structure as ready
DEBUG [U=2027,P=36583] singularity_priv_init() Obtaining home directory
VERBOSE [U=2027,P=36583] singularity_priv_init() Set home (via SINGULARITY_HOME) to: /home/[email protected]
VERBOSE [U=2027,P=36583] singularity_priv_init() Set the home directory (via SINGULARITY_HOME) to: /home-1/[email protected]
VERBOSE [U=2027,P=36583] singularity_suid_init() Running NON-SUID program workflow
DEBUG [U=2027,P=36583] singularity_suid_init() Checking program has appropriate permissions
VERBOSE [U=2027,P=36583] singularity_priv_userns() Invoking the user namespace
DEBUG [U=2027,P=36583] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow user ns, yes)
DEBUG [U=2027,P=36583] singularity_config_get_value_impl() No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG [U=2027,P=36583] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow user ns, yes) = 1
DEBUG [U=2027,P=36583] singularity_priv_userns() Attempting to virtualize the USER namespace
ERROR [U=2027,P=36583] singularity_priv_userns() Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT [U=2027,P=36583] singularity_priv_userns() Retval = 255
blakedewey
on 26 Sep 2017
@stillill did you figure out the issue?
caot
on 16 Aug 2018
im still having the same issues. singularity was install to root level system paths, though my users are still getting namespace errors
$ uname -r
2.6.32-573.12.1.el6.x86_64
$ singularity -vvv shell hello-world.simg
Increasing verbosity level (4)
Singularity version: 2.5.2-dist
Exec'ing: /usr/local/libexec/singularity/cli/shell.exec
Evaluating args: 'hello-world.simg'
VERBOSE: Set messagelevel to: 4
VERBOSE: Initialize configuration file: /usr/local/etc/singularity/singularity.conf
VERBOSE: Got config key allow setuid = 'yes'
VERBOSE: Got config key max loop devices = '256'
VERBOSE: Got config key allow pid ns = 'yes'
VERBOSE: Got config key config passwd = 'yes'
VERBOSE: Got config key config group = 'yes'
VERBOSE: Got config key config resolv_conf = 'yes'
VERBOSE: Got config key mount proc = 'yes'
VERBOSE: Got config key mount sys = 'yes'
VERBOSE: Got config key mount dev = 'yes'
VERBOSE: Got config key mount devpts = 'yes'
VERBOSE: Got config key mount home = 'yes'
VERBOSE: Got config key mount tmp = 'yes'
VERBOSE: Got config key mount hostfs = 'no'
VERBOSE: Got config key bind path = '/etc/localtime'
VERBOSE: Got config key bind path = '/etc/hosts'
VERBOSE: Got config key user bind control = 'yes'
VERBOSE: Got config key enable overlay = 'try'
VERBOSE: Got config key mount slave = 'yes'
VERBOSE: Got config key sessiondir max size = '16'
VERBOSE: Got config key allow container squashfs = 'yes'
VERBOSE: Got config key allow container extfs = 'yes'
VERBOSE: Got config key allow container dir = 'yes'
VERBOSE: Initializing Singularity Registry
VERBOSE: Adding value to registry: 'LIBEXECDIR' = '/usr/local/libexec'
VERBOSE: Adding value to registry: 'COMMAND' = 'shell'
VERBOSE: Adding value to registry: 'MESSAGELEVEL' = '4'
VERBOSE: Adding value to registry: 'VERSION' = '2.5.2-dist'
VERBOSE: Adding value to registry: 'LOCALSTATEDIR' = '/usr/local/var'
VERBOSE: Adding value to registry: 'SYSCONFDIR' = '/usr/local/etc'
VERBOSE: Adding value to registry: 'BINDIR' = '/usr/local/bin'
VERBOSE: Adding value to registry: 'IMAGE' = 'hello-world.simg'
VERBOSE: Set home (via getpwuid()) to: /home/a.cri.mforde
VERBOSE: Running NON-SUID program workflow
VERBOSE: Invoking the user namespace
ERROR : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT : Retval = 255
mforde84
on 16 Aug 2018
singularity couldn't parse the follows. Thank you @mforde84
$ singularity shell hello-world.simg -vvv
@mforde84 Indeed unprivileged user namespaces are not supported on el6. They're supported on el7 only as a technology preview. I can give you a pointer to instructions for enabling it there if you want to try it on el7.
@caot yes the -vvv has to come immediately after the singularity command.
DrDaveD
on 17 Aug 2018
@mforde84
The follows works for root user, however it's not acceptable in cluster.
[none-root@node-1001 demo]$ singularity shell hello-world.simg
ERROR : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT : Retval = 255
[none-root@node-1001 demo]$ su
[root@node-1001 demo]# singularity shell hello-world.simg
Singularity: Invoking an interactive shell within container...
Singularity hello-world.simg:~>
Ok, this is really confusing, why does this work?
$ sudo /usr/local/bin/singularity create container-centos7-test.img
...
$ sudo /usr/local/bin/singularity bootstrap container-centos7-test.img centos.def
...
$ /usr/local/bin/singularity shell container-centos7-test.img
Singularity: Invoking an interactive shell within container...
Singularity.container-centos7-test.img>
mforde84
on 17 Aug 2018
Im sorry, I don't understand what you're trying to say.
mforde84
on 17 Aug 2018
@mforde84 It could be helpful https://youtu.be/29NLgM9fnh4?t=437
caot
on 17 Aug 2018
I'm not clear how this is relevant to my question. I'm being told that I can only run containers with suid due to my kernel headers, yet I'm still able to generate a container and run it as an unprivileged user. Some clarification on why this works, yet other containers don't would be helpful. That or how I can generate containers with different kernel versions to support later versions of glibc would be helpful.
mforde84
on 17 Aug 2018
@mforde84 Assuming you're still running on el6, if you are successfully invoking a container as an unprivileged user, you must now be using a singularity with setuid enabled. The setuid bit is not on the singularity executable itself, it is on a helper executable ending with "-suid" in /usr/local/libexec/singularity/bin. -vvv should tell you whether it is using the NON-SUID workflow or not; the previous one you posted said it was.
DrDaveD
on 17 Aug 2018
Yep,
VERBOSE: Checking for sexec-suid at /usr/local/libexec/singularity/sexec-suid
VERBOSE: Invoking SUID sexec: /usr/local/libexec/singularity/sexec-suid
Is there a way I can force this behavior for dockers I pull or build from someone elses repo/tags? Just for functionality sake? For instance, the hello world container (from above) is running from non-suid, can I throw a flag to the command forcing the suid path? Or can I completely disable user namespace execution with a compile / configure option?
mforde84
on 17 Aug 2018
The image file run should make no difference to that, as far as I know. Are you sure you didn't change something? Try switching back and forth between the images with the same singularity installation.
If you're still seeing sexec-suid you must have an old version. For a while it has been called action-suid. I'm quite sure that all versions that use sexec have security vulnerabilities. Please upgrade.
DrDaveD
on 17 Aug 2018
Yea, I'm running 2.2.1 I believe. I can upgrade. Still testing to make sure this works as intended.
So just a clarification, should configuration and the initial make be performed by root user? e.g,
sudo su -
./configure
make
make install
Thats the only real differences I can see across my test cases. Maybe version difference as well. The installation that allows set uid was compiled as follows:
su - nonrootuser
./configure
make
sudo make install
mforde84
on 17 Aug 2018
Yes the second one is the right way to do it. Also ./autogetn.sh
If you're using el6 I advise getting it from EPEL. I support the rpm there. 2.6.0 is in epel-testing and will be in epel next week.
DrDaveD
on 17 Aug 2018
Great. Thanks. One additional question, and I can generate another ticket if you prefer. But one thing I'm running into issues with other peoples containers is a mount permission error:
eg.,
ERROR : Failed to mount image in (read only): Invalid argument
ABORT : Retval = 255
If I understand correctly, it's, by default, trying to mount to loop block device which a read only fs. Do you have suggestions on how to mount to a shared path with read/write/exec mount flags, say within a lmod sourced build directory?
mforde84
on 17 Aug 2018
Please create a new issue with all the details on how to reproduce. What you've given isn't sufficient for me to think of anything helpful.
DrDaveD
on 17 Aug 2018
Hi, I am running into the same error when running on my local cluster.
$uname --all
Linux irisa.cluster 2.6.32-754.10.1.el6.x86_64 #1 SMP Tue Jan 15 17:07:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$singularity --version
2.6.1-HEAD.9103f01
$singularity -d shell soft.simg
Exec'ing: /fusion/usc/opt/singularity/libexec/singularity/cli/shell.exec
Evaluating args: 'soft'
VERBOSE [U=6336,P=3039] message_init() Set messagelevel to: 5
VERBOSE [U=6336,P=3039] singularity_config_parse() Initialize configuration file: /fusion/usc/opt/singularity/etc/singularity/singularity.conf
DEBUG [U=6336,P=3039] singularity_config_parse() Starting parse of configuration file /fusion/usc/opt/singularity/etc/singularity/singularity.conf
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key allow setuid = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key max loop devices = '256'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key allow pid ns = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key config passwd = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key config group = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key config resolv_conf = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key mount proc = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key mount sys = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key mount dev = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key mount devpts = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key mount home = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key mount tmp = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key mount hostfs = 'no'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key bind path = '/etc/localtime'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key bind path = '/etc/hosts'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key user bind control = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key enable overlay = 'try'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key mount slave = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key sessiondir max size = '16'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key allow container squashfs = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key allow container extfs = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key allow container dir = 'yes'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key memory fs type = 'tmpfs'
VERBOSE [U=6336,P=3039] singularity_config_parse() Got config key always use nv = 'no '
DEBUG [U=6336,P=3039] singularity_config_parse() Finished parsing configuration file '/fusion/usc/opt/singularity/etc/singularity/singularity.conf'
DEBUG [U=6336,P=3039] singularity_config_get_value_impl() Returning configuration value always use nv='no '
VERBOSE [U=6336,P=3036] message_init() Set messagelevel to: 5
VERBOSE [U=6336,P=3036] singularity_config_parse() Initialize configuration file: /fusion/usc/opt/singularity/etc/singularity/singularity.conf
DEBUG [U=6336,P=3036] singularity_config_parse() Starting parse of configuration file /fusion/usc/opt/singularity/etc/singularity/singularity.conf
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key allow setuid = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key max loop devices = '256'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key allow pid ns = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key config passwd = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key config group = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key config resolv_conf = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key mount proc = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key mount sys = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key mount dev = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key mount devpts = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key mount home = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key mount tmp = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key mount hostfs = 'no'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key bind path = '/etc/localtime'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key bind path = '/etc/hosts'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key user bind control = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key enable overlay = 'try'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key mount slave = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key sessiondir max size = '16'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key allow container squashfs = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key allow container extfs = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key allow container dir = 'yes'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key memory fs type = 'tmpfs'
VERBOSE [U=6336,P=3036] singularity_config_parse() Got config key always use nv = 'no '
DEBUG [U=6336,P=3036] singularity_config_parse() Finished parsing configuration file '/fusion/usc/opt/singularity/etc/singularity/singularity.conf'
VERBOSE [U=6336,P=3036] singularity_registry_init() Initializing Singularity Registry
VERBOSE [U=6336,P=3036] singularity_registry_set() Adding value to registry: 'LIBEXECDIR' = '/fusion/usc/opt/singularity/libexec'
DEBUG [U=6336,P=3036] singularity_registry_set() Returning singularity_registry_set(libexecdir, /fusion/usc/opt/singularity/libexec) = 0
VERBOSE [U=6336,P=3036] singularity_registry_set() Adding value to registry: 'COMMAND' = 'shell'
DEBUG [U=6336,P=3036] singularity_registry_set() Returning singularity_registry_set(COMMAND, shell) = 0
VERBOSE [U=6336,P=3036] singularity_registry_set() Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG [U=6336,P=3036] singularity_registry_set() Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=6336,P=3036] singularity_registry_set() Adding value to registry: 'VERSION' = '2.6.1-HEAD.9103f01'
DEBUG [U=6336,P=3036] singularity_registry_set() Returning singularity_registry_set(version, 2.6.1-HEAD.9103f01) = 0
VERBOSE [U=6336,P=3036] singularity_registry_set() Adding value to registry: 'LOCALSTATEDIR' = '/fusion/usc/opt/singularity/var'
DEBUG [U=6336,P=3036] singularity_registry_set() Returning singularity_registry_set(localstatedir, /fusion/usc/opt/singularity/var) = 0
VERBOSE [U=6336,P=3036] singularity_registry_set() Adding value to registry: 'SYSCONFDIR' = '/fusion/usc/opt/singularity/etc'
DEBUG [U=6336,P=3036] singularity_registry_set() Returning singularity_registry_set(sysconfdir, /fusion/usc/opt/singularity/etc) = 0
VERBOSE [U=6336,P=3036] singularity_registry_set() Adding value to registry: 'BINDIR' = '/fusion/usc/opt/singularity/bin'
DEBUG [U=6336,P=3036] singularity_registry_set() Returning singularity_registry_set(bindir, /fusion/usc/opt/singularity/bin) = 0
VERBOSE [U=6336,P=3036] singularity_registry_set() Adding value to registry: 'IMAGE' = 'soft'
DEBUG [U=6336,P=3036] singularity_registry_set() Returning singularity_registry_set(IMAGE, soft) = 0
DEBUG [U=6336,P=3036] singularity_registry_get() Returning NULL on 'HOME'
DEBUG [U=6336,P=3036] singularity_registry_get() Returning NULL on 'TARGET_UID'
DEBUG [U=6336,P=3036] singularity_registry_get() Returning NULL on 'TARGET_GID'
DEBUG [U=6336,P=3036] singularity_priv_init() Initializing user info
DEBUG [U=6336,P=3036] singularity_priv_init() Set the calling user's username to: stagnerl
DEBUG [U=6336,P=3036] singularity_priv_init() Marking uinfo structure as ready
DEBUG [U=6336,P=3036] singularity_priv_init() Obtaining home directory
VERBOSE [U=6336,P=3036] singularity_priv_init() Set home (via getpwuid()) to: /home/stagnerl
VERBOSE [U=6336,P=3036] singularity_suid_init() Running NON-SUID program workflow
DEBUG [U=6336,P=3036] singularity_suid_init() Checking program has appropriate permissions
VERBOSE [U=6336,P=3036] singularity_priv_userns() Invoking the user namespace
DEBUG [U=6336,P=3036] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow user ns, yes)
DEBUG [U=6336,P=3036] singularity_config_get_value_impl() No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG [U=6336,P=3036] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow user ns, yes) = 1
DEBUG [U=6336,P=3036] singularity_priv_userns() Attempting to virtualize the USER namespace
ERROR [U=6336,P=3036] singularity_priv_userns() Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT [U=6336,P=3036] singularity_priv_userns() Retval = 255
The administrator compiled singularity from source using the following commands (he didn't want to pull it EPEL and singularity 3.0 has too many dependencies)
./autogen.sh
./configure --prefix=/usr/local --sysconfdir=/etc
make
sudo make install
only changing the configure paths. These commands should work or are there some configure flags missing? What needs to be done to get things working?
lstagner
on 14 Feb 2019
The key verbose message is "Running NON-SUID program workflow". Does /fusion/usc/opt/singularity/libexec/singularity/bin/action-suid exist, and does it have setuid root permissions? If so maybe /fusion doesn't allow executing setuid binaries.
DrDaveD
on 14 Feb 2019
action-suid does exist but it does not have setuid permissions
-rwxr-xr-x 1 admin usc 263992 Feb 13 13:31 action
-rwxr-xr-x 1 admin usc 606390 Feb 13 13:31 action-suid
...
The directory where singularity is installed has permissions drwxrwsx-x
So what is the fix? Its it just a matter of changing the permissions of the files in libexec/singularity/bin with chmod or does singularity need to be installed in a different place?
lstagner
on 14 Feb 2019
Something messed with the ownership of the files, and that probably cleared the setuid bit. Yes the *suid files in that directory need to be owned by root and chmod u+s.
DrDaveD
on 14 Feb 2019
Thanks that fixed it. Also for future readers the singularity.conf file also needs to owned by root.
lstagner
on 14 Feb 2019
@mforde84 Indeed unprivileged user namespaces are not supported on el6. They're supported on el7 only as a technology preview. I can give you a pointer to instructions for enabling it there if you want to try it on el7.
@caot yes the -vvv has to come immediately after the singularity command.
I came out the same problem, how to enable user namespaces unser el7.
I install singularity using conda.
$ singularity -vvv -d shell hello-world.simg
Increasing verbosity level (4)
Enabling debugging
Ending argument loop
Singularity version: 2.6.0-master.72a2295
Exec'ing: /PATH/TO/CONDA/conda_env/onecellpipe/libexec/singularity/cli/shell.exec
Evaluating args: 'hello-world.simg'
VERBOSE [U=1011,P=28675] message_init() Set messagelevel to: 5
VERBOSE [U=1011,P=28675] singularity_config_parse() Initialize configuration file: /PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf
DEBUG [U=1011,P=28675] singularity_config_parse() Starting parse of configuration file /PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key allow setuid = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key max loop devices = '256'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key allow pid ns = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key config passwd = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key config group = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key config resolv_conf = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key mount proc = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key mount sys = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key mount dev = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key mount devpts = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key mount home = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key mount tmp = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key mount hostfs = 'no'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key bind path = '/etc/localtime'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key bind path = '/etc/hosts'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key user bind control = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key enable overlay = 'try'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key mount slave = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key sessiondir max size = '16'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key allow container squashfs = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key allow container extfs = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key allow container dir = 'yes'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key memory fs type = 'tmpfs'
VERBOSE [U=1011,P=28675] singularity_config_parse() Got config key always use nv = 'no '
DEBUG [U=1011,P=28675] singularity_config_parse() Finished parsing configuration file '/PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf'
DEBUG [U=1011,P=28675] singularity_config_get_value_impl() Returning configuration value always use nv='no '
VERBOSE [U=1011,P=28671] message_init() Set messagelevel to: 5
VERBOSE [U=1011,P=28671] singularity_config_parse() Initialize configuration file: /PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf
DEBUG [U=1011,P=28671] singularity_config_parse() Starting parse of configuration file /PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key allow setuid = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key max loop devices = '256'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key allow pid ns = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key config passwd = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key config group = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key config resolv_conf = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key mount proc = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key mount sys = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key mount dev = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key mount devpts = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key mount home = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key mount tmp = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key mount hostfs = 'no'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key bind path = '/etc/localtime'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key bind path = '/etc/hosts'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key user bind control = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key enable overlay = 'try'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key mount slave = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key sessiondir max size = '16'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key allow container squashfs = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key allow container extfs = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key allow container dir = 'yes'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key memory fs type = 'tmpfs'
VERBOSE [U=1011,P=28671] singularity_config_parse() Got config key always use nv = 'no '
DEBUG [U=1011,P=28671] singularity_config_parse() Finished parsing configuration file '/PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf'
VERBOSE [U=1011,P=28671] singularity_registry_init() Initializing Singularity Registry
VERBOSE [U=1011,P=28671] singularity_registry_set() Adding value to registry: 'LIBEXECDIR' = '/PATH/TO/CONDA/conda_env/onecellpipe/libexec'
DEBUG [U=1011,P=28671] singularity_registry_set() Returning singularity_registry_set(libexecdir, /PATH/TO/CONDA/conda_env/onecellpipe/libexec) = 0
VERBOSE [U=1011,P=28671] singularity_registry_set() Adding value to registry: 'COMMAND' = 'shell'
DEBUG [U=1011,P=28671] singularity_registry_set() Returning singularity_registry_set(COMMAND, shell) = 0
VERBOSE [U=1011,P=28671] singularity_registry_set() Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG [U=1011,P=28671] singularity_registry_set() Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=1011,P=28671] singularity_registry_set() Adding value to registry: 'VERSION' = '2.6.0-master.72a2295'
DEBUG [U=1011,P=28671] singularity_registry_set() Returning singularity_registry_set(version, 2.6.0-master.72a2295) = 0
VERBOSE [U=1011,P=28671] singularity_registry_set() Adding value to registry: 'LOCALSTATEDIR' = '/PATH/TO/CONDA/conda_env/onecellpipe/var'
DEBUG [U=1011,P=28671] singularity_registry_set() Returning singularity_registry_set(localstatedir, /PATH/TO/CONDA/conda_env/onecellpipe/var) = 0
VERBOSE [U=1011,P=28671] singularity_registry_set() Adding value to registry: 'SYSCONFDIR' = '/PATH/TO/CONDA/conda_env/onecellpipe/etc'
DEBUG [U=1011,P=28671] singularity_registry_set() Returning singularity_registry_set(sysconfdir, /PATH/TO/CONDA/conda_env/onecellpipe/etc) = 0
VERBOSE [U=1011,P=28671] singularity_registry_set() Adding value to registry: 'BINDIR' = '/PATH/TO/CONDA/conda_env/onecellpipe/bin'
DEBUG [U=1011,P=28671] singularity_registry_set() Returning singularity_registry_set(bindir, /PATH/TO/CONDA/conda_env/onecellpipe/bin) = 0
VERBOSE [U=1011,P=28671] singularity_registry_set() Adding value to registry: 'IMAGE' = 'hello-world.simg'
DEBUG [U=1011,P=28671] singularity_registry_set() Returning singularity_registry_set(IMAGE, hello-world.simg) = 0
DEBUG [U=1011,P=28671] singularity_registry_get() Returning NULL on 'HOME'
DEBUG [U=1011,P=28671] singularity_registry_get() Returning NULL on 'TARGET_UID'
DEBUG [U=1011,P=28671] singularity_registry_get() Returning NULL on 'TARGET_GID'
DEBUG [U=1011,P=28671] singularity_priv_init() Initializing user info
DEBUG [U=1011,P=28671] singularity_priv_init() Set the calling user's username to: luyang
DEBUG [U=1011,P=28671] singularity_priv_init() Marking uinfo structure as ready
DEBUG [U=1011,P=28671] singularity_priv_init() Obtaining home directory
VERBOSE [U=1011,P=28671] singularity_priv_init() Set home (via getpwuid()) to: /home/luyang
VERBOSE [U=1011,P=28671] singularity_suid_init() Running NON-SUID program workflow
DEBUG [U=1011,P=28671] singularity_suid_init() Checking program has appropriate permissions
VERBOSE [U=1011,P=28671] singularity_priv_userns() Invoking the user namespace
DEBUG [U=1011,P=28671] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow user ns, yes)
DEBUG [U=1011,P=28671] singularity_config_get_value_impl() No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG [U=1011,P=28671] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow user ns, yes) = 1
DEBUG [U=1011,P=28671] singularity_priv_userns() Attempting to virtualize the USER namespace
ERROR [U=1011,P=28671] singularity_priv_userns() Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT [U=1011,P=28671] singularity_priv_userns() Retval = 255
luyang93
on 24 Jul 2019
@luyang93
Can you move your comment to a new issue and link to this one since it's over two years old and closed?
Additionally, can you include the output of:
cat /proc/sys/user/max_user_namespaces
Thanks!
jscook2345
on 24 Jul 2019
For the record, el7.6 supports user namespaces without being a technology preview. It just needs to be enabled, for example with
echo "user.max_user_namespaces = 15000" > /etc/sysctl.d/90-max_user_namespaces.conf
sysctl -p /etc/sysctl.d/90-max_user_namespaces.conf
Related issues
dtrudg
路
3Comments
onlyjob
路
5Comments
chrisgorgo
路
3Comments
mr-c
路
4Comments
GerardBCN
路
4Comments
Most helpful comment
RedHat's implementation of user namespaces is very misleading.... While the kernel reports to support it, and the user space appears to be present, it is considered by Red Hat to be a "technology preview" and thus can only be enabled via a kernel boot argument (and even then, I'm not sure how functional it truly is).
To get proper support on the system, you will need to ask your system administrator to install Singularity to the system as root so it can leverage a set UID code path which does not require the user namespace.
Hope that helps!