Singularity: Failed to mount squashfs image in (read only): Input/output error

[lidaof]

Version of Singularity:

2.4.2

Expected behavior

went to the shell of singularity image

Actual behavior

ERROR : Failed to mount squashfs image in (read only): Input/output error
ABORT : Retval = 255

Steps to reproduce behavior

singularity shell my_custome_image

I can run the test image without problem:

singularity shell vsoch-hello-world-master-latest.simg 
Singularity: Invoking an interactive shell within container...

Singularity vsoch-hello-world-master-latest.simg:~>

also my_custome_image can be run on another server...not sure how to check the difference between these 2 servers...

Thanks.

[lidaof]

` $ singularity -vvv shell my_custome_image.simg Increasing verbosity level (4) Singularity version: 2.4.2-dist Exec'ing: /opt/apps/singularity/2.4.2/libexec/singularity/cli/shell.exec Evaluating args: 'my_custome_image.simg' VERBOSE: Set messagelevel to: 4 VERBOSE: Initialize configuration file: /opt/apps/singularity/2.4.2/etc/singularity/singularity.conf VERBOSE: Got config key allow setuid = 'yes' VERBOSE: Got config key max loop devices = '256' VERBOSE: Got config key allow pid ns = 'yes' VERBOSE: Got config key config passwd = 'yes' VERBOSE: Got config key config group = 'yes' VERBOSE: Got config key config resolv_conf = 'yes' VERBOSE: Got config key mount proc = 'yes' VERBOSE: Got config key mount sys = 'yes' VERBOSE: Got config key mount dev = 'yes' VERBOSE: Got config key mount devpts = 'yes' VERBOSE: Got config key mount home = 'yes' VERBOSE: Got config key mount tmp = 'yes' VERBOSE: Got config key mount hostfs = 'no' VERBOSE: Got config key bind path = '/etc/localtime' VERBOSE: Got config key bind path = '/etc/hosts' VERBOSE: Got config key user bind control = 'yes' VERBOSE: Got config key enable overlay = 'try' VERBOSE: Got config key mount slave = 'yes' VERBOSE: Got config key sessiondir max size = '16' VERBOSE: Got config key allow container squashfs = 'yes' VERBOSE: Got config key allow container extfs = 'yes' VERBOSE: Got config key allow container dir = 'yes' VERBOSE: Initializing Singularity Registry VERBOSE: Adding value to registry: 'LIBEXECDIR' = '/opt/apps/singularity/2.4.2/libexec' VERBOSE: Adding value to registry: 'COMMAND' = 'shell' VERBOSE: Adding value to registry: 'MESSAGELEVEL' = '4' VERBOSE: Adding value to registry: 'VERSION' = '2.4.2-dist' VERBOSE: Adding value to registry: 'LOCALSTATEDIR' = '/opt/apps/singularity/2.4.2/var' VERBOSE: Adding value to registry: 'SYSCONFDIR' = '/opt/apps/singularity/2.4.2/etc' VERBOSE: Adding value to registry: 'BINDIR' = '/opt/apps/singularity/2.4.2/bin' VERBOSE: Adding value to registry: 'IMAGE' = 'my_custome_image.simg' VERBOSE: Set home (via getpwuid()) to: /home/d VERBOSE: Running SUID program workflow VERBOSE: Checking program has appropriate permissions VERBOSE: Checking configuration file is properly owned by root VERBOSE: Checking if singularity.conf allows us to run as suid VERBOSE: Invoking the user namespace VERBOSE: Not virtualizing USER namespace: running as SUID VERBOSE: No autofs bug path in configuration, skipping VERBOSE: Instantiating read only container image object VERBOSE: Checking that file pointer is a Singularity image VERBOSE: File is a valid SquashFS image VERBOSE: Adding value to registry: 'CLEANUPD_FD' = '-1' VERBOSE: Not virtualizing IPC namespace on user request VERBOSE: Not virtualizing PID namespace on user request VERBOSE: Not virtualizing network namespace on user request VERBOSE: Using session directory: /opt/apps/singularity/2.4.2/var/singularity/mnt/session VERBOSE: Adding value to registry: 'SESSIONDIR' = '/opt/apps/singularity/2.4.2/var/singularity/mnt/session' VERBOSE: Found available loop device: /dev/loop0 VERBOSE: Using loop device: /dev/loop0 VERBOSE: Mounting squashfs image: /dev/loop0 -> /opt/apps/singularity/2.4.2/var/singularity/mnt/container ERROR : Failed to mount squashfs image in (read only): Input/output error ABORT : Retval = 255

[lidaof]

`` $ ll /dev/loop* brw-rw---- 1 root disk 7, 0 Mar 3 23:59 /dev/loop0 brw-rw---- 1 root disk 7, 1 Mar 9 2017 /dev/loop1 brw-rw---- 1 root disk 7, 2 Mar 9 2017 /dev/loop2 brw-rw---- 1 root disk 7, 3 Mar 9 2017 /dev/loop3 brw-rw---- 1 root disk 7, 4 Mar 9 2017 /dev/loop4 brw-rw---- 1 root disk 7, 5 Mar 9 2017 /dev/loop5 brw-rw---- 1 root disk 7, 6 Mar 9 2017 /dev/loop6 brw-rw---- 1 root disk 7, 7 Mar 9 2017 /dev/loop7 crw-rw---- 1 root disk 10, 237 Feb 6 02:17 /dev/loop-control

[dtrudg]

Hi @lidaof - are you able to check for any messages in the output of dmesg when your experience this error?

[fennellt]

Did anything come of this?

[rkpandya]

Any update on this?

[dtrudg]

Hi @fennellt @rkpandya - I haven't ever been able to reproduce this error. and we haven't had follow-yup information. If you have also come across it are you able to provide singularity version, debug output, any kernel dmesg output at the time of the error, and information about the host operating system you are using? Thank you.

[mforde84]

having the same issue

[MontrealSergiy]

Have same issue with Singularity version: 2.5.2-dist on kwilczynski/ubuntu-16.04-docker

[jmstover]

I'm not able to reproduce this either. Using the standard docker Ubuntu image (can't access kwilczynski):

$ sudo /usr/local/singularity/2.5.2/bin/singularity build ub_1604.img docker://ubuntu:16.04
 [ ... ]
$ /usr/local/singularity/2.5.2/bin/singularity exec ub_1604.img cat /etc/os-release | grep ^VERSION=
VERSION="16.04.5 LTS (Xenial Xerus)"

@MontrealSergiy @mforde84 ... can either of you provide more information? dmesg output during a failed run? Any debug log output? Output of singularity selftest as well?

[MontrealSergiy]

selftest passes succesfully, debug messages for hello world vagrant@ubuntu1604:/vagrant$ sudo singularity -d run shub://vsoch/hello-world
Enabling debugging
Ending argument loop
Singularity version: 2.5.2-dist
Exec'ing: /usr/lib/x86_64-linux-gnu/singularity/cli/run.exec
Evaluating args: 'shub://vsoch/hello-world'
VERBOSE2 SINGULARITY_COMMAND_ASIS found as False
VERBOSE2 SINGULARITY_ROOTFS not defined (None)
VERBOSE2 SINGULARITY_METADATA_FOLDER found as None/.singularity.d
VERBOSE2 SINGULARITY_FIX_PERMS found as False
VERBOSE2 SINGULARITY_COLORIZE not defined (None)
VERBOSE2 SINGULARITY_DISABLE_CACHE found as False
VERBOSE2 SINGULARITY_CACHEDIR found as /root/.singularity
VERBOSE2 REGISTRY not defined (None)
VERBOSE2 NAMESPACE not defined (None)
VERBOSE2 SINGULARITY_DOCKER_ARCHITECTURE found as amd64
VERBOSE2 SINGULARITY_DOCKER_OS found as linux
VERBOSE2 SINGULARITY_ENVIRONMENT found as None/.singularity.d/env/90-environment.sh
VERBOSE2 SINGULARITY_RUNSCRIPT found as None/singularity
VERBOSE2 SINGULARITY_TESTFILE found as None/.singularity.d/test
VERBOSE2 SINGULARITY_DEFFILE found as None/.singularity.d/Singularity
VERBOSE2 SINGULARITY_HELPFILE found as None/.singularity.d/runscript.help
VERBOSE2 SINGULARITY_ENVBASE found as None/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as None/.singularity.d/labels.json
VERBOSE2 SINGULARITY_INCLUDECMD found as False
VERBOSE2 SINGULARITY_NOHTTPS found as False
VERBOSE2 SINGULARITY_PULLFOLDER found as .
VERBOSE2 SHUB_NAMEBYHASH not defined (None)
VERBOSE2 SHUB_NAMEBYCOMMIT not defined (None)
VERBOSE2 SHUB_CONTAINERNAME not defined (None)
VERBOSE2 SINGULARITY_CONTENTS found as /tmp/.singularity-layerfile.OcAGn8
VERBOSE2 SINGULARITY_PYTHREADS found as 9
DEBUG
* STARTING SINGULARITY PYTHON PULL **
VERBOSE2 SINGULARITY_CONTAINER found as shub://vsoch/hello-world
VERBOSE2 SINGULARITY_PULLFOLDER found as .
DEBUG Found uri shub://
DEBUG Headers found: Content-Type,Accept
DEBUG GET https://www.singularity-hub.org/api/container/vsoch/hello-world:latest
DEBUG Pull folder set to /vagrant
DEBUG Pulling to /vagrant/vsoch-hello-world-master-latest
DEBUG GET (stream) https://www.googleapis.com/download/storage/v1/b/singularityhub/o/singularityhub%2Fgithub.com%2Fvsoch%2Fhello-world%2Fe279432e6d3962777bb7b5e8d54f30f4347d867e%2Fed9755a0871f04db3e14971bec56a33f%2Fed9755a0871f04db3e14971bec56a33f.simg?generation=1508072025589820&alt=media
Progress |===================================| 100.0%
ERROR : Called singularity_config_get_value on uninitialized config subsystem
ABORT : Retval = 255
DEBUG Found None image
DEBUG Writing Singularity Hub image path to /tmp/.singularity-layerfile.OcAGn8
VERBOSE2 Writing file /tmp/.singularity-layerfile.OcAGn8 with mode w.
VERBOSE [U=0,P=11739] message_init() Set messagelevel to: 5
VERBOSE [U=0,P=11739] singularity_config_parse() Initialize configuration file: /etc/singularity/singularity.conf
DEBUG [U=0,P=11739] singularity_config_parse() Starting parse of configuration file /etc/singularity/singularity.conf
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key allow setuid = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key max loop devices = '256'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key allow pid ns = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key config passwd = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key config group = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key config resolv_conf = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key mount proc = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key mount sys = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key mount dev = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key mount devpts = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key mount home = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key mount tmp = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key mount hostfs = 'no'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key bind path = '/etc/localtime'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key bind path = '/etc/hosts'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key user bind control = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key enable overlay = 'try'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key mount slave = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key sessiondir max size = '16'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key allow container squashfs = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key allow container extfs = 'yes'
VERBOSE [U=0,P=11739] singularity_config_parse() Got config key allow container dir = 'yes'
DEBUG [U=0,P=11739] singularity_config_parse() Finished parsing configuration file '/etc/singularity/singularity.conf'
VERBOSE [U=0,P=11739] singularity_registry_init() Initializing Singularity Registry
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'PULLFOLDER' = '.'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(PULLFOLDER, .) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'LIBEXECDIR' = '/usr/lib/x86_64-linux-gnu'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(libexecdir, /usr/lib/x86_64-linux-gnu) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'COMMAND' = 'run'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(COMMAND, run) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'VERSION' = '2.5.2-dist'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(version, 2.5.2-dist) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'LOCALSTATEDIR' = '/var/lib'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(localstatedir, /var/lib) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'CONTENTS' = '/tmp/.singularity-layerfile.OcAGn8'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(CONTENTS, /tmp/.singularity-layerfile.OcAGn8) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'SYSCONFDIR' = '/etc'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(sysconfdir, /etc) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'BINDIR' = '/usr/bin'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(bindir, /usr/bin) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'CONTAINER' = 'shub://vsoch/hello-world'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(CONTAINER, shub://vsoch/hello-world) = 0
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'IMAGE' = '/vagrant/vsoch-hello-world-master-latest.simg'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(IMAGE, /vagrant/vsoch-hello-world-master-latest.simg) = 0
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'HOME'
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'TARGET_UID'
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'TARGET_GID'
DEBUG [U=0,P=11739] singularity_priv_init() Initializing user info
DEBUG [U=0,P=11739] singularity_priv_init() Set the calling user's username to: root
DEBUG [U=0,P=11739] singularity_priv_init() Marking uinfo structure as ready
DEBUG [U=0,P=11739] singularity_priv_init() Obtaining home directory
VERBOSE [U=0,P=11739] singularity_priv_init() Set home (via getpwuid()) to: /root
VERBOSE [U=0,P=11739] singularity_suid_init() Running SUID program workflow
VERBOSE [U=0,P=11739] singularity_suid_init() Checking program has appropriate permissions
VERBOSE [U=0,P=11739] singularity_suid_init() Checking configuration file is properly owned by root
VERBOSE [U=0,P=11739] singularity_suid_init() Checking if singularity.conf allows us to run as suid
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow setuid, yes)
DEBUG [U=0,P=11739] singularity_config_get_value_impl() Returning configuration value allow setuid='yes'
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow setuid, yes) = 1
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'NOSUID'
VERBOSE [U=0,P=11739] singularity_priv_userns() Invoking the user namespace
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow user ns, yes)
DEBUG [U=0,P=11739] singularity_config_get_value_impl() No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow user ns, yes) = 1
VERBOSE [U=0,P=11739] singularity_priv_userns() Not virtualizing USER namespace: running as root
DEBUG [U=0,P=11739] singularity_priv_userns() Returning singularity_priv_init(void)
DEBUG [U=0,P=11739] singularity_priv_drop() Running as root, not changing privileges
DEBUG [U=0,P=11739] singularity_config_get_value_multi_impl() No configuration entry found for 'autofs bug path'; returning default value ''
VERBOSE [U=0,P=11739] singularity_runtime_autofs() No autofs bug path in configuration, skipping
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'DAEMON_START'
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'DAEMON_JOIN'
DEBUG [U=0,P=11739] singularity_daemon_init() Not joining a daemon, daemon join not set
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'WRITABLE'
VERBOSE [U=0,P=11739] main() Instantiating read only container image object
DEBUG [U=0,P=11739] singularity_registry_get() Returning value from registry: 'IMAGE' = '/vagrant/vsoch-hello-world-master-latest.simg'
DEBUG [U=0,P=11739] singularity_image_init() Calling image_init for each file system module
DEBUG [U=0,P=11739] singularity_image_dir_init() Opening file descriptor to directory: /vagrant/vsoch-hello-world-master-latest.simg
DEBUG [U=0,P=11739] singularity_image_dir_init() This is not a directory based image
DEBUG [U=0,P=11739] singularity_image_squashfs_init() Checking if writable image requested
DEBUG [U=0,P=11739] singularity_image_squashfs_init() Opening file descriptor to image: /vagrant/vsoch-hello-world-master-latest.simg
VERBOSE [U=0,P=11739] singularity_image_squashfs_init() Checking that file pointer is a Singularity image
DEBUG [U=0,P=11739] singularity_image_squashfs_init() Checking for magic in the top of the file
VERBOSE [U=0,P=11739] singularity_image_squashfs_init() File is a valid SquashFS image
DEBUG [U=0,P=11739] singularity_image_init() got image_init type for squashfs
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow container squashfs, yes)
DEBUG [U=0,P=11739] singularity_config_get_value_impl() Returning configuration value allow container squashfs='yes'
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow container squashfs, yes) = 1
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'DAEMON_JOIN'
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'CLEANUPDIR'
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'CLEANUPD_FD' = '-1'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(CLEANUPD_FD, -1) = 0
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'DAEMON_JOIN'
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'NOSESSIONCLEANUP'
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'NOCLEANUP'
DEBUG [U=0,P=11739] singularity_cleanupd() Not running a cleanup thread, no 'SINGULARITY_CLEANUPDIR' defined
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'DAEMON_JOIN'
DEBUG [U=0,P=11739] singularity_runtime_ns() Calling: _singularity_runtime_ns_ipc()
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow ipc ns, yes)
DEBUG [U=0,P=11739] singularity_config_get_value_impl() No configuration entry found for 'allow ipc ns'; returning default value 'yes'
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow ipc ns, yes) = 1
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'UNSHARE_IPC'
VERBOSE [U=0,P=11739] singularity_runtime_ns_ipc() Not virtualizing IPC namespace on user request
DEBUG [U=0,P=11739] singularity_runtime_ns() Calling: _singularity_runtime_ns_pid()
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow pid ns, yes)
DEBUG [U=0,P=11739] singularity_config_get_value_impl() Returning configuration value allow pid ns='yes'
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow pid ns, yes) = 1
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'UNSHARE_PID'
VERBOSE [U=0,P=11739] singularity_runtime_ns_pid() Not virtualizing PID namespace on user request
DEBUG [U=0,P=11739] singularity_runtime_ns() Calling: _singularity_runtime_ns_net()
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'UNSHARE_NET'
VERBOSE [U=0,P=11739] singularity_runtime_ns_net() Not virtualizing network namespace on user request
DEBUG [U=0,P=11739] singularity_runtime_ns() Calling: _singularity_runtime_ns_mnt()
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount slave, yes)
DEBUG [U=0,P=11739] singularity_config_get_value_impl() Returning configuration value mount slave='yes'
DEBUG [U=0,P=11739] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount slave, yes) = 1
DEBUG [U=0,P=11739] singularity_priv_escalate() Running as root, not changing privileges
DEBUG [U=0,P=11739] singularity_runtime_ns_mnt() Virtualizing FS namespace
DEBUG [U=0,P=11739] singularity_runtime_ns_mnt() Virtualizing mount namespace
DEBUG [U=0,P=11739] singularity_priv_drop() Running as root, not changing privileges
DEBUG [U=0,P=11739] singularity_runtime_ns_mnt() Making mounts slave
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'DAEMON_JOIN'
DEBUG [U=0,P=11739] singularity_sessiondir() Setting sessiondir
VERBOSE [U=0,P=11739] singularity_sessiondir() Using session directory: /var/lib/singularity/mnt/session
DEBUG [U=0,P=11739] singularity_sessiondir() Checking for session directory: /var/lib/singularity/mnt/session
DEBUG [U=0,P=11739] singularity_sessiondir() Obtaining the default sessiondir size
DEBUG [U=0,P=11739] singularity_config_get_value_impl() Returning configuration value sessiondir max size='16'
DEBUG [U=0,P=11739] singularity_sessiondir() Converted sessiondir size to: 16
DEBUG [U=0,P=11739] singularity_sessiondir() Creating the sessiondir size mount option length
DEBUG [U=0,P=11739] singularity_sessiondir() Got size length of: 9
DEBUG [U=0,P=11739] singularity_sessiondir() Creating the sessiondir size mount option string
DEBUG [U=0,P=11739] singularity_sessiondir() Checking to make sure the string was allocated correctly
DEBUG [U=0,P=11739] singularity_sessiondir() Mounting sessiondir tmpfs: /var/lib/singularity/mnt/session
VERBOSE [U=0,P=11739] singularity_registry_set() Adding value to registry: 'SESSIONDIR' = '/var/lib/singularity/mnt/session'
DEBUG [U=0,P=11739] singularity_registry_set() Returning singularity_registry_set(SESSIONDIR, /var/lib/singularity/mnt/session) = 0
DEBUG [U=0,P=11739] singularity_registry_get() Returning NULL on 'DAEMON_JOIN'
DEBUG [U=0,P=11739] singularity_image_mount() Figuring out which mount module to use...
DEBUG [U=0,P=11739] singularity_image_mount() Calling squashfs_mount
DEBUG [U=0,P=11739] singularity_config_get_value_impl() Returning configuration value max loop devices='256'
DEBUG [U=0,P=11739] singularity_image_bind() Entered singularity_image_bind()
DEBUG [U=0,P=11739] singularity_image_bind() Converting max_loop_devs_string to int: '256'
DEBUG [U=0,P=11739] singularity_image_bind() Converted max_loop_devs_string to int: '256' -> 256
DEBUG [U=0,P=11739] singularity_image_bind() Checking if this image has been properly opened
DEBUG [U=0,P=11739] singularity_priv_escalate() Running as root, not changing privileges
DEBUG [U=0,P=11739] singularity_image_bind() Finding next available loop device...
DEBUG [U=0,P=11739] singularity_priv_drop() Running as root, not changing privileges
VERBOSE [U=0,P=11739] singularity_image_bind() Found available loop device: /dev/loop0
DEBUG [U=0,P=11739] singularity_image_bind() Setting LO_FLAGS_AUTOCLEAR
DEBUG [U=0,P=11739] singularity_image_bind() Using image offset: 31
DEBUG [U=0,P=11739] singularity_priv_escalate() Running as root, not changing privileges
DEBUG [U=0,P=11739] singularity_image_bind() Setting loop device flags
DEBUG [U=0,P=11739] singularity_priv_drop() Running as root, not changing privileges
VERBOSE [U=0,P=11739] singularity_image_bind() Using loop device: /dev/loop0
VERBOSE [U=0,P=11739] singularity_image_squashfs_mount() Mounting squashfs image: /dev/loop0 -> /var/lib/singularity/mnt/container
ERROR [U=0,P=11739] singularity_image_squashfs_mount() Failed to mount squashfs image in (read only): Input/output error
ABORT [U=0,P=11739] singularity_image_squashfs_mount() Retval = 255
vagrant@ubuntu1604:/vagrant$

[jmstover]

So, this is the specific call that's failing:

singularity_mount(loop_dev, mount_point, "squashfs", MS_NOSUID|MS_RDONLY|MS_NODEV, "errors=remount-ro")

I'm guessing that singularity was built in that vagrant VM as well? So it's not something like MS_NODEV, or MS_RDONLY, being defined where you _built_ singularity, but not existing where you're trying to run it?

[MontrealSergiy]

hard to tell, I think I was truing to build but then installed. It was not there yet. Is there some way to change the permissions to whatever singularity needs?

[MontrealSergiy]

Yup I was using /vagrant folder instead of /home/vagrant/ now have less severe problem /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
RaawwWWWWWRRRR!

[jmstover]

warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

That's just the locale coming from the host, and being passed to container (I think). Try running like:
SINGULARITYENV_LC_ALL=C singularity run shub://vsoch/hello-world

[mforde84]

no luck:

/apps/software/singularity-2.6.0/tensorflow/tensorflow:1.4.0-rc0-gpu-py3# SINGULARITYENV_LC_ALL=C /usr/bin/singularity shell --writable -B /gpfs/data /apps/software/singularity-2.6.0/tensorflow/tensorflow:1.4.0-rc0-gpu-py3/
Singularity: Invoking an interactive shell within container...

Singularity tensorflow:1.4.0-rc0-gpu-py3:~> pip3 install summa
Looking in indexes: https://pypi.org/simple, https://packagecloud.io/github/git-lfs/pypi/simple
Collecting summa
  Using cached https://files.pythonhosted.org/packages/02/11/2be56c3740268f9a6e2379a74ffd624c5c0237c4b54093b48a7de8325987/summa-1.0.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-h_49_ap4/summa/setup.py", line 11, in <module>
        long_description = open('README').read(),
      File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 23: ordinal not in range(128)

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-h_49_ap4/summa/

[MontrealSergiy]

It is ok, despite the warning, actually RaawwWWWWWRRRR!! is expected output (probably hello world in Singularish)

[jmstover]

@mforde84 ... you're getting a Python error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 23: ordinal not in range(128)

[mforde84]

Yes, I'm aware. lol. My assumption is the problem is caused by the same issue in both instances. Specifically the env isn't recognizing utf-8 encodings, though nothing I've tried so far fixes the issue unfortunately.

[dtrudg]

@mforde84 - what's the container you are running here? It is possibly missing locales.

E.g. for debian/ubuntu containers and things expecting UTF-8 you want to locale-gen en_US.UTF-8 during the container build - and then don't set local vars to C.

[mforde84]

It's ubuntu 16.04. I built with:

/usr/bin/singularity build --sandbox /path/to/sand docker://tensorflow/tensorflow:1.4.0-rc0-gpu-py3

You'll have to excuse me, I'm a novice when it comes to containerization. Is there a way I can grab just the Singularity build file from docker without building the default they provide? That way I can check and see what localization is present in the build file, plus build in any additional python pip support I need for my users.

Thanks for the help,
M

[dtrudg]

@mforde84 - I'm afraid there isn't a way to get the Singularity definition file for a docker container. When you build a docker:// container into a Singularity container it is downloading and extracting the pre-built docker layers.

The locales stuff required to install that python package isn't in the tensorflow container. You can modify your sandbox as root to add it with....

sudo singularity shell -w /path/to/sand/

apt update && apt install locale
locale-gen en_US.UTF-8
pip3 install summa

This is ensuring the environment inside your container has a UTF8 locale available so you don't get the unicode encoding error.

[mforde84]

Unfortunate really. Thats okay though. Still getting the same error with the en_US.UTF-8 installed to the container:

$ apt update && apt install locales
$ locale-gen en_US.UTF-8
$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C
$ pip3 install summa 
Looking in indexes: https://pypi.org/simple, https://packagecloud.io/github/git-lfs/pypi/simple
Collecting summa
  Using cached https://files.pythonhosted.org/packages/02/11/2be56c3740268f9a6e2379a74ffd624c5c0237c4b54093b48a7de8325987/summa-1.0.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-v7_anyka/summa/setup.py", line 11, in <module>
        long_description = open('README').read(),
      File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 23: ordinal not in range(128)

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-v7_anyka/summa/

Im going to try to do the SINGULARITYENV variable option as well.

[mforde84]

yea, LC_ALL and SINGULARITYENV_LC_ALL dont affect it. Maybe it has to do with the python installation?

[jmstover]

@mforde84 ... Yeah, I've had this happen when in a Egg file there was a character that wasn't recognized (specifically on a added author name in my case). So, the install would just blow up when it got to that point.

I ended up needing to unpack the remote package, edit files to remove any non-ascii character, re-tar up. And then install from the local tar file.

[mforde84]

I can curl / wget the source distributable. Is there are a regex I can use to find non-ascii? If not, is the egg just a plain text file with installation instructions or something similar?

[mforde84]

Not a big python guy. Most of my stuff is bash or c.

[dtrudg]

@mforde84 - did you have SINGULARITYENV_LC_ALL LC_ALL set while you tried my workaround? you need to have them not forced for that workaround to be ok. Tested okay for me on an Ubuntu 18.04 and CentOS7 host machine.

[mforde84]

I thought I tried with both. I can try to unset both explicitly. Give me a sec.

[mforde84]

f* genius! i owe you all a round of beers! thanks for all the help guys.

m

[dtrudg]

@mforde84 - glad that worked. It's something that comes up a lot for these ML frameworks... similar issues can be found in the Torch, tensorflow etc. issues pages. Since Python 3.0 the str type is unicode, and the default source file encoding is UTF-8 so this comes up quite a bit. Most container images try to be as minimal as possible, so the omit locales if not absolutely needed - but this can cause issues if you want to add something later. Cheers.

[fennellt]

Hi @fennellt @rkpandya - I haven't ever been able to reproduce this error. and we haven't had follow-yup information. If you have also come across it are you able to provide singularity version, debug output, any kernel dmesg output at the time of the error, and information about the host operating system you are using? Thank you.

Sorry didn't mean to disappear, I was able to to get this to work in vagrant vm, but on a "standalone" vm am having trouble.

Package squashfs-tools-4.3-0.21.gitaae0aff4.el7.x86_64 already installed and latest version

singularity build lolcow.simg docker://godlovedc/lolcow
Docker image path: index.docker.io/godlovedc/lolcow:latest
Cache folder set to /home//.singularity/docker
[6/6] |===================================| 100.0%
Importing: base Singularity environment
Exploding layer: sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118.tar.gz
Exploding layer: sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a.tar.gz
Exploding layer: sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2.tar.gz
Exploding layer: sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e.tar.gz
Exploding layer: sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9.tar.gz
Exploding layer: sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945.tar.gz
Exploding layer: sha256:736a219344fbca3099ce5bd1d2dbfea74b22b830bac0e85ecca812c2983390cd.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: lolcow.simg
Cleaning up...

singularity shell --writable ./lolcow.simg
ERROR : Unable to open squashfs image in read-write mode: Read-only file system
ABORT : Retval = 255

sudo singularity shell --writable ./lolcow.simg
ERROR : Unable to open squashfs image in read-write mode: Read-only file system
ABORT : Retval = 255

I don't see anything interesting in dmesg. I am running with selinux Enforcing.
All I see in messages is

action-suid (redacted)> Unable to open squashfs image in read-write mode: Read-only file system
Singularity: action-suid (redacted)> Retval = 255

cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

singularity --version
2.6.0-HEAD.579c415

[fennellt]

Possible those containers need to run as root? I was able to run:

singularity run docker://library/python
Docker image path: index.docker.io/library/python:latest
Cache folder set to //.singularity/docker
Creating container runtime...
Exploding layer: sha256:05d1a5232b461a4b35424129580054caa878cd56f100e34282510bd4b4082e4d.tar.gz
Exploding layer: sha256:5cee356eda6bfe3a5a229cd3d964e722ade1da4381842b24e943b03a37aec1f2.tar.gz
Exploding layer: sha256:89d3385f0fd3c0c904ff6e87195bb46f5d9d309e8ddd91bc9b20855d103eeffb.tar.gz
Exploding layer: sha256:80ae6b477848b6e03aad8ec9c74f1fb80364e5c8b5fe9ca3ec793df84247f027.tar.gz
Exploding layer: sha256:28bdf9e584cc733044425cc166ce90743b25f7003cb06408b905030f76200b2a.tar.gz
Exploding layer: sha256:523b203f62bdce082cb1795a0fb6fef54fe3f1fb4d73ee09d7552be959ad0c9c.tar.gz
Exploding layer: sha256:e423ae9d5ac773a713d3494dd92cec42af173f94dc5ac8cf01fa7e0eddcdf826.tar.gz
Exploding layer: sha256:adc78e8180f739722c864bb84d5cc556e5fbead322f3b82489980b0855d37a1c.tar.gz
Exploding layer: sha256:5c4f0bc7295a43bb510f6378d4b2e02ba93024f54a2b6e4e1d770698dc132414.tar.gz
Exploding layer: sha256:29cd1db38a0b6319a12b64aabb5b484eebc89b2da3c2104b01561f3eaccd866d.tar.gz
Python 3.7.0 (default, Sep 5 2018, 03:25:31)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.

>

Same thing doesn't work if I try to pull and run

singularity pull docker://library/python
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/library/python:latest
Cache folder set to /redacted/.singularity/docker
Importing: base Singularity environment
Exploding layer: sha256:05d1a5232b461a4b35424129580054caa878cd56f100e34282510bd4b4082e4d.tar.gz
Exploding layer: sha256:5cee356eda6bfe3a5a229cd3d964e722ade1da4381842b24e943b03a37aec1f2.tar.gz
Exploding layer: sha256:89d3385f0fd3c0c904ff6e87195bb46f5d9d309e8ddd91bc9b20855d103eeffb.tar.gz
Exploding layer: sha256:80ae6b477848b6e03aad8ec9c74f1fb80364e5c8b5fe9ca3ec793df84247f027.tar.gz
Exploding layer: sha256:28bdf9e584cc733044425cc166ce90743b25f7003cb06408b905030f76200b2a.tar.gz
Exploding layer: sha256:523b203f62bdce082cb1795a0fb6fef54fe3f1fb4d73ee09d7552be959ad0c9c.tar.gz
Exploding layer: sha256:e423ae9d5ac773a713d3494dd92cec42af173f94dc5ac8cf01fa7e0eddcdf826.tar.gz
Exploding layer: sha256:adc78e8180f739722c864bb84d5cc556e5fbead322f3b82489980b0855d37a1c.tar.gz
Exploding layer: sha256:5c4f0bc7295a43bb510f6378d4b2e02ba93024f54a2b6e4e1d770698dc132414.tar.gz
Exploding layer: sha256:29cd1db38a0b6319a12b64aabb5b484eebc89b2da3c2104b01561f3eaccd866d.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ./python.simg
Cleaning up...
Done. Container is at: ./python.simg

singularity run python.simg
ERROR : Failed to mount squashfs image in (read only): No such device
ABORT : Retval = 255

[mforde84]

On my end, i had blocked squashfs kernel modules. Once i stopped blacklisting those, it mounted fine.

[mforde84]

Alternatively you could always build - - sandbox and that should work around your issue

[fennellt]

On

DO you remember how you went about doing that I don't see anything in sysctl.conf that looks like it would be doing that.

[fennellt]

I had a line in a file in /etc/modprobe.d/ . You nailed it!

[waynezw0618]

I am running singularity in virtualbox ubuntu installation, the host is MacOS. I still have this problem. Can anyone gives me some suggestion?

[rherban]

@waynezw0618 This thread has covered quite a bit of territory, can you please open a new issue with relevant information (OS version, singularity version, command and error output) to help troubleshoot your problem?

Thanks