Singularity: Incorrect permissions for container pulled from dockerhub

Created on 2 Nov 2017 · 18Comments · Source: hpcng/singularity

Version of Singularity:

2.3.2 and 2.4

Expected behavior

Pulling a container from dockerhub should allow non-root users to run the container.

Actual behavior

Permissions are set to root within the singularity container and unprivileged user is unable to run the container.

Steps to reproduce behavior

Creating the image with 2.3.2

mvdoc@euclid ~/singularity $ singularity --version
2.3.2-heads/2.3.2.gbf5853e
mvdoc@euclid ~/singularity $ singularity pull --size 8000 docker://poldracklab/mriqc:latest

Running the image fails for permission errors:

mvdoc@euclid ~/singularity $ singularity run -e -C mriqc-latest.img
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/mriqc", line 6, in <module>
    from pkg_resources import load_entry_point
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2985, in <module>
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2971, in _call_aside
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2998, in _initialize_master_working_set
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 651, in _build_master
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 644, in __init__
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 700, in add_entry
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 1990, in find_on_path
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2399, in from_location
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2740, in _reload_version
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2364, in _version_from_file
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2532, in _get_metadata
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 1873, in get_metadata_lines
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 1861, in get_metadata
PermissionError: [Errno 13] Permission denied: '/usr/local/miniconda/lib/python3.6/site-packages/.wh.conda-4.3.11-py3.6.egg-info'

All is owned by root

mvdoc@euclid ~/singularity $ singularity exec mriqc-latest.img ls -l /usr/local/miniconda
total 140
-rw-rw-r--  1 root root  3699 Feb  4  2016 LICENSE.txt
drwxr-xr-x  2 root root 12288 Oct 10 06:21 bin
drwxr-xr-x  2 root root  4096 Oct 10 06:23 conda-bld
drwxr-xr-x  2 root root  4096 Oct 10 06:21 conda-meta
drwxr-xr-x  3 root root  4096 Oct 10 06:23 doc
drwxr-xr-x  2 root root  4096 Oct 10 06:21 envs
drwxr-xr-x  6 root root  4096 Oct 10 06:21 etc
drwxr-xr-x 23 root root 12288 Oct 10 06:21 include
drwxr-xr-x 14 root root 36864 Oct 10 06:21 lib
drwxr-xr-x  4 root root  4096 Oct 10 06:23 libexec
drwxr-xr-x  3 root root  4096 Oct 10 06:23 man
drwxr-xr-x 97 root root  4096 Oct 10 06:23 mkspecs
drwxr-xr-x  2 root root  4096 Oct 10 06:23 phrasebooks
drwxr-xr-x 97 root root  4096 Oct 10 06:21 pkgs
drwxr-xr-x 16 root root  4096 Oct 10 06:23 plugins
drwxr-xr-x 14 root root  4096 Oct 10 06:23 qml
drwxr-xr-x 19 root root  4096 Oct 10 06:21 share
drwxr-xr-x  3 root root  4096 Oct 10 06:21 ssl
drwxr-xr-x  2 root root 12288 Oct 10 06:23 translations
drwxr-xr-x  3 root root  4096 Oct 10 06:23 var
drwxr-xr-x  3 root root  4096 Oct 10 06:23 x86_64-conda_cos6-linux-gnu

Creating the image with 2.4

contematto@head1 ~/singularity $ ~/bin/singularity/bin/singularity --version
2.4-heads/2.4.g81bf030
contematto@head1 ~/singularity $ ~/bin/singularity/bin/singularity pull docker://poldracklab/mriqc:latest

contematto@head1 ~/singularity $ ~/bin/singularity/bin/singularity run mriqc-latest.img
ERROR  : Failed invoking the NEWUSER namespace runtime: Operation not permitted
ABORT  : Retval = 255

Source

mvdoc

Most helpful comment

The PR https://github.com/singularityware/singularity/pull/1106 is updated - hopefully moving toward a proper fix for the whiteout issues - comments sought!

dtrudg on 3 Nov 2017

😄2

All 18 comments

I think this is expected given a non sudo pull, isn't there a warning message (that you might have left out) when you issue pull without sudo?

vsoch on 2 Nov 2017

For 2.3.2 I get the same error pulling with sudo.

With 2.4 I haven't tried pulling will sudo (trying now). But I do get a warning which I cannot parse completely, because even if I run the image with sudo, I do get the same error.

WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.

mvdoc on 2 Nov 2017

@mvdoc I wonder if this is related to #1060 and https://github.com/singularityware/singularity/pull/1061? If so, it should be fixed in latest development. Can you test please?

GodloveD on 2 Nov 2017

With 2.4 even by pulling with sudo I get the same problem

contematto@head1 ~/singularity $ ~/bin/singularity/bin/singularity run mriqc-latest.img
ERROR  : Failed invoking the NEWUSER namespace runtime: Operation not permitted
ABORT  : Retval = 255

Latest dev

contematto@head1 ~/singularity $ ~/bin/singularity-devel/bin/singularity --version
2.4-development.gadd848c

building as non-root

contematto@head1 ~/singularity $ ~/bin/singularity-devel/bin/singularity pull docker://poldracklab/mriqc:latest
...
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
ingularity container built: ./mriqc-latest.img
Cleaning up...
Done. Container is at: ./mriqc-latest.img


contematto@head1 ~/singularity $ ~/bin/singularity-devel/bin/singularity run mriqc-latest.img
ERROR  : Failed invoking the NEWUSER namespace runtime: Operation not permitted
ABORT  : Retval = 255

building as root

contematto@head1 ~/singularity $ sudo ~/bin/singularity-devel/bin/singularity pull docker://poldracklab/mriqc:latest
...
Singularity container built: ./mriqc-latest.img
Cleaning up...
Done. Container is at: ./mriqc-latest.img


contematto@head1 ~/singularity $ ~/bin/singularity-devel/bin/singularity run mriqc-latest.img
ERROR  : Failed invoking the NEWUSER namespace runtime: Operation not permitted
ABORT  : Retval = 255

contematto@head1 ~/singularity $ sudo ~/bin/singularity-devel/bin/singularity run mriqc-latest.img
ERROR  : No valid /bin/sh in container
ABORT  : Retval = 255
contematto@head1 ~/singularity $ sudo ~/bin/singularity-devel/bin/singularity shell mriqc-latest.img
ERROR  : No valid /bin/sh in container
ABORT  : Retval = 255

mvdoc on 2 Nov 2017

OK. At least you got a different error.

Is it possible that there could be something wrong with the container runscript. In other words, is the runscript trying to carry out an operation that requires root? What if you just attempt to shell into the container after building with development? Or what if you try to run it as root (on a system that you don't care if you burn down 😆 ).

GodloveD on 3 Nov 2017

Running everything as root (from pulling to run/shell), this is what I get

euclid# /home/mvdoc/bin/singularity-devel/bin/singularity run mriqc-latest.img
ERROR  : No valid /bin/sh in container
ABORT  : Retval = 255

euclid# /home/mvdoc/bin/singularity-devel/bin/singularity shell mriqc-latest.img
ERROR  : No valid /bin/sh in container
ABORT  : Retval = 255

EDIT:

hold on, there's something weird happening -- probably how I built the devel version. I will try rebuilding again and will post more tomorrow. It seems that it's checking whether the container is valid at a wrong location

DEBUG   [U=0,P=31636]      singularity_registry_get()                Returning NULL on 'DAEMON_JOIN'
DEBUG   [U=0,P=31636]      singularity_image_mount()                 Figuring out which mount module to use...
DEBUG   [U=0,P=31636]      singularity_image_mount()                 Calling squashfs_mount
DEBUG   [U=0,P=31636]      singularity_config_get_value_impl()       Returning configuration value max loop devices='256'
DEBUG   [U=0,P=31636]      singularity_image_bind()                  Entered singularity_image_bind()
DEBUG   [U=0,P=31636]      singularity_image_bind()                  Converting max_loop_devs_string to int: '256'
DEBUG   [U=0,P=31636]      singularity_image_bind()                  Converted max_loop_devs_string to int: '256' -> 256
DEBUG   [U=0,P=31636]      singularity_image_bind()                  Checking if this image has been properly opened
DEBUG   [U=0,P=31636]      singularity_priv_escalate()               Running as root, not changing privileges
DEBUG   [U=0,P=31636]      singularity_image_bind()                  Finding next available loop device...
DEBUG   [U=0,P=31636]      singularity_priv_drop()                   Running as root, not changing privileges
VERBOSE [U=0,P=31636]      singularity_image_bind()                  Found available loop device: /dev/loop0
DEBUG   [U=0,P=31636]      singularity_image_bind()                  Setting LO_FLAGS_AUTOCLEAR
DEBUG   [U=0,P=31636]      singularity_image_bind()                  Using image offset: 31
DEBUG   [U=0,P=31636]      singularity_priv_escalate()               Running as root, not changing privileges
DEBUG   [U=0,P=31636]      singularity_image_bind()                  Setting loop device flags
DEBUG   [U=0,P=31636]      singularity_priv_drop()                   Running as root, not changing privileges
VERBOSE [U=0,P=31636]      singularity_image_bind()                  Using loop device: /dev/loop0
DEBUG   [U=0,P=31636]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=31636]      singularity_image_squashfs_mount()        Mounting squashfs image: /dev/loop0 -> /home/mvdoc/bin/singularity-devel/var/singularity/mnt/container
DEBUG   [U=0,P=31636]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=31636]      action_ready()                            Checking if container is valid at: /home/mvdoc/bin/singularity-2.4/var/singularity/mnt/container
ERROR   [U=0,P=31636]      action_ready()                            No valid /bin/sh in container
ABORT   [U=0,P=31636]      action_ready()                            Retval = 255

that is the squashfs is mounted at /home/mvdoc/bin/singularity-devel/var/singularity/mnt/container but at the last line it's checking at /home/mvdoc/bin/singularity-2.4/var/singularity/mnt/container.

mvdoc on 3 Nov 2017

@mvdoc one thing I learned while working on a different PR here is that, for development, you better not have any other singularity installed than the one you're developing or testing. Also, it doesn't seem to play very well when installed in non-standard locations, so stick to /usr/local/. The best way to do it without polluting your system (at least for me) is to spin up a VM running ubuntu, then git clone and test singularity there. Using vagrant makes this especially easy, I use the ubuntu/xenial64 box for reference. Also, every time you make a change and test something, you need to sudo make install or it will use bits and pieces from the old one. Sometimes I go overkill and recreate the whole VM just to make sure the system is clean ;)

ilmagico on 3 Nov 2017

This python error on the 2.3.2 problem is the aufs whiteout files issue: #884 #571

PermissionError: [Errno 13] Permission denied: '/usr/local/miniconda/lib/python3.6/site-packages/.wh.conda-4.3.11-py3.6.egg-info'

I suspect the same comes up with 2.4 when 2.4 is working correctly:

On latest development branch here I get the python error from the whitelist files still present in the extracted docker layers:

singularity run mriqc-latest.img
08:36 PM $ singularity run mriqc-latest.img
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/mriqc", line 6, in <module>
    from pkg_resources import load_entry_point
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2985, in <module>
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2971, in _call_aside
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2998, in _initialize_master_working_set
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 651, in _build_master
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 644, in __init__
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 700, in add_entry
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 1990, in find_on_path
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2399, in from_location
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2740, in _reload_version
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2364, in _version_from_file
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2532, in _get_metadata
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 1873, in get_metadata_lines
  File "/usr/local/miniconda/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 1861, in get_metadata
PermissionError: [Errno 13] Permission denied: '/usr/local/miniconda/lib/python3.6/site-packages/.wh.conda-4.3.11-py3.6.egg-info'

It works with my dodgy WIP aufs whiteout PR (https://github.com/singularityware/singularity/pull/1106):

singularity pull docker://poldracklab/mriqc:latest
...
singularity run mriqc-latest.img
08:23 PM $ singularity run mriqc-latest.img
Fontconfig warning: ignoring C.UTF-8: not a valid language tag
usage: mriqc [-h] [--version]
             [--participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]
             [--session-id SESSION_ID [SESSION_ID ...]]
             [--run-id RUN_ID [RUN_ID ...]] [--task-id TASK_ID [TASK_ID ...]]
             [-m [{T1w,bold,T2w} [{T1w,bold,T2w} ...]]] [-w WORK_DIR]
             [--report-dir REPORT_DIR] [--verbose-reports] [--write-graph]
             [--dry-run] [--profile] [--use-plugin USE_PLUGIN] [--no-sub]
             [--email EMAIL] [-v] [--webapi-url WEBAPI_URL]
             [--webapi-port WEBAPI_PORT] [--upload-strict] [--n_procs N_PROCS]
             [--mem_gb MEM_GB] [--testing] [-f] [--ica] [--hmc-afni]
             [--hmc-fsl] [--fft-spikes-detector] [--fd_thres FD_THRES]
             [--ants-nthreads ANTS_NTHREADS] [--ants-float]
             [--ants-settings ANTS_SETTINGS] [--deoblique] [--despike]
             [--start-idx START_IDX] [--stop-idx STOP_IDX]
             [--correct-slice-timing]
             bids_dir output_dir {participant,group} [{participant,group} ...]
mriqc: error: the following arguments are required: bids_dir, output_dir, analysis_level
2017-11-03 01:27:28,746 interface:INFO Closing display (if virtual)

dtrudg on 3 Nov 2017

@ilmagico yes I think that many problems are caused by multiple versions of Singularity on the same system. This issue seems like it could be related.

And let me also add that you should execute a make clean when installing a new version of Singularity.

GodloveD on 3 Nov 2017

Good call @dctrud! It's irritating that there are 3 different candidate bugs for this particular problem. Grrr!

GodloveD on 3 Nov 2017

This is the mriqc container - it's infamous for having 1) a whiteout file that triggers that permissions error so a library isn't correctly linked, and 2) most executables live in /root home. So - I would resolve the known issues (whiteout) and (doing clean install) and then go into debugging the specifics, if the error is still present.

vsoch on 3 Nov 2017

The PR https://github.com/singularityware/singularity/pull/1106 is updated - hopefully moving toward a proper fix for the whiteout issues - comments sought!

dtrudg on 3 Nov 2017

😄2

Closing this as PRs fixing whiteout issues have been merged to development. Let us know if there are any remaining problems. Thanks!

dtrudg on 22 Feb 2018

Hi,

I seem to have the same problem. I'm using 2.5.1 on CentOS7.

When I use exec or shell on a docker image the directories permissions are set to the docker image default user. But when I use pull it changes the directories permissions to root. In the example below /home/atlas is the home of the default user in the image

With exec used on a cache I get

singularity exec docker://afortiorama/image ls -ld /home/atlas
Docker image path: index.docker.io/afortiorama/image
[.....]
total 8
drwx------   2 atlas aforti 4096 Jul 22 16:24 atlas

but with the image from the pull I get

singularity exec -B $HOME $SINGULARITY_CACHEDIR/image-latest.simg ls -l /home/atlas
total 4
drwx------   2 root  root    95 Jul 22 13:09 atlas

/home/atlas contains some setup scripts that in the pull image are not accessible anymore.

afortiorama on 22 Jul 2018

Hi @afortiorama - the behavior you are seeing isn't quite the same as the issue originally reported here. In the original the permission errors stemmed from whiteout files that weren't being handled properly (but are in newer versions of Singularity).

The behavior you are seeing is actually by design for Singularity. When you run using exec docker://... directly, the container is temporarily created for a single use, under your account. The files can be assigned to your user account without any issue, as they won't be used again, or elsewhere.

When you build a container image, ownership is forced to root when the squashfs image is built. This is because the container image is a permanent thing, that can be moved to another host, where your user account may not exist - which could prevent it from working at all. With Singularity, unlike docker, the user who runs the container is the user inside the container, always.

For this reason, and the fact that we mount home directories, Singularity compatible containers should ideally avoid installing software and/or data into directories under /home, which are often chmod 700 by convention. I'm afraid the way around this is to modify the Dockerfile to build the original docker container differently - or to use a Singularity def file to chmod the directory (note due to /home mounts in Singularity this may not be a full solution for all usage).

dtrudg on 23 Jul 2018

Thanks for the detailed explanation. It is very useful.

afortiorama on 23 Jul 2018

Glad this is useful. I also wrote up some detail on singularity compatibility with docker containers as a lab note on the Sylabs web site here - https://www.sylabs.io/2018/04/singularity-compatibility-with-docker-containers/ - which may contain some other useful hints for issues you may or may not hit.

dtrudg on 23 Jul 2018

👍1