Singularity: Build from Docker. No valid /bin/sh in container

[A33a]

Version of Singularity:

2.4

Expected behavior

Build and shell container using Docker image

Actual behavior

After the container is build and executed,

singularity build sjupyter.simg docker://a33a/sjupyter
singularity shell sjupyter.simg

It returns:

ERROR  : No valid /bin/sh in container
ABORT  : Retval = 255

Steps to reproduce behavior

Try to build and shell the container on Scientific Linux 7.3. The Docker image uses autobuild so it is safe to test ;)

Environment

I use Scientific Linux 7.3 and the issue might be connected to my previous question and build command. Because I compiled the squashfs-tools and placed it in /local/usr/bin. I tried to build the image on Ubuntu and then moved the .simg file and it apparently worked.

Thank you in advance.

[A33a]

I see now what is wrong here. I was building as a user without sudo... Apparantlty, I though that users could build images without sudo.

In order to build images you need to have SUDO, a short answer for someome who is ended up here :)

[GodloveD]

Thank you for letting us know what you found out. I'm sure it will be of help to users in the future.

Just for completeness, let me note that some types of builds necessitate sudo but others don't. In this diagram red arrows represent builds that require root. Green arrows should work without root, but the resulting container may be missing some features if it is run as root.

[A33a]

Hi @GodloveD, something is wrong then... :( Because I was building from SingularityHub and DockerHub and it still was saying "No valid /bin/sh in container".

[GodloveD]

Hey @A33a! I'm sorry, but I'm having some trouble replicating this error. I spun up this Scientific-Linux VM using Vagrant and executed the following commands:

git clone https://github.com/singularityware/singularity.git
cd singularity/
./autogen.sh
./configure
sudo make
sudo make install
singularity build sjupyter.simg docker://a33a/sjupyter
singularity shell sjupyter.simg

Everything worked just fine and I got a shell within the container. Is there something different I should attempt to replicate? Thanks!

[A33a]

Hi @GodloveD, maybe you have already fixed it in the master branch actually. Because I was using the release 2.4 version. Or if it is not a case, can you tell me how to obtain verbose build output for analyzing the problem? I have tried singularity -vvv, but nothing suspicious there. Thank you.

[bauerm97]

@A33a You can obtain debug output from Singularity by using the -d flag:

singularity -d ...

Go ahead and run that and see what it shows us. Maybe it's already fixed in master and we just don't realize it

[A33a]

Hi @bauerm97, thanks. I tried it and here is the output, everything looks fine by me, maybe you will find something:

frontend:60 14:51:25 ~/singularity > singularity -d build sjupyter2.simg docker://a33a/sjupyter
Enabling debugging
Ending argument loop
Singularity version: 2.4-dist
Exec'ing: /cm/shared/apps/singularity/libexec/singularity/cli/build.exec
Building into existing container: sjupyter2.simg
VERBOSE2 SINGULARITY_COMMAND_ASIS found as False
VERBOSE2 SINGULARITY_ROOTFS found as /tmp/.singularity-build.w70Bq5
VERBOSE2 SINGULARITY_METADATA_FOLDER found as /tmp/.singularity-build.w70Bq5/.singularity.d
VERBOSE2 SINGULARITY_FIX_PERMS found as False
VERBOSE2 SINGULARITY_COLORIZE not defined (None)
VERBOSE2 SINGULARITY_DISABLE_CACHE found as False
VERBOSE2 SINGULARITY_CACHEDIR found as /usr/users/akhuziy/.singularity
VERBOSE2 REGISTRY not defined (None)
VERBOSE2 NAMESPACE not defined (None)
VERBOSE2 SINGULARITY_DOCKER_ARCHITECTURE found as amd64
VERBOSE2 SINGULARITY_DOCKER_OS found as linux
VERBOSE2 SINGULARITY_ENVIRONMENT found as /tmp/.singularity-build.w70Bq5/.singularity.d/env/90-environment.sh
VERBOSE2 SINGULARITY_RUNSCRIPT found as /tmp/.singularity-build.w70Bq5/singularity
VERBOSE2 SINGULARITY_TESTFILE found as /tmp/.singularity-build.w70Bq5/.singularity.d/test
VERBOSE2 SINGULARITY_DEFFILE found as /tmp/.singularity-build.w70Bq5/.singularity.d/Singularity
VERBOSE2 SINGULARITY_HELPFILE found as /tmp/.singularity-build.w70Bq5/.singularity.d/runscript.help
VERBOSE2 SINGULARITY_ENVBASE found as /tmp/.singularity-build.w70Bq5/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as /tmp/.singularity-build.w70Bq5/.singularity.d/labels.json
VERBOSE2 SINGULARITY_INCLUDECMD found as False
VERBOSE2 SINGULARITY_NOHTTPS found as False
VERBOSE2 SINGULARITY_PULLFOLDER found as /home/uni05/akhuziy/singularity
VERBOSE2 SHUB_NAMEBYHASH not defined (None)
VERBOSE2 SHUB_NAMEBYCOMMIT not defined (None)
VERBOSE2 SHUB_CONTAINERNAME not defined (None)
VERBOSE2 SINGULARITY_CONTENTS found as /tmp/.singularity-layers.snWIMC2L
VERBOSE2 SINGULARITY_PYTHREADS found as 9
VERBOSE2 SINGULARITY_CONTAINER found as docker://a33a/sjupyter
DEBUG Found uri docker://
DEBUG 
*** STARTING DOCKER IMPORT PYTHON  ****
DEBUG Docker layers and metadata will be written to: /tmp/.singularity-layers.snWIMC2L
VERBOSE2 SINGULARITY_DOCKER_USERNAME not defined (None)
VERBOSE2 SINGULARITY_DOCKER_PASSWORD found
DEBUG Starting Docker IMPORT, includes env, runscript, and metadata.
VERBOSE Docker image: a33a/sjupyter
VERBOSE2 Specified Docker ENTRYPOINT as %runscript.
DEBUG Headers found: Content-Type,Accept
VERBOSE Registry: index.docker.io
VERBOSE Namespace: a33a
VERBOSE Repo Name: sjupyter
VERBOSE Repo Tag: latest
VERBOSE Version: None
VERBOSE Obtaining tags: https://index.docker.io/v2/a33a/sjupyter/tags/list
DEBUG GET https://index.docker.io/v2/a33a/sjupyter/tags/list
DEBUG Http Error with code 401
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:a33a/sjupyter:pull
DEBUG Headers found: Content-Type,Authorization,Accept
VERBOSE3 Response on obtaining token is None.
Docker image path: index.docker.io/a33a/sjupyter:latest
DEBUG Updating manifests.
DEBUG MANIFEST (Primary): not found, making initial call.
VERBOSE Obtaining manifest: https://index.docker.io/v2/a33a/sjupyter/manifests/latest
DEBUG GET https://index.docker.io/v2/a33a/sjupyter/manifests/latest
DEBUG MANIFEST (Metadata): not found, making initial call.
VERBOSE Obtaining manifest: https://index.docker.io/v2/a33a/sjupyter/manifests/latest
DEBUG GET https://index.docker.io/v2/a33a/sjupyter/manifests/latest
DEBUG Image manifest version 2.2 found.
DEBUG Adding digest sha256:ae79f251470513c2a0ec750117a81f2d58a50727901ca416efecf297b8a03913
DEBUG Adding digest sha256:5ad56d5fc14905886c560200ab69f905b5c5287eaf12f8f761a7ab54f7a61c1b
DEBUG Adding digest sha256:170e558760e8b2e484a022b7d7272cf284fc4e1936ba7a0a671fc586440ad272
DEBUG Adding digest sha256:395460e233f5bdcd910d618a3b615e0d881e09ad27d58f3065eef53ecae6a808
DEBUG Adding digest sha256:6f01dc62e444044e3ce494269837ef0aedb80fef69c679416137f17812d2eb9c
DEBUG Adding digest sha256:7dc54d6b399796df373f532d3447686d6cfd3f78be43a19c2f5d6c5e07fe654a
DEBUG Adding digest sha256:b6103a8951ef4c022c677baed9e15481e1948f374e16f32fb4a7dc7adfcc6241
DEBUG Adding digest sha256:c60ae09059e277855c1205e330ddb99838d5b3421adb717e04bba1a66d537407
DEBUG Adding digest sha256:e75c1be0719a8667db2f5e028b99a4a97d3bceaa43f3986fc39d851d0281bb3e
DEBUG Adding digest sha256:9415bec0133cb01dcabc12f44a9a672116197991d6838a948f315b100b052763
DEBUG Adding digest sha256:c2e43f6807a1452bde0c333a4935ea47dce0d7a24bb36921a01418ca009f764f
DEBUG Adding digest sha256:6d253f32fc2d3dbadb386227e7dc42eb874d571313684bbb1c95802e99da807d
Cache folder set to /home/uni05/akhuziy/.singularity/docker
DEBUG Using 9 workers for multiprocess.
VERBOSE3 Found Docker command (Entrypoint) None
VERBOSE3 Found Docker command (Cmd) [u'/bin/sh', u'-c', u'/bin/bash']
VERBOSE3 Adding Docker CMD as Singularity runscript...
VERBOSE3 Found Docker command (Env) [u'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', u'XDG_RUNTIME_DIR=']
VERBOSE3 Found Docker container environment!
VERBOSE3 Adding Docker environment to metadata tar
DEBUG Found template for tarinfo
VERBOSE3 Found Docker command (Labels) None
VERBOSE3 Adding Docker runscript to metadata tar
DEBUG Found template for tarinfo
DEBUG metadata file /home/uni05/akhuziy/.singularity/metadata/sha256:bc54208ccc13436455745dc134d0cee8a5d7477e7475bde574ee9db65e61d674.tar.gz already exists will over-write.
VERBOSE2 Tar file with Docker env and labels: /home/uni05/akhuziy/.singularity/metadata/sha256:bc54208ccc13436455745dc134d0cee8a5d7477e7475bde574ee9db65e61d674.tar.gz
VERBOSE3 Writing Docker layers files to /tmp/.singularity-layers.snWIMC2L
VERBOSE2 Writing file /tmp/.singularity-layers.snWIMC2L with mode w.
VERBOSE2 Writing file /tmp/.singularity-layers.snWIMC2L with mode a.
DEBUG *** FINISHING DOCKER IMPORT PYTHON PORTION ****
Importing: base Singularity environment
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:ae79f251470513c2a0ec750117a81f2d58a50727901ca416efecf297b8a03913.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:5ad56d5fc14905886c560200ab69f905b5c5287eaf12f8f761a7ab54f7a61c1b.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:170e558760e8b2e484a022b7d7272cf284fc4e1936ba7a0a671fc586440ad272.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:395460e233f5bdcd910d618a3b615e0d881e09ad27d58f3065eef53ecae6a808.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:6f01dc62e444044e3ce494269837ef0aedb80fef69c679416137f17812d2eb9c.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:7dc54d6b399796df373f532d3447686d6cfd3f78be43a19c2f5d6c5e07fe654a.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:b6103a8951ef4c022c677baed9e15481e1948f374e16f32fb4a7dc7adfcc6241.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:c60ae09059e277855c1205e330ddb99838d5b3421adb717e04bba1a66d537407.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:e75c1be0719a8667db2f5e028b99a4a97d3bceaa43f3986fc39d851d0281bb3e.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:9415bec0133cb01dcabc12f44a9a672116197991d6838a948f315b100b052763.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:c2e43f6807a1452bde0c333a4935ea47dce0d7a24bb36921a01418ca009f764f.tar.gz
Importing: /home/uni05/akhuziy/.singularity/docker/sha256:6d253f32fc2d3dbadb386227e7dc42eb874d571313684bbb1c95802e99da807d.tar.gz
Importing: /home/uni05/akhuziy/.singularity/metadata/sha256:bc54208ccc13436455745dc134d0cee8a5d7477e7475bde574ee9db65e61d674.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: sjupyter2.simg
Cleaning up...
frontend:60 14:51:59 ~/singularity > singularity shell sjupyter2.simg 
ERROR  : No valid /bin/sh in container
ABORT  : Retval = 255

[GodloveD]

This line indicates that a container of the name sjupyter2.simg already exists.

Building into existing container: sjupyter2.simg

Can you either delete that container before building to start with a clean slate or use the --force option with build to completely overwrite?

[A33a]

HI @GodloveD, I deleted the image and builded it again. Same result: No valid /bin/sh in container. I think, it might be because of squashfs-tools, is there a way to test if it works properly with Singularity?

[vsoch]

Where do you have them (squashfs-tools) installed? Singularity will clear the path of "non root/system" locations, so if you have it in a custom spot, this could be a reason for failure.

[GodloveD]

@A33a if you build using the --sandbox option, you will remove squashfs from the equation. When you are finished creating the sandbox you should test that it has a /bin/sh, by trying to shell into it. If it does, you can test your squashfs installation with the following:

mksquashfs "/path/to/sandbox" "/path/to/output" -noappend

If you execute this command as nonroot it is normal to see some tar errors and should not be cause for concern.

[A33a]

Hi @vsoch, I've installed it in the custom location, but added the path to bin/singularity.in script.

@GodloveD it took some time to get a sanbox built :) but now I think we got more usefull information. After shelling the sandbox, it says:

frontend:74 17:47:22 ~/singularity > singularity shell ubuntu.simg
ERROR  : Failed to execv() /.singularity.d/actions/shell, continuing to /bin/sh: No such file or directory
ERROR  : Failed to execv() /bin/sh: No such file or directory
ABORT  : Retval = 255

Here is the /.singularity.d/actions/shell cannot be executed, but I can run it outside the container and it is there at ./ubuntu.simg/.singularity.d/actions/shell.

And this issue might be relevant to the issue #1088, because my instance is also installed not in a standart location. Thank you for the help, once again :)

[GodloveD]

1088 was just resolved. I wonder if this could be the same problem? Is there more than one installation of Singularity on this machine? Was the previous version completely uninstalled prior to installing the new one? Have you tried make clean (and praying to the open source gods, etc.)? 😺

[A33a]

@GodloveD I see... I don't quite remember if I deleted everything (I guess I did) but I can tell for sure that I didn't make clean. I will try it tomorrow, first without praying to open source gods and then praying and dancing with a tambourine ^_^

[GodloveD]

Hey @A33a! Just wanted to check in and see if that fixed your issue. Are we good now?

[A33a]

Hi @GodloveD, sorry for the late answer, I just tried it out. Deleted everything and run make clean, but unfortunately it didn't help :(

[GodloveD]

Hi @A33a! And sorry for my late reply. 😺

I'm almost totally convinced that there has to be something wrong with your installation. Can you run the following commands please to verify that your system can run Singularity 2.4 without an issue.

SINGDIR=`mktemp -d`
cd $SINGDIR
wget https://github.com/singularityware/singularity/releases/download/2.4/singularity-2.4.tar.gz
tar xvf singularity-2.4.tar.gz
cd singularity-2.4/
./configure --prefix=$SINGDIR
make
sudo make install
$SINGDIR/bin/singularity build sjupyter.simg docker://a33a/sjupyter
$SINGDIR/bin/singularity exec sjupyter.simg echo 'It works!'

If this works, then there has to be something wrong with the system installation.

[vsoch]

hey @A33a any update on this issue?

[A33a]

Hi @vsoch, Hi @GodloveD. Sorry for the late reply, but I just left it as it is right now. I also beleive that there must be something not compatible in Singularity with my current installation.

Regarding the reinstallation. I already did reinstall singularity as shown in the code above and the result is the same. However, there was one empty folder that I could not delete before reinstalling: SINGULARITY/var/singularity/mnt/session/. But I guess the problem is not caused by it. I don't know how else I can easily debug my installation, thus I will just leave it as it is, because I couldn't reproduce the error on other systems.

But anyways, thank you for your efforts and help :)

[yarikoptic]

I thought I saw this issue before and found a workaround listed, but now forgot...
ATM using 2.4.1 singularity, singularity pull docker://repronim/simple_workflow:latest

$> singularity --version
2.4.1-dist

$> apt-cache policy squashfs-tools
squashfs-tools:     
  Installed: 1:4.3-3+deb9u1
  Candidate: 1:4.3-3+deb9u1
  Version table:
 *** 1:4.3-3+deb9u1 100
        100 http://debian.csail.mit.edu/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
$> ./simple_workflow-latest.simg                                   
ERROR  : No valid /bin/sh in container
ABORT  : Retval = 255

$> singularity mount ./simple_workflow-latest.simg  
simple_workflow-latest.simg is mounted at: /var/lib/singularity/mnt/final

Singularity> ls /bin/sh
/bin/sh
Singularity> /bin/sh
Singularity> 
Singularity> 

edit 1: works fine for a basic singularity pull shub://vsoch/hello-world
edit 2: forgot actually to demo the issue:

$> singularity  exec ./simple_workflow-latest.simg /bin/bash     
ERROR  : No valid /bin/sh in container
ABORT  : Retval = 255

[gnguy]

We ran into this issue at our organization. For our setup using Singularity 2.4.2, if a user's umask was set to 002, the singularity build would work perfectly, but if their umask was set to 007, the built image would fail when trying to singularity exec with the No valid /bin/sh in container error.

Any idea on the Singularity side what might trigger this?

[yarikoptic]

oh hoh... I thought that umask related issues were addressed awhile back after my initial experiences. That might be yet another, thanks for sharing @gnguy -- great point , I will try (mine is 077)

[A33a]

Many thanks to @gnguy and @yarikoptic.

@vsoch, @GodloveD: Indeed, my problem is also because of umask (in our system we keep it 077). If I set it to 002, build works without any problems.

And one more question. Is it designed to work this way?

[vsoch]

If the minimum requirements for a container are /bin/sh and then umask 077 requires an explicit chmod u+x to make it work (reading here --> https://askubuntu.com/questions/186747/why-umask-077-does-not-allow-user-to-execute-files-directories) that would be the behavior that I would expect. If the umask is set in the user's profile you could try fiddling with not mounting home (or mounting somewhere else) and containing the environment. You could also try explicitly making /bin/sh (or some pointer to it) executable before. I haven't run into this, so that's the extent of what I can suggest! Definitely if changing it to 002 fixes the issue, it's likely coming down to permissions with executing /bin/sh and all workarounds lead to the road of fixing that.

[gnguy]

@vsoch Thank you for looking into this. One point of clarification -- the umask setting only matters when building the image. For example, if someone with a umask of 007 builds an image and another user with a umask of 002 tries to run the image, it will error out with /bin/sh not found. However, if a user with a umask of 002 builds an image, someone with a umask of 007 will be able to run it without the error.

Is there anything in the Singularity build process that runs as a different user than the one who invokes the singularity build command? I wonder if, within the build process there files that are being saved under the user account with a user-set umask of 007 and which prevent them from being appropriately imported by a subsequent step in the build process running under a different user. The restrictions of umask = 007 would prevent "Other" users (not the user who created the file or within the primary usergroup) from being able to read/write/execute.

[GodloveD]

@gnguy thanks for the insight. Now I can replicate the error.

@vsoch I think it is a little more complicated because it depends on both the permission and the ownership.

For some reason, files in a squashfs image end up with root ownership even if the squashfs image is built as a non-privileged user. If your umask is 077 means that /bin/sh (or dash in the case of ubuntu) ends up with the following permissions and ownership:

-rwx------ 1 root root  154072 Feb 17  2016 dash

So that is obviously not going to work for a measly user.

But I need to dig a bit deeper because a sandbox doesn't work the same way. When I build a sandbox with umask 077, /bin/dash ends up being owned by me, not root. So I can still execute the container even though it has permissions == 700.

I don't know why we made the decision for the ownership to differ in those two cases and I will need to dig/think about whether that is a reasonable thing to do. But in the meantime, @A33a is it a sensible workaround to just set your umask to something less restrictive when creating Singularity containers?

[vsoch]

Interesting! Let us know what you find @GodloveD.

[A33a]

@GodloveD Thank you. Sure, now we finally know what is happening at least :)

[yarikoptic]

note that the issue is not exactly new, and even some of the "(ab)users" are the same ;-) : https://github.com/singularityware/singularity/issues/139 , https://github.com/singularityware/singularity/issues/142

[vsoch]

lol @yarikoptic, this is why I appreciate your comments ;)

[GodloveD]

[GodloveD]

OK. We are expressly making sure that everything in the image is owned by root.
https://github.com/singularityware/singularity/blob/master/libexec/cli/build.exec#L362-L366
But this happens when the temporary build directory is compressed and turned into a squashfs image. Now we need to have a discussion about why we are doing this and whether it is causing more trouble than it is alleviating. ping @gmkurtzer

[prioux]

We just spent four hours trying to debug why an image wouldn't be built properly in production while it worked fine in development, interactively... all because our umask in production is 077. Thanks to everyone for having identified this. Thanks in particular to @gnguy !

[vsoch]

hiphip hooray, @gnguy! And that username! Sir, I sincerely appreciate your swagger :sunglasses:

[pescobar]

We have also hit this problem with one of our users with umask 007 using singularity 2.4.0

[ponomarevsy]

Not sure if you solved this issue, but I have noticed that switching back to "create" and "bootstrap" from "build" gets rid of the "No valid /bin/sh in container" error in v2.4.2 of singularity.

[dtrudg]

Hi @pescobar @ponomarevsy @A33a - are you able to check the behaviour under the release-2.4 branch, which contains a lot of changes to the docker extraction process that fix various issues? The changes will become part of the next stable 2.4.3 release. Thanks!

[dtrudg]

Closing as a umask fix was included in 2.5.0. Please feel to reopen if you experience further issues.

[zhangwewenwen]

I meet this problem too.I am using a supercomputer to run a singularity container, and I can't usr root, nor can I install squashfs. How can I sove this problem? It drives me crazy already! I have already tried to build or convert it into a sandbox ,which all failed.

[denisovlev]

I have the same issue. I am building the image on one machine and try to execute it on another.

Version (same on both machines):

$ singularity --version
2.5.1-dist

I build it on one machine and on the same machine shell works just fine:

$ sudo singularity build --writable horovod.img docker://uber/horovod:0.13.11-tf1.10.0-torch0.4.0-py2.7

$ singularity shell --nv ./horovod.img

When I upload the built image to another machine and try to execute shell I get:

$ singularity --debug shell --nv ./horovod.img
INFO    [U=2353,P=103091]  action_shell()                            Singularity: Invoking an interactive shell within container...

DEBUG   [U=2353,P=103091]  action_shell()                            Exec'ing /.singularity.d/actions/shell
ERROR   [U=2353,P=103091]  action_shell()                            Failed to execv() /.singularity.d/actions/shell, continuing to /bin/sh: Exec format error
VERBOSE [U=2353,P=103091]  action_shell()                            Invoking the container's /bin/sh
DEBUG   [U=2353,P=103091]  action_shell()                            Exec'ing /bin/sh
ERROR   [U=2353,P=103091]  action_shell()                            Failed to execv() /bin/sh: Exec format error
ABORT   [U=2353,P=103091]  action_shell()                            Retval = 255

I cannot build the image on the same machine I run it because it does not have internet and I do not have sudo rights there.

[dtrudg]

@denisovlev - what type of machine, and what OS is running on the build machine vs the execution machine? The Exec format error is slightly different than there being no /bin/sh found in the container.

[denisovlev]

@dctrud

Build:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:    16.04
Codename:   xenial

Execution (HPC cluster):

$ lsb_release -a
LSB Version:    :core-4.1-noarch:core-4.1-ppc64le
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.4 (Maipo)
Release:    7.4
Codename:   Maipo

Both are real physical machines

[robertlugg]

* Please ignore. Left here in hopes that it might help someone else *

(update: I believe I resolved the missing /bin/sh problem. I should "start fresh" to confirm that I can do it again, but that's fun for another day. I believe my problem was that I didn't run sh ./autogen.sh first when I built singularity.)

VERSION:
2.6.0-dist

SETUP:
VirtualBox CentOS7

Machine configuration
Linux osboxes 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
CONFIG_USER_NS=y

which singularity
/usr/local/bin/singularity

singularity selftest
 + sh -c test -f /usr/local/etc/singularity/singularity.conf                           (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/action-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/mount-suid                               (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/start-suid                               (retval=0) OK

STEPS to REPEAT:
sudo /usr/local/bin/singularity pull --name hello-world.simg shub://vsoch/hello-world
singularity shell hello-world.simg

As above, 'mount' does appear to work.

I did build from source. I did NOT do the following.

./autogen.sh
./configure --prefix=/usr/local

instead I only ran .configure without args while su - in root.

* Note also this error *

# singularity apps hello-world.simg 
ERROR: The Singularity metadata directory does not exist in image

[soichih]

I was having this problem and the problem went away when I did rm -rf ~/.singularity

[kwohlfahrt]

I had a similar problem with Singularity 2.6 - the issue was the .tar exported from Docker had the owner of /bin/sh as 0/0. Setting the user & group names (so that the owner was root/root) fixed the issue.