Singularity: Error with config subsystem with apt install

Created on 9 Sep 2018  路  44Comments  路  Source: hpcng/singularity

When installed with

See the conversation here: https://github.com/singularityhub/singularity-cli/pull/53#issuecomment-419745928. When we are pulling singularity images with both of:

sudo apt install singularity-container

(2.4.2) and then

sudo apt upgrade

(2.5.2)

there is an error about the config subsystem (see below) although the command seems to run ok. Note that this isn't reproducible with the install from source (I cannot reproduce it).

Version of Singularity:

2.4.2 and 2.5.2

Expected behavior

singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: /tmp/tmpuqycevfa/d13898a106f9678c695ab479335e285c634d3686.simg

Actual behavior

singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: /tmp/tmpuqycevfa/d13898a106f9678c695ab479335e285c634d3686.simg
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255

/home/matt/temp/d13898a106f9678c695ab479335e285c634d3686.simg

Steps to reproduce behavior

See the issue linked for very helpful description from @mwiens91. I wasn't able to reproduce the error with an install from source, so I suspect this is an issue with the packaged version. I'm happy to update the python client with the pull by commit and hash because overall it works, but I'd like to get some insight to this issue first. Thanks!

picture vsoch

Most helpful comment

RTFM helps at times ;) There is a dedicated env variable to deal with such cases, so here is the "fix": https://salsa.debian.org/hpc-team/singularity-container/commit/c049302689fcb830ef4465b5b70c9ce6eed84826
building packages now, will upload 2.5.2-3 to (Neuro)Debians probably early tomorrow after minor re-verification

picture yarikoptic on 12 Sep 2018
👍2 🎉1

All 44 comments

Hi @vsoch - this appears to be an issue where an upgrade to the latest neurodebian package doesn't work 100% if the original install was the old 2.4.x ubuntu package?

This would likely best be reported to the neurodebian maintainers via http://neuro.debian.net/pkgs/singularity-container.html as the various debian/ubuntu/neurodebian/EPEL packagings are maintained outside the scope of the main source repository here. Since it is not reproducible from source installs it's likely a packaging issue (or non-issue if it is not expected the neurodebian packages can upgrade the ubuntu package trivially).

Cheers!

dtrudg picture dtrudg on 9 Sep 2018

@dctrud, it's definitely not an upgrade issue. Installing fresh from Debian Neuroscience still results in this error.

mwiens91 picture mwiens91 on 9 Sep 2018

@mwiens91 - okay - please could you report to the maintainers of the neurodebian package, as it's not reproducible from a source install. If there's some kind of issue that they find in source then it can be addressed, but the particular neurodebian packaging is outside of the scope of the source development here. Thanks.

dtrudg picture dtrudg on 9 Sep 2018

ping @yarikoptic be this you? 馃

vsoch picture vsoch on 9 Sep 2018

And thanks @mwiens91 for the nice before and after shots! I think @yarikoptic and the neurodebian maintainers will be able to help for sure!

vsoch picture vsoch on 10 Sep 2018

Sorry, brain is off - what is the actual reason, anyone knows? Some stale config?

yarikoptic picture yarikoptic on 10 Sep 2018

@yarikoptic I'm not sure, but here is some searching that might help:

  • question on list of same issue, also from neurodebian, but a very old version (2017). It looks like it wasn't answered so there isn't follow up.
  • this might be helpful something about the --prefix or setuid? It looks like that conversation got some steam but wasn't resolved. Points to...
  • this one what is the ownership of singularity.conf in the distribution?
  • and this one again permissions issues.

Let me know if you need further digging! It might be a good start to talk about these points with respect to the package.

vsoch picture vsoch on 10 Sep 2018

well, I have tried a few cycles of purge, install 2.4 (or 2.4.2-3) and then upgrade to 2.5.2-2~nd+1 without success replicating it.
this one suggests that the reason might be incorrect ownership of the /etc/singularity/singularity.conf. In my cases it was always owned by root:

$> grep /etc/singularity/singularity.conf singularity.list.2.*
singularity.list.2.4:  3702727      8 -rw-r--r--   1 root     root         6273 Oct 12  2017 /etc/singularity/singularity.conf
singularity.list.2.4.2-3:  3702731      8 -rw-r--r--   1 root     root         6273 Feb  4  2018 /etc/singularity/singularity.conf
singularity.list.2.5.2:  3702734      8 -rw-r--r--   1 root     root         6717 Aug  1 14:39 /etc/singularity/singularity.conf

NB use sudo find /etc /var -iname singular\* -ls | grep -v docker/ >| singularity.list to generate similar lists.

only now spotted above that it was stated to not be an upgrade issue ... (although not sure if full purge was done between trials), but first we need to figure out permissions I guess (share your listings from above command?), if @singularityware/singularity-maintainers have no better idea for the cause of that error message.

yarikoptic picture yarikoptic on 10 Sep 2018

I can't think of anything else immediately.

@mwiens91 - could you supply the output running the pull with the debug flag singularity -d pull .... and also give the output of an ls -lah /etc/singularity/*? Thanks

dtrudg picture dtrudg on 11 Sep 2018

Sure:

$ ls -lah /etc/singularity/*
-rw-r--r-- 1 root root   87 Feb 17  2018 /etc/singularity/default-nsswitch.conf
-rw-r--r-- 1 root root 1.5K Aug  1 12:12 /etc/singularity/init
-rw-r--r-- 1 root root  866 Feb 17  2018 /etc/singularity/nvliblist.conf
-rw-r--r-- 1 root root 6.6K Aug  1 12:12 /etc/singularity/singularity.conf

$ singularity -d pull --hash shub://mwiens91/shub-debug
Enabling debugging
Ending argument loop
Singularity version: 2.5.2-dist
Exec'ing: /usr/lib/x86_64-linux-gnu/singularity/cli/pull.exec
VERBOSE2 SINGULARITY_COMMAND_ASIS found as False
VERBOSE2 SINGULARITY_ROOTFS not defined (None)
VERBOSE2 SINGULARITY_METADATA_FOLDER found as None/.singularity.d
VERBOSE2 SINGULARITY_FIX_PERMS found as False
VERBOSE2 SINGULARITY_COLORIZE not defined (None)
VERBOSE2 SINGULARITY_DISABLE_CACHE found as False
VERBOSE2 SINGULARITY_CACHEDIR found as /home/matt/.singularity
VERBOSE2 REGISTRY not defined (None)
VERBOSE2 NAMESPACE not defined (None)
VERBOSE2 SINGULARITY_DOCKER_ARCHITECTURE found as amd64
VERBOSE2 SINGULARITY_DOCKER_OS found as linux
VERBOSE2 SINGULARITY_ENVIRONMENT found as None/.singularity.d/env/90-environment.sh
VERBOSE2 SINGULARITY_RUNSCRIPT found as None/singularity
VERBOSE2 SINGULARITY_TESTFILE found as None/.singularity.d/test
VERBOSE2 SINGULARITY_DEFFILE found as None/.singularity.d/Singularity
VERBOSE2 SINGULARITY_HELPFILE found as None/.singularity.d/runscript.help
VERBOSE2 SINGULARITY_ENVBASE found as None/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as None/.singularity.d/labels.json
VERBOSE2 SINGULARITY_INCLUDECMD found as False
VERBOSE2 SINGULARITY_NOHTTPS found as False
VERBOSE2 SINGULARITY_PULLFOLDER found as .
VERBOSE2 SHUB_NAMEBYHASH found as yes
VERBOSE2 SHUB_NAMEBYCOMMIT not defined (None)
VERBOSE2 SHUB_CONTAINERNAME not defined (None)
VERBOSE2 SINGULARITY_CONTENTS found as /tmp/.singularity-layerfile.U3OeXK
VERBOSE2 SINGULARITY_PYTHREADS found as 9
DEBUG 
*** STARTING SINGULARITY PYTHON PULL ****
VERBOSE2 SINGULARITY_CONTAINER found as shub://mwiens91/shub-debug
VERBOSE2 SINGULARITY_PULLFOLDER found as .
DEBUG Found uri shub://
DEBUG Headers found: Content-Type,Accept
DEBUG GET https://www.singularity-hub.org/api/container/mwiens91/shub-debug:latest
DEBUG Pull folder set to /home/matt/temp
DEBUG Pulling to /home/matt/temp/07299790f4a52fdfeb6e95aa6fa97eaf
DEBUG GET (stream) https://www.googleapis.com/download/storage/v1/b/singularityhub/o/singularityhub%2Fgithub.com%2Fmwiens91%2Fshub-debug%2Fd13898a106f9678c695ab479335e285c634d3686%2F07299790f4a52fdfeb6e95aa6fa97eaf%2F07299790f4a52fdfeb6e95aa6fa97eaf.simg?generation=1534267440148151&alt=media
Progress |===================================| 100.0% 
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255
DEBUG Found None image
DEBUG Writing Singularity Hub image path to /tmp/.singularity-layerfile.U3OeXK
VERBOSE2 Writing file /tmp/.singularity-layerfile.U3OeXK with mode w.
Done. Container is at: /home/matt/temp/07299790f4a52fdfeb6e95aa6fa97eaf.simg
mwiens91 picture mwiens91 on 11 Sep 2018

What about all of those None - are they problematic?

ERBOSE2 SINGULARITY_ENVIRONMENT found as None/.singularity.d/env/90-environment.sh
VERBOSE2 SINGULARITY_RUNSCRIPT found as None/singularity
VERBOSE2 SINGULARITY_TESTFILE found as None/.singularity.d/test
VERBOSE2 SINGULARITY_DEFFILE found as None/.singularity.d/Singularity
VERBOSE2 SINGULARITY_HELPFILE found as None/.singularity.d/runscript.help
VERBOSE2 SINGULARITY_ENVBASE found as None/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as None/.singularity.d/labels.json
vsoch picture vsoch on 11 Sep 2018

I'm guessing that's a sub for SINGULARITY_ROOTFS and just an artifact that is still there from previous versions?

vsoch picture vsoch on 11 Sep 2018

Debug output here is exactly the same (except the ERROR line) as my installs which do not raise an error. I'm afraid I'm out of ideas except - @mwiens91 have you tried an apt purge singularity-container and then a reinstall? When you installed fresh from neurodebian and had the same error, had you done a purge, or was it installing on a fresh VM/machine where Singularity had not been at all before?

dtrudg picture dtrudg on 11 Sep 2018

That's a good idea @dctrud . @mwiens91 if you are using travis (or circle or similar) also make sure to rebuild and disable caching.

vsoch picture vsoch on 11 Sep 2018

or was it installing on a fresh VM/machine where Singularity had not been at all before?

This. Both Canonical's (https://launchpad.net/ubuntu/+source/singularity-container) and NeuroDebian's packages caused the same error on fresh VM installs.

if you are using travis (or circle or similar) also make sure to rebuild and disable caching.

Caching was never enabled here.

mwiens91 picture mwiens91 on 11 Sep 2018

How does one reproduce how the package is installed? I just downloaded the actual archive from here:
https://launchpad.net/ubuntu/+source/singularity-container/2.5.2-2
(the tar.gz) and unguzzed it, installed per standard protocol, and it works ok:

$ singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: /home/vanessa/Downloads/singularity-2.5.2/d13898a106f9678c695ab479335e285c634d3686.simg

@mwiens91 maybe you can try the same (with that file) on your travis to determine if you get the bug then? If you don't, then it's something about how it's installed as a package. @yarikoptic do you know how this works?

vsoch picture vsoch on 11 Sep 2018

@vsoch apt install singularity-container to install the beast

@mwiens91

on fresh VM installs

not sure how but may be that is related -- what is the base system (I think that was never disclosed), e.g. what is the exact version of the package from NeuroDebian (~nd suffix should point us to the debian/ubuntu release)

yarikoptic picture yarikoptic on 11 Sep 2018

and

if you are using travis (or circle or similar) also make sure to rebuild and disable caching.

Caching was never enabled here.

so what kind of "VM" are we talking here about?

yarikoptic picture yarikoptic on 11 Sep 2018

but what exactly is that command doing? Can we reproduce that without apt? I suggested using the tar guz from the repo and seeing if the bug triggered from an install with just that. If it doesn鈥檛 we can suspect it鈥檚 whatever happens with apt.

vsoch picture vsoch on 11 Sep 2018

so what kind of "VM" are we talking here about?

Standard Travis Ubuntu 14.04 image. If you want specific details on the VM I'm sure they're somewhere around their site. To point to a specific build, see

https://travis-ci.com/mwiens91/saltant/jobs/144620574#L12

And I've also tested this on a Ubuntu 16.04 VM; and my main computer which is running Ubuntu 18.04. Same issue across the board when installing Singularity with apt.

mwiens91 picture mwiens91 on 11 Sep 2018

Okay so here is more along the lines of what I had in mind to debug. Note that the docker container isn't the best way to debug, but minimally it let's us look at the files under the hood for the apt install. Here is an example of starting a container with neurodebian (and @mwiens91 we would want to do this with the same container being used by travis, if you know?)

docker run -it --rm neurodebian

Install singularity

apt-get update && apt-get install -y wget
apt-get update && apt-get install -y singularity-container

Here is the whole install, in case something useful here?

Hit:1 http://neuro.debian.net/debian stretch InRelease
Hit:2 http://security.debian.org/debian-security stretch/updates InRelease
Hit:3 http://neuro.debian.net/debian data InRelease                      
Ign:4 http://cdn-fastly.deb.debian.org/debian stretch InRelease                              
Hit:5 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease
Hit:6 http://cdn-fastly.deb.debian.org/debian stretch Release
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  bzip2 cron exim4 exim4-base exim4-config exim4-daemon-light file guile-2.0-libs krb5-locales libarchive13 libexpat1 libfribidi0 libgc1c2 libgpm2 libgsasl7
  libgssapi-krb5-2 libicu57 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libkyotocabinet16v5 libltdl7 liblzo2-2 libmagic-mgc libmagic1 libmailutils5
  libmariadbclient18 libncurses5 libntlm0 libpython-stdlib libpython2.7 libpython2.7-minimal libpython2.7-stdlib libwrap0 libxml2 mailutils mailutils-common mime-support
  mysql-common netbase neurodebian-popularity-contest popularity-contest psmisc python python-minimal python2.7 python2.7-minimal sgml-base squashfs-tools tcpd xml-core
  xz-utils
Suggested packages:
  bzip2-doc anacron logrotate checksecurity eximon4 exim4-doc-html | exim4-doc-info spf-tools-perl swaks lrzip gpm krb5-doc krb5-user mailutils-mh mailutils-doc python-doc
  python-tk python2.7-doc binutils binfmt-support sgml-base-doc debhelper
The following NEW packages will be installed:
  bzip2 cron exim4 exim4-base exim4-config exim4-daemon-light file guile-2.0-libs krb5-locales libarchive13 libexpat1 libfribidi0 libgc1c2 libgpm2 libgsasl7
  libgssapi-krb5-2 libicu57 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libkyotocabinet16v5 libltdl7 liblzo2-2 libmagic-mgc libmagic1 libmailutils5
  libmariadbclient18 libncurses5 libntlm0 libpython-stdlib libpython2.7 libpython2.7-minimal libpython2.7-stdlib libwrap0 libxml2 mailutils mailutils-common mime-support
  mysql-common netbase neurodebian-popularity-contest popularity-contest psmisc python python-minimal python2.7 python2.7-minimal sgml-base singularity-container
  squashfs-tools tcpd xml-core xz-utils
0 upgraded, 54 newly installed, 0 to remove and 0 not upgraded.
Need to get 25.1 MB of archives.
After this operation, 95.3 MB of additional disk space will be used.
Get:1 http://neuro.debian.net/debian stretch/main amd64 singularity-container amd64 2.5.2-2~nd90+1 [344 kB]
Get:2 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 cron amd64 3.0pl1-128+deb9u1 [95.4 kB]
Get:3 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libpython2.7-minimal amd64 2.7.13-2+deb9u2 [389 kB]
Get:4 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 python2.7-minimal amd64 2.7.13-2+deb9u2 [1382 kB]
Get:5 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 python-minimal amd64 2.7.13-2 [40.5 kB]     
Get:6 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 mime-support all 3.60 [36.7 kB]           
Get:7 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libexpat1 amd64 2.2.0-2+deb9u1 [83.4 kB]
Get:8 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libpython2.7-stdlib amd64 2.7.13-2+deb9u2 [1896 kB]
Get:9 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 python2.7 amd64 2.7.13-2+deb9u2 [285 kB]       
Get:10 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libpython-stdlib amd64 2.7.13-2 [20.0 kB]
Get:11 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 python amd64 2.7.13-2 [154 kB]                
Get:12 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 popularity-contest all 1.64 [70.6 kB]
Get:13 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 sgml-base all 1.29 [14.8 kB]                  
Get:14 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 netbase all 5.4 [19.1 kB]              
Get:15 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 bzip2 amd64 1.0.6-8.1 [47.5 kB]      
Get:16 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libmagic-mgc amd64 1:5.30-1+deb9u2 [222 kB]
Get:17 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libmagic1 amd64 1:5.30-1+deb9u2 [111 kB]
Get:18 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 file amd64 1:5.30-1+deb9u2 [64.1 kB]    
Get:19 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 krb5-locales all 1.15-1+deb9u1 [93.8 kB]
Get:20 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libgc1c2 amd64 1:7.4.2-8 [208 kB]        
Get:21 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libgpm2 amd64 1.20.4-6.2+b1 [34.2 kB]
Get:22 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libkeyutils1 amd64 1.5.9-9 [12.4 kB] 
Get:23 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libkrb5support0 amd64 1.15-1+deb9u1 [61.9 kB]
Get:24 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libk5crypto3 amd64 1.15-1+deb9u1 [119 kB]    
Get:25 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libkrb5-3 amd64 1.15-1+deb9u1 [311 kB]
Get:26 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libgssapi-krb5-2 amd64 1.15-1+deb9u1 [155 kB]
Get:27 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libwrap0 amd64 7.6.q-26 [58.2 kB]
Get:28 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libicu57 amd64 57.1-6+deb9u2 [7700 kB]
Get:29 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libxml2 amd64 2.9.4+dfsg1-2.2+deb9u2 [920 kB]
Get:30 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 xz-utils amd64 5.2.2-1.2+b1 [266 kB]
Get:31 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 exim4-config all 4.89-2+deb9u3 [377 kB]
Get:32 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 exim4-base amd64 4.89-2+deb9u3 [1093 kB]
Get:33 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 exim4-daemon-light amd64 4.89-2+deb9u3 [542 kB]
Get:34 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 exim4 all 4.89-2+deb9u3 [7852 B]
Get:35 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libltdl7 amd64 2.4.6-2 [389 kB]
Get:36 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libncurses5 amd64 6.0+20161126-1+deb9u2 [93.4 kB]
Get:37 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 guile-2.0-libs amd64 2.0.13+1-4 [2232 kB]
Get:38 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 liblzo2-2 amd64 2.08-1.2+b2 [55.0 kB]
Get:39 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libarchive13 amd64 3.2.2-2 [294 kB]
Get:40 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libfribidi0 amd64 0.19.7-1+b1 [45.7 kB]
Get:41 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libntlm0 amd64 1.4-8 [20.9 kB]
Get:42 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libgsasl7 amd64 1.8.0-8+b2 [207 kB]
Get:43 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libkyotocabinet16v5 amd64 1.2.76-4.2+b1 [298 kB]
Get:44 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 mailutils-common all 1:3.1.1-1 [642 kB]
Get:45 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 mysql-common all 5.8+1.0.2 [5608 B]
Get:46 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libmariadbclient18 amd64 10.1.26-0+deb9u1 [779 kB]
Get:47 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libpython2.7 amd64 2.7.13-2+deb9u2 [1072 kB]
Get:48 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 libmailutils5 amd64 1:3.1.1-1 [817 kB]
Get:49 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 mailutils amd64 1:3.1.1-1 [573 kB]
Get:50 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 neurodebian-popularity-contest all 0.37.6 [12.3 kB]
Get:51 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 psmisc amd64 22.21-2.1+b2 [123 kB]
Get:52 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 squashfs-tools amd64 1:4.3-3+deb9u1 [124 kB]
Get:53 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 tcpd amd64 7.6.q-26 [23.3 kB]
Get:54 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 xml-core all 0.17 [23.2 kB]
Fetched 25.1 MB in 2s (9777 kB/s)     
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package cron.
(Reading database ... 7131 files and directories currently installed.)
Preparing to unpack .../0-cron_3.0pl1-128+deb9u1_amd64.deb ...
Unpacking cron (3.0pl1-128+deb9u1) ...
Selecting previously unselected package libpython2.7-minimal:amd64.
Preparing to unpack .../1-libpython2.7-minimal_2.7.13-2+deb9u2_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.13-2+deb9u2) ...
Selecting previously unselected package python2.7-minimal.
Preparing to unpack .../2-python2.7-minimal_2.7.13-2+deb9u2_amd64.deb ...
Unpacking python2.7-minimal (2.7.13-2+deb9u2) ...
Selecting previously unselected package python-minimal.
Preparing to unpack .../3-python-minimal_2.7.13-2_amd64.deb ...
Unpacking python-minimal (2.7.13-2) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../4-mime-support_3.60_all.deb ...
Unpacking mime-support (3.60) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../5-libexpat1_2.2.0-2+deb9u1_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.0-2+deb9u1) ...
Selecting previously unselected package libpython2.7-stdlib:amd64.
Preparing to unpack .../6-libpython2.7-stdlib_2.7.13-2+deb9u2_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.13-2+deb9u2) ...
Selecting previously unselected package python2.7.
Preparing to unpack .../7-python2.7_2.7.13-2+deb9u2_amd64.deb ...
Unpacking python2.7 (2.7.13-2+deb9u2) ...
Selecting previously unselected package libpython-stdlib:amd64.
Preparing to unpack .../8-libpython-stdlib_2.7.13-2_amd64.deb ...
Unpacking libpython-stdlib:amd64 (2.7.13-2) ...
Setting up libpython2.7-minimal:amd64 (2.7.13-2+deb9u2) ...
Setting up python2.7-minimal (2.7.13-2+deb9u2) ...
Linking and byte-compiling packages for runtime python2.7...
Setting up python-minimal (2.7.13-2) ...
Selecting previously unselected package python.
(Reading database ... 7950 files and directories currently installed.)
Preparing to unpack .../00-python_2.7.13-2_amd64.deb ...
Unpacking python (2.7.13-2) ...
Selecting previously unselected package popularity-contest.
Preparing to unpack .../01-popularity-contest_1.64_all.deb ...
Unpacking popularity-contest (1.64) ...
Selecting previously unselected package sgml-base.
Preparing to unpack .../02-sgml-base_1.29_all.deb ...
Unpacking sgml-base (1.29) ...
Selecting previously unselected package netbase.
Preparing to unpack .../03-netbase_5.4_all.deb ...
Unpacking netbase (5.4) ...
Selecting previously unselected package bzip2.
Preparing to unpack .../04-bzip2_1.0.6-8.1_amd64.deb ...
Unpacking bzip2 (1.0.6-8.1) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../05-libmagic-mgc_1%3a5.30-1+deb9u2_amd64.deb ...
Unpacking libmagic-mgc (1:5.30-1+deb9u2) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../06-libmagic1_1%3a5.30-1+deb9u2_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.30-1+deb9u2) ...
Selecting previously unselected package file.
Preparing to unpack .../07-file_1%3a5.30-1+deb9u2_amd64.deb ...
Unpacking file (1:5.30-1+deb9u2) ...
Selecting previously unselected package krb5-locales.
Preparing to unpack .../08-krb5-locales_1.15-1+deb9u1_all.deb ...
Unpacking krb5-locales (1.15-1+deb9u1) ...
Selecting previously unselected package libgc1c2:amd64.
Preparing to unpack .../09-libgc1c2_1%3a7.4.2-8_amd64.deb ...
Unpacking libgc1c2:amd64 (1:7.4.2-8) ...
Selecting previously unselected package libgpm2:amd64.
Preparing to unpack .../10-libgpm2_1.20.4-6.2+b1_amd64.deb ...
Unpacking libgpm2:amd64 (1.20.4-6.2+b1) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../11-libkeyutils1_1.5.9-9_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.5.9-9) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../12-libkrb5support0_1.15-1+deb9u1_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.15-1+deb9u1) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../13-libk5crypto3_1.15-1+deb9u1_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.15-1+deb9u1) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../14-libkrb5-3_1.15-1+deb9u1_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.15-1+deb9u1) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../15-libgssapi-krb5-2_1.15-1+deb9u1_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.15-1+deb9u1) ...
Selecting previously unselected package libwrap0:amd64.
Preparing to unpack .../16-libwrap0_7.6.q-26_amd64.deb ...
Unpacking libwrap0:amd64 (7.6.q-26) ...
Selecting previously unselected package libicu57:amd64.
Preparing to unpack .../17-libicu57_57.1-6+deb9u2_amd64.deb ...
Unpacking libicu57:amd64 (57.1-6+deb9u2) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../18-libxml2_2.9.4+dfsg1-2.2+deb9u2_amd64.deb ...
Unpacking libxml2:amd64 (2.9.4+dfsg1-2.2+deb9u2) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../19-xz-utils_5.2.2-1.2+b1_amd64.deb ...
Unpacking xz-utils (5.2.2-1.2+b1) ...
Selecting previously unselected package exim4-config.
Preparing to unpack .../20-exim4-config_4.89-2+deb9u3_all.deb ...
Unpacking exim4-config (4.89-2+deb9u3) ...
Selecting previously unselected package exim4-base.
Preparing to unpack .../21-exim4-base_4.89-2+deb9u3_amd64.deb ...
Unpacking exim4-base (4.89-2+deb9u3) ...
Selecting previously unselected package exim4-daemon-light.
Preparing to unpack .../22-exim4-daemon-light_4.89-2+deb9u3_amd64.deb ...
Unpacking exim4-daemon-light (4.89-2+deb9u3) ...
Selecting previously unselected package exim4.
Preparing to unpack .../23-exim4_4.89-2+deb9u3_all.deb ...
Unpacking exim4 (4.89-2+deb9u3) ...
Selecting previously unselected package libltdl7:amd64.
Preparing to unpack .../24-libltdl7_2.4.6-2_amd64.deb ...
Unpacking libltdl7:amd64 (2.4.6-2) ...
Selecting previously unselected package libncurses5:amd64.
Preparing to unpack .../25-libncurses5_6.0+20161126-1+deb9u2_amd64.deb ...
Unpacking libncurses5:amd64 (6.0+20161126-1+deb9u2) ...
Selecting previously unselected package guile-2.0-libs:amd64.
Preparing to unpack .../26-guile-2.0-libs_2.0.13+1-4_amd64.deb ...
Unpacking guile-2.0-libs:amd64 (2.0.13+1-4) ...
Selecting previously unselected package liblzo2-2:amd64.
Preparing to unpack .../27-liblzo2-2_2.08-1.2+b2_amd64.deb ...
Unpacking liblzo2-2:amd64 (2.08-1.2+b2) ...
Selecting previously unselected package libarchive13:amd64.
Preparing to unpack .../28-libarchive13_3.2.2-2_amd64.deb ...
Unpacking libarchive13:amd64 (3.2.2-2) ...
Selecting previously unselected package libfribidi0:amd64.
Preparing to unpack .../29-libfribidi0_0.19.7-1+b1_amd64.deb ...
Unpacking libfribidi0:amd64 (0.19.7-1+b1) ...
Selecting previously unselected package libntlm0:amd64.
Preparing to unpack .../30-libntlm0_1.4-8_amd64.deb ...
Unpacking libntlm0:amd64 (1.4-8) ...
Selecting previously unselected package libgsasl7.
Preparing to unpack .../31-libgsasl7_1.8.0-8+b2_amd64.deb ...
Unpacking libgsasl7 (1.8.0-8+b2) ...
Selecting previously unselected package libkyotocabinet16v5:amd64.
Preparing to unpack .../32-libkyotocabinet16v5_1.2.76-4.2+b1_amd64.deb ...
Unpacking libkyotocabinet16v5:amd64 (1.2.76-4.2+b1) ...
Selecting previously unselected package mailutils-common.
Preparing to unpack .../33-mailutils-common_1%3a3.1.1-1_all.deb ...
Unpacking mailutils-common (1:3.1.1-1) ...
Selecting previously unselected package mysql-common.
Preparing to unpack .../34-mysql-common_5.8+1.0.2_all.deb ...
Unpacking mysql-common (5.8+1.0.2) ...
Selecting previously unselected package libmariadbclient18:amd64.
Preparing to unpack .../35-libmariadbclient18_10.1.26-0+deb9u1_amd64.deb ...
Unpacking libmariadbclient18:amd64 (10.1.26-0+deb9u1) ...
Selecting previously unselected package libpython2.7:amd64.
Preparing to unpack .../36-libpython2.7_2.7.13-2+deb9u2_amd64.deb ...
Unpacking libpython2.7:amd64 (2.7.13-2+deb9u2) ...
Selecting previously unselected package libmailutils5:amd64.
Preparing to unpack .../37-libmailutils5_1%3a3.1.1-1_amd64.deb ...
Unpacking libmailutils5:amd64 (1:3.1.1-1) ...
Selecting previously unselected package mailutils.
Preparing to unpack .../38-mailutils_1%3a3.1.1-1_amd64.deb ...
Unpacking mailutils (1:3.1.1-1) ...
Selecting previously unselected package neurodebian-popularity-contest.
Preparing to unpack .../39-neurodebian-popularity-contest_0.37.6_all.deb ...
Unpacking neurodebian-popularity-contest (0.37.6) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../40-psmisc_22.21-2.1+b2_amd64.deb ...
Unpacking psmisc (22.21-2.1+b2) ...
Selecting previously unselected package squashfs-tools.
Preparing to unpack .../41-squashfs-tools_1%3a4.3-3+deb9u1_amd64.deb ...
Unpacking squashfs-tools (1:4.3-3+deb9u1) ...
Selecting previously unselected package singularity-container.
Preparing to unpack .../42-singularity-container_2.5.2-2~nd90+1_amd64.deb ...
Unpacking singularity-container (2.5.2-2~nd90+1) ...
Selecting previously unselected package tcpd.
Preparing to unpack .../43-tcpd_7.6.q-26_amd64.deb ...
Unpacking tcpd (7.6.q-26) ...
Selecting previously unselected package xml-core.
Preparing to unpack .../44-xml-core_0.17_all.deb ...
Unpacking xml-core (0.17) ...
Setting up libncurses5:amd64 (6.0+20161126-1+deb9u2) ...
Setting up libexpat1:amd64 (2.2.0-2+deb9u1) ...
Setting up psmisc (22.21-2.1+b2) ...
Setting up exim4-config (4.89-2+deb9u3) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Adding system-user for exim (v4)
Setting up mime-support (3.60) ...
Setting up libgpm2:amd64 (1.20.4-6.2+b1) ...
Setting up mysql-common (5.8+1.0.2) ...
update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Setting up libgc1c2:amd64 (1:7.4.2-8) ...
Setting up libmariadbclient18:amd64 (10.1.26-0+deb9u1) ...
Setting up sgml-base (1.29) ...
Setting up libicu57:amd64 (57.1-6+deb9u2) ...
Setting up libxml2:amd64 (2.9.4+dfsg1-2.2+deb9u2) ...
Setting up libmagic-mgc (1:5.30-1+deb9u2) ...
Setting up bzip2 (1.0.6-8.1) ...
Setting up libmagic1:amd64 (1:5.30-1+deb9u2) ...
Setting up libntlm0:amd64 (1.4-8) ...
Setting up krb5-locales (1.15-1+deb9u1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up xz-utils (5.2.2-1.2+b1) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
Setting up libltdl7:amd64 (2.4.6-2) ...
Setting up libfribidi0:amd64 (0.19.7-1+b1) ...
Setting up xml-core (0.17) ...
Setting up libkeyutils1:amd64 (1.5.9-9) ...
Setting up cron (3.0pl1-128+deb9u1) ...
Adding group `crontab' (GID 102) ...
Done.
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up libpython2.7-stdlib:amd64 (2.7.13-2+deb9u2) ...
Setting up liblzo2-2:amd64 (2.08-1.2+b2) ...
Setting up libwrap0:amd64 (7.6.q-26) ...
Setting up mailutils-common (1:3.1.1-1) ...
Setting up popularity-contest (1.64) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Configuring popularity-contest
------------------------------

The system may anonymously supply the distribution developers with statistics about the most used packages on this system.  This information influences decisions such as
which packages should go on the first distribution CD.

If you choose to participate, the automatic submission script will run once every week, sending statistics to the distribution developers. The collected statistics can be
viewed on http://popcon.debian.org/.

This choice can be later modified by running "dpkg-reconfigure popularity-contest".

Participate in the package usage survey? [yes/no] no

Setting up netbase (5.4) ...
Setting up neurodebian-popularity-contest (0.37.6) ...
Setting up libarchive13:amd64 (3.2.2-2) ...
Setting up python2.7 (2.7.13-2+deb9u2) ...
Setting up exim4-base (4.89-2+deb9u3) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
exim: DB upgrade, deleting hints-db
Setting up file (1:5.30-1+deb9u2) ...
Setting up tcpd (7.6.q-26) ...
Setting up libkrb5support0:amd64 (1.15-1+deb9u1) ...
Setting up libpython-stdlib:amd64 (2.7.13-2) ...
Setting up squashfs-tools (1:4.3-3+deb9u1) ...
Setting up guile-2.0-libs:amd64 (2.0.13+1-4) ...
Setting up libpython2.7:amd64 (2.7.13-2+deb9u2) ...
Setting up exim4-daemon-light (4.89-2+deb9u3) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Initializing GnuTLS DH parameter file
Setting up libkyotocabinet16v5:amd64 (1.2.76-4.2+b1) ...
Setting up python (2.7.13-2) ...
Setting up libk5crypto3:amd64 (1.15-1+deb9u1) ...
Setting up exim4 (4.89-2+deb9u3) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Setting up libkrb5-3:amd64 (1.15-1+deb9u1) ...
Setting up singularity-container (2.5.2-2~nd90+1) ...
Setting up libgssapi-krb5-2:amd64 (1.15-1+deb9u1) ...
Setting up libgsasl7 (1.8.0-8+b2) ...
Setting up libmailutils5:amd64 (1:3.1.1-1) ...
Setting up mailutils (1:3.1.1-1) ...
update-alternatives: using /usr/bin/frm.mailutils to provide /usr/bin/frm (frm) in auto mode
update-alternatives: using /usr/bin/from.mailutils to provide /usr/bin/from (from) in auto mode
update-alternatives: using /usr/bin/messages.mailutils to provide /usr/bin/messages (messages) in auto mode
update-alternatives: using /usr/bin/movemail.mailutils to provide /usr/bin/movemail (movemail) in auto mode
update-alternatives: using /usr/bin/readmsg.mailutils to provide /usr/bin/readmsg (readmsg) in auto mode
update-alternatives: using /usr/bin/dotlock.mailutils to provide /usr/bin/dotlock (dotlock) in auto mode
update-alternatives: using /usr/bin/mail.mailutils to provide /usr/bin/mailx (mailx) in auto mode
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Processing triggers for sgml-base (1.29) ...
root@8594c9868539:/# which singularity
/usr/bin/singularity

But just a regular pull seems to work ok:

# singularity pull shub://vsoch/hello-world
Progress |===================================| 100.0% 
Done. Container is at: //vsoch-hello-world-master-latest.simg

Your pull seems to work okay too:

# singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Done. Container is at: //d13898a106f9678c695ab479335e285c634d3686.simg

You could look at files under the hood, these ones

# ls /var/lib/dpkg/info/singularity-container.
singularity-container.conffiles  singularity-container.list       singularity-container.md5sums    singularity-container.shlibs     singularity-container.triggers

cat /var/lib/dpkg/info/singularity-container.conffiles
/etc/singularity/default-nsswitch.conf
/etc/singularity/init
/etc/singularity/nvliblist.conf
/etc/singularity/singularity.conf



md5-d760d72ab42b29934f9fa7c751585f4a



ls -l /etc/singularity
total 20
-rw-r--r-- 1 root root   87 Aug  1 18:19 default-nsswitch.conf
-rw-r--r-- 1 root root 1464 Aug  1 18:19 init
-rw-r--r-- 1 root root  866 Aug  1 18:19 nvliblist.conf
-rw-r--r-- 1 root root 6717 Aug  1 18:19 singularity.conf



md5-6fc4ccfd99b516728a96b919bd80d725



cat /var/lib/dpkg/info/singularity-container.list
/.
/etc
/etc/singularity
/etc/singularity/default-nsswitch.conf
/etc/singularity/init
/etc/singularity/nvliblist.conf
/etc/singularity/singularity.conf
/usr
/usr/bin
/usr/bin/run-singularity
/usr/bin/singularity
/usr/include
/usr/include/singularity
/usr/include/singularity/bind.h
/usr/include/singularity/image.h
/usr/include/singularity/runtime.h
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/singularity
/usr/lib/x86_64-linux-gnu/singularity/bin
/usr/lib/x86_64-linux-gnu/singularity/bin/action
/usr/lib/x86_64-linux-gnu/singularity/bin/action-suid
/usr/lib/x86_64-linux-gnu/singularity/bin/builddef
/usr/lib/x86_64-linux-gnu/singularity/bin/cleanupd
/usr/lib/x86_64-linux-gnu/singularity/bin/docker-extract
/usr/lib/x86_64-linux-gnu/singularity/bin/get-section
/usr/lib/x86_64-linux-gnu/singularity/bin/image-type
/usr/lib/x86_64-linux-gnu/singularity/bin/mount
/usr/lib/x86_64-linux-gnu/singularity/bin/mount-suid
/usr/lib/x86_64-linux-gnu/singularity/bin/prepheader
/usr/lib/x86_64-linux-gnu/singularity/bin/start
/usr/lib/x86_64-linux-gnu/singularity/bin/start-suid
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/checks.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-driver-arch.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-driver-busybox.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-driver-debootstrap.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-driver-docker.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-driver-localimage.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-driver-self.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-driver-shub.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-driver-yum.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-driver-zypper.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-post.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/deffile-sections.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/environment.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/environment.tar
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/functions
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/main-deffile.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/main-dockerhub.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/main-null.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/post.sh
/usr/lib/x86_64-linux-gnu/singularity/bootstrap-scripts/pre.sh
/usr/lib/x86_64-linux-gnu/singularity/cli
/usr/lib/x86_64-linux-gnu/singularity/cli/action_argparser.sh
/usr/lib/x86_64-linux-gnu/singularity/cli/apps.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/apps.info
/usr/lib/x86_64-linux-gnu/singularity/cli/bootstrap.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/bootstrap.info
/usr/lib/x86_64-linux-gnu/singularity/cli/build.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/build.info
/usr/lib/x86_64-linux-gnu/singularity/cli/check.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/check.info
/usr/lib/x86_64-linux-gnu/singularity/cli/create.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/create.info
/usr/lib/x86_64-linux-gnu/singularity/cli/exec.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/exec.info
/usr/lib/x86_64-linux-gnu/singularity/cli/help.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/help.info
/usr/lib/x86_64-linux-gnu/singularity/cli/image.create.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/image.create.info
/usr/lib/x86_64-linux-gnu/singularity/cli/image.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/image.expand.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/image.expand.info
/usr/lib/x86_64-linux-gnu/singularity/cli/image.export.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/image.export.info
/usr/lib/x86_64-linux-gnu/singularity/cli/image.import.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/image.import.info
/usr/lib/x86_64-linux-gnu/singularity/cli/image.info
/usr/lib/x86_64-linux-gnu/singularity/cli/inspect.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/inspect.info
/usr/lib/x86_64-linux-gnu/singularity/cli/instance.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/instance.info
/usr/lib/x86_64-linux-gnu/singularity/cli/instance.list.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/instance.list.info
/usr/lib/x86_64-linux-gnu/singularity/cli/instance.start.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/instance.start.info
/usr/lib/x86_64-linux-gnu/singularity/cli/instance.stop.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/instance.stop.info
/usr/lib/x86_64-linux-gnu/singularity/cli/mount.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/mount.info
/usr/lib/x86_64-linux-gnu/singularity/cli/pull.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/pull.info
/usr/lib/x86_64-linux-gnu/singularity/cli/run.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/run.info
/usr/lib/x86_64-linux-gnu/singularity/cli/selftest.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/selftest.info
/usr/lib/x86_64-linux-gnu/singularity/cli/shell.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/shell.info
/usr/lib/x86_64-linux-gnu/singularity/cli/test.exec
/usr/lib/x86_64-linux-gnu/singularity/cli/test.info
/usr/lib/x86_64-linux-gnu/singularity/functions
/usr/lib/x86_64-linux-gnu/singularity/handlers
/usr/lib/x86_64-linux-gnu/singularity/handlers/archive-cpio.sh
/usr/lib/x86_64-linux-gnu/singularity/handlers/archive-tar.sh
/usr/lib/x86_64-linux-gnu/singularity/handlers/image-docker.sh
/usr/lib/x86_64-linux-gnu/singularity/handlers/image-http.sh
/usr/lib/x86_64-linux-gnu/singularity/handlers/image-instance.sh
/usr/lib/x86_64-linux-gnu/singularity/handlers/image-shub.sh
/usr/lib/x86_64-linux-gnu/singularity/helpers
/usr/lib/x86_64-linux-gnu/singularity/helpers/apps
/usr/lib/x86_64-linux-gnu/singularity/helpers/apps/list.sh
/usr/lib/x86_64-linux-gnu/singularity/helpers/check.sh
/usr/lib/x86_64-linux-gnu/singularity/helpers/checks
/usr/lib/x86_64-linux-gnu/singularity/helpers/checks/1-bash-hiddens.py
/usr/lib/x86_64-linux-gnu/singularity/helpers/checks/1-cache-content.py
/usr/lib/x86_64-linux-gnu/singularity/helpers/checks/1-docker.py
/usr/lib/x86_64-linux-gnu/singularity/helpers/checks/1-hello-world.sh
/usr/lib/x86_64-linux-gnu/singularity/helpers/checks/3-cve.py
/usr/lib/x86_64-linux-gnu/singularity/helpers/help.sh
/usr/lib/x86_64-linux-gnu/singularity/helpers/image.sh
/usr/lib/x86_64-linux-gnu/singularity/helpers/inspect.sh
/usr/lib/x86_64-linux-gnu/singularity/helpers/record-env.sh
/usr/lib/x86_64-linux-gnu/singularity/image-handler.sh
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-image.a
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-image.la
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-image.so.1.0.0
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-runtime.a
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-runtime.la
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-runtime.so.1.0.0
/usr/lib/x86_64-linux-gnu/singularity/python
/usr/lib/x86_64-linux-gnu/singularity/python/__init__.py
/usr/lib/x86_64-linux-gnu/singularity/python/base.py
/usr/lib/x86_64-linux-gnu/singularity/python/defaults.py
/usr/lib/x86_64-linux-gnu/singularity/python/docker
/usr/lib/x86_64-linux-gnu/singularity/python/docker/__init__.py
/usr/lib/x86_64-linux-gnu/singularity/python/docker/api.py
/usr/lib/x86_64-linux-gnu/singularity/python/docker/main.py
/usr/lib/x86_64-linux-gnu/singularity/python/docker/tasks.py
/usr/lib/x86_64-linux-gnu/singularity/python/helpers
/usr/lib/x86_64-linux-gnu/singularity/python/helpers/__init__.py
/usr/lib/x86_64-linux-gnu/singularity/python/helpers/json
/usr/lib/x86_64-linux-gnu/singularity/python/helpers/json/__init__.py
/usr/lib/x86_64-linux-gnu/singularity/python/helpers/json/add.py
/usr/lib/x86_64-linux-gnu/singularity/python/helpers/json/delete.py
/usr/lib/x86_64-linux-gnu/singularity/python/helpers/json/dump.py
/usr/lib/x86_64-linux-gnu/singularity/python/helpers/json/get.py
/usr/lib/x86_64-linux-gnu/singularity/python/helpers/json/inspect.py
/usr/lib/x86_64-linux-gnu/singularity/python/helpers/json/main.py
/usr/lib/x86_64-linux-gnu/singularity/python/import.py
/usr/lib/x86_64-linux-gnu/singularity/python/message.py
/usr/lib/x86_64-linux-gnu/singularity/python/pull.py
/usr/lib/x86_64-linux-gnu/singularity/python/shell.py
/usr/lib/x86_64-linux-gnu/singularity/python/shub
/usr/lib/x86_64-linux-gnu/singularity/python/shub/__init__.py
/usr/lib/x86_64-linux-gnu/singularity/python/shub/api.py
/usr/lib/x86_64-linux-gnu/singularity/python/shub/main.py
/usr/lib/x86_64-linux-gnu/singularity/python/size.py
/usr/lib/x86_64-linux-gnu/singularity/python/sutils.py
/usr/lib/x86_64-linux-gnu/singularity/python/templates.py
/usr/share
/usr/share/bash-completion
/usr/share/bash-completion/completions
/usr/share/bash-completion/completions/singularity
/usr/share/doc
/usr/share/doc/singularity-container
/usr/share/doc/singularity-container/CONTRIBUTORS.md
/usr/share/doc/singularity-container/README.md.gz
/usr/share/doc/singularity-container/changelog.Debian.gz
/usr/share/doc/singularity-container/changelog.gz
/usr/share/doc/singularity-container/copyright
/usr/share/doc/singularity-container/examples
/usr/share/doc/singularity-container/examples/README.md
/usr/share/doc/singularity-container/examples/apps
/usr/share/doc/singularity-container/examples/apps/README.md
/usr/share/doc/singularity-container/examples/apps/Singularity
/usr/share/doc/singularity-container/examples/apps/Singularity.cowsay
/usr/share/doc/singularity-container/examples/arch
/usr/share/doc/singularity-container/examples/arch/README.md
/usr/share/doc/singularity-container/examples/arch/Singularity
/usr/share/doc/singularity-container/examples/asciinema
/usr/share/doc/singularity-container/examples/asciinema/Singularity
/usr/share/doc/singularity-container/examples/busybox
/usr/share/doc/singularity-container/examples/busybox/Singularity
/usr/share/doc/singularity-container/examples/centos
/usr/share/doc/singularity-container/examples/centos/Singularity
/usr/share/doc/singularity-container/examples/docker
/usr/share/doc/singularity-container/examples/docker/Singularity
/usr/share/doc/singularity-container/examples/legacy
/usr/share/doc/singularity-container/examples/legacy/2.1
/usr/share/doc/singularity-container/examples/legacy/2.1/arch.def
/usr/share/doc/singularity-container/examples/legacy/2.1/busybox.def
/usr/share/doc/singularity-container/examples/legacy/2.1/centos.def
/usr/share/doc/singularity-container/examples/legacy/2.1/debian.def
/usr/share/doc/singularity-container/examples/legacy/2.1/fedora.def
/usr/share/doc/singularity-container/examples/legacy/2.1/scientific.def
/usr/share/doc/singularity-container/examples/legacy/2.1/ubuntu.def
/usr/share/doc/singularity-container/examples/legacy/2.2
/usr/share/doc/singularity-container/examples/legacy/2.2/README
/usr/share/doc/singularity-container/examples/legacy/2.2/arch.def
/usr/share/doc/singularity-container/examples/legacy/2.2/busybox.def
/usr/share/doc/singularity-container/examples/legacy/2.2/centos.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/centos-minimal.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/centos7-ompi_cuda.def.gz
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/centos7-ompi_master.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/debian85-tensorflow-0.10.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/fedora.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/linuxbrew_and_non-root_software_example.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/r_python_julia.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/ubuntu-bio.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/ubuntu-openfoam.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/ubuntu-root.def
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/ubuntu16-tensorflow-0.12.1-gpu.def.gz
/usr/share/doc/singularity-container/examples/legacy/2.2/contrib/ubuntu16-tensorflow-0.12.1.def
/usr/share/doc/singularity-container/examples/legacy/2.2/debian.def
/usr/share/doc/singularity-container/examples/legacy/2.2/docker.def
/usr/share/doc/singularity-container/examples/legacy/2.2/scientific.def
/usr/share/doc/singularity-container/examples/legacy/2.2/ubuntu.def
/usr/share/doc/singularity-container/examples/legacy/2.3
/usr/share/doc/singularity-container/examples/legacy/2.3/contrib
/usr/share/doc/singularity-container/examples/legacy/2.3/contrib/raspbian.def
/usr/share/doc/singularity-container/examples/opensuse
/usr/share/doc/singularity-container/examples/opensuse/Singularity
/usr/share/doc/singularity-container/examples/raspbian
/usr/share/doc/singularity-container/examples/raspbian/Singularity
/usr/share/doc/singularity-container/examples/scientific
/usr/share/doc/singularity-container/examples/scientific/Singularity
/usr/share/doc/singularity-container/examples/self
/usr/share/doc/singularity-container/examples/self/README.md
/usr/share/doc/singularity-container/examples/self/Singularity
/usr/share/doc/singularity-container/examples/shub
/usr/share/doc/singularity-container/examples/shub/Singularity
/usr/share/doc/singularity-container/examples/ubuntu
/usr/share/doc/singularity-container/examples/ubuntu/Singularity
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/singularity-container
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/run-singularity.1.gz
/usr/share/man/man1/singularity.1.gz
/var
/var/lib
/var/lib/singularity
/var/lib/singularity/mnt
/var/lib/singularity/mnt/container
/var/lib/singularity/mnt/final
/var/lib/singularity/mnt/overlay
/var/lib/singularity/mnt/session
/var/lib/singularity/mnt/source
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-image.so
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-image.so.1
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-runtime.so
/usr/lib/x86_64-linux-gnu/singularity/libsingularity-runtime.so.1



md5-6fc4ccfd99b516728a96b919bd80d725



cat /var/lib/dpkg/info/singularity-container.triggers 
# Triggers added by dh_makeshlibs
activate-noawait ldconfig

@mwiens91 could you provide something akin to the above to reproduce your issue? It's hard to be able to debug if this is not possible for us in the peanut gallery :)

vsoch picture vsoch on 11 Sep 2018

And might want to ask Dave Love? https://launchpadlibrarian.net/381358601/singularity-container_2.5.2-2.dsc

vsoch picture vsoch on 11 Sep 2018

@mwiens91 please point to your Travis setup

yarikoptic picture yarikoptic on 11 Sep 2018

@mwiens91 please point to your Travis setup

This is one of them where it's spitting out the error.

https://github.com/mwiens91/saltant/blob/master/.travis.yml

It's hard to be able to debug if this is not possible for us in the peanut gallery :)

This should be very easy to reproduce. Create any Ubuntu VM (I've tried Ubuntu 14.04, 16.04, and 18.04 and the error occurs on all of them) and install Singularity using apt.

For example, I just spawned a fresh Ubuntu 18.04 Microsoft Azure VM and ran the following:

matt@raidboss ~ $ ssh 52.175.200.32
The authenticity of host '52.175.200.32 (52.175.200.32)' can't be established.
ECDSA key fingerprint is SHA256:OfMs1E0ZoVYxYIuA46a1HodHDC2YwZ48dckbHahwEKQ.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '52.175.200.32' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1022-azure x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

 System information disabled due to load higher than 1.0

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

_____________________________________________________________________
WARNING! Your environment specifies an invalid locale.
 The unknown environment variables are:
   LC_CTYPE=en_CA.UTF-8 LC_ALL=
 This can affect your user experience significantly, including the
 ability to manage packages. You may install the locales by running:

   sudo apt-get install language-pack-en
     or
   sudo locale-gen en_CA.UTF-8

To see all available language packs, run:
   apt-cache search "^language-pack-[a-z][a-z]$"
To disable this message for all users, run:
   sudo touch /var/lib/cloud/instance/locale-check.skip
_____________________________________________________________________

matt@asdf:~$ wget -O- http://neuro.debian.net/lists/trusty.us-ca.libre | sudo tee /etc/apt/sources.list.d/neurodebian.sources.list
--2018-09-11 17:22:41--  http://neuro.debian.net/lists/trusty.us-ca.libre
Resolving neuro.debian.net (neuro.debian.net)... 129.170.30.193
Connecting to neuro.debian.net (neuro.debian.net)|129.170.30.193|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 194
Saving to: 'STDOUT'

-                             100%[===============================================>]     194  --.-KB/s    in 0s      

2018-09-11 17:22:42 (37.8 MB/s) - written to stdout [194/194]

deb http://neurodeb.pirsquared.org data main
#deb-src http://neurodeb.pirsquared.org data main
deb http://neurodeb.pirsquared.org trusty main
#deb-src http://neurodeb.pirsquared.org trusty main
matt@asdf:~$ sudo apt-key adv --recv-keys --keyserver hkp://pool.sks-keyservers.net:80 0xA5D32F012649A5A9
Executing: /tmp/apt-key-gpghome.XkfdsJVOZb/gpg.1.sh --recv-keys --keyserver hkp://pool.sks-keyservers.net:80 0xA5D32F012649A5A9
gpg: key A5D32F012649A5A9: 7 signatures not checked due to missing keys
gpg: key A5D32F012649A5A9: public key "NeuroDebian Archive Key <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
matt@asdf:~$ sudo apt update
Hit:1 http://azure.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://azure.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://azure.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:4 http://azure.archive.ubuntu.com/ubuntu bionic/restricted Sources [5324 B]                                      
Get:5 http://azure.archive.ubuntu.com/ubuntu bionic/main Sources [829 kB]                                            
Get:6 http://azure.archive.ubuntu.com/ubuntu bionic/multiverse Sources [181 kB]                                      
Get:7 http://azure.archive.ubuntu.com/ubuntu bionic/universe Sources [9051 kB]                                       
Get:8 http://neurodeb.pirsquared.org data InRelease [25.7 kB]                                                        
Get:9 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]                                          
Get:10 http://neurodeb.pirsquared.org trusty InRelease [18.4 kB]                                                     
Get:11 http://azure.archive.ubuntu.com/ubuntu bionic-updates/multiverse Sources [2672 B]                             
Get:12 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main Sources [163 kB]                              
Get:13 http://azure.archive.ubuntu.com/ubuntu bionic-updates/universe Sources [61.7 kB]                         
Get:14 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [319 kB]                        
Get:15 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [120 kB]                        
Get:16 http://azure.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [189 kB]              
Get:17 http://azure.archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [88.6 kB]                   
Get:18 http://azure.archive.ubuntu.com/ubuntu bionic-backports/universe Sources [1184 B]                             
Get:19 http://neurodeb.pirsquared.org data/main amd64 Packages [6579 B]   
Get:20 http://neurodeb.pirsquared.org trusty/main amd64 Packages [84.7 kB]
Get:21 http://security.ubuntu.com/ubuntu bionic-security/main Sources [46.5 kB]      
Get:22 http://security.ubuntu.com/ubuntu bionic-security/multiverse Sources [1336 B]
Get:23 http://security.ubuntu.com/ubuntu bionic-security/universe Sources [16.9 kB]
Get:24 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [166 kB]
Get:25 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [62.6 kB]
Get:26 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [66.0 kB]
Get:27 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [38.9 kB]
Fetched 11.8 MB in 3s (3928 kB/s)          
Reading package lists... Done
Building dependency tree       
Reading state information... Done
27 packages can be upgraded. Run 'apt list --upgradable' to see them.
matt@asdf:~$ sudo apt install singularity-container
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libarchive13 neurodebian-popularity-contest
Suggested packages:
  lrzip
The following NEW packages will be installed:
  libarchive13 neurodebian-popularity-contest singularity-container
0 upgraded, 3 newly installed, 0 to remove and 27 not upgraded.
Need to get 635 kB of archives.
After this operation, 3357 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libarchive13 amd64 3.2.2-3.1ubuntu0.1 [289 kB]
Get:2 http://neurodeb.pirsquared.org trusty/main amd64 singularity-container amd64 2.5.2-2~nd14.04+1 [337 kB]
Get:3 http://azure.archive.ubuntu.com/ubuntu bionic/universe amd64 neurodebian-popularity-contest all 0.37.6 [8764 B]
Fetched 635 kB in 1s (1249 kB/s)                                                                 
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "en_CA.UTF-8",
    LANG = "C.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("C.UTF-8").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
Selecting previously unselected package libarchive13:amd64.
(Reading database ... 56510 files and directories currently installed.)
Preparing to unpack .../libarchive13_3.2.2-3.1ubuntu0.1_amd64.deb ...
Unpacking libarchive13:amd64 (3.2.2-3.1ubuntu0.1) ...
Selecting previously unselected package neurodebian-popularity-contest.
Preparing to unpack .../neurodebian-popularity-contest_0.37.6_all.deb ...
Unpacking neurodebian-popularity-contest (0.37.6) ...
Selecting previously unselected package singularity-container.
Preparing to unpack .../singularity-container_2.5.2-2~nd14.04+1_amd64.deb ...
Unpacking singularity-container (2.5.2-2~nd14.04+1) ...
Setting up neurodebian-popularity-contest (0.37.6) ...
Setting up libarchive13:amd64 (3.2.2-3.1ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for man-db (2.8.3-2) ...
Setting up singularity-container (2.5.2-2~nd14.04+1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
matt@asdf:~$ singularity pull --commit shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
ERROR  : Called singularity_config_get_value on uninitialized config subsystem
ABORT  : Retval = 255
Done. Container is at: /home/matt/d13898a106f9678c695ab479335e285c634d3686.simg
mwiens91 picture mwiens91 on 11 Sep 2018

I don't think you can assume a VM provided by Microsoft (Azure) is comparable to any other provider, and I can't offer up institutional (Stanford) dollars to debug an error not related directly to my work. And I definitely don't use Azure. :P I'd like to help and give it a try - can you give instructions for a Vagrantfile or similar, if that reproduces it?

vsoch picture vsoch on 11 Sep 2018

FWIW, reproduced on AWS EC2 ubuntu xenial, will try to have a close look

yarikoptic picture yarikoptic on 11 Sep 2018

hey @vsoch - how do you gdb singularity pull -- wanted to see WTF that config_initialized is not getting set properly since it seems to read the singularity.conf just

but also - why don't we see any of VERBOSE (not VERBOSE2 or DEBUG) messages such as

$> git grep VERBOSE, | grep config_
src/util/config_parser.c:    singularity_message(VERBOSE, "Initialize configuration file: %s\n", config_path);

there also should be all those Starting parse of configuration file etc messages -- is there more than --debug to pass?

most likely problem is that singularity_config_parse somehow but we do not see for some reason any of VERBOSE or DEBUG messages coming from it to actually get an idea. any clues
?

yarikoptic picture yarikoptic on 11 Sep 2018

reason for radio silence is that .py backend doesn't care about cmdline --debug (would be nice to have it fixed!):

> /usr/lib/x86_64-linux-gnu/singularity/python/sutils.py(254)get_image_format()
-> image_format = run_command([sbin, image_file], env=custom_env, quiet=True)
(Pdb) p sbin
'/usr/lib/x86_64-linux-gnu/singularity/bin/image-type'
(Pdb) p image_file
u'/tmp/mwiens91-shub-debug-master-latest'

see unconditional quiet=True

yarikoptic picture yarikoptic on 11 Sep 2018

and the ultimate "reason" is that config_initialized seems to be not that global (i.e. static) as it hoped/assumed to be:

(gdb) up
#2  0x0000555555556db6 in main (argc=2, argv=0x7fffffffe0e8) at image-type.c:83
83          image = singularity_image_init(argv[1], O_RDONLY);
(gdb) p config_initialized
$13 = 1
(gdb) p &config_initialized
$14 = (int *) 0x55555575f500 <config_initialized>
(gdb) down
#1  0x00007ffff7bc3c8c in singularity_image_init (
    path=0x7fffffffe35d "/tmp/mwiens91-shub-debug-master-latest", open_flags=0)
    at image.c:101
101             if ( ( singularity_config_get_bool(ALLOW_CONTAINER_SQUASHFS) <= 0 ) && ( singularity_priv_getuid() != 0 ) ) {
(gdb) p config_initialized
$15 = 1
(gdb) down
#0  _singularity_config_get_bool_impl (
    key=0x7ffff7bcf452 "allow container squashfs", def=1)
    at ../../util/config_parser.c:300
300         return _singularity_config_get_bool_char_impl(key, def ? "yes" : "no");
(gdb) p config_initialized
$16 = 0
(gdb) p &config_initialized
$17 = (int *) 0x7ffff7dd5390 <config_initialized>

nnot yet sure what is exactly the reason here -- my C is very rusty

yarikoptic picture yarikoptic on 11 Sep 2018

just to make sure - it relates to build on ubuntu, installing http://neuro.debian.net/debian/pool/main/s/singularity-container/singularity-container_2.5.2-2~nd90+1_amd64.deb doesn't show that.

yarikoptic picture yarikoptic on 12 Sep 2018

I can't comment on the C - the python backend I believe is largely being deprecated / going away, and was never responsible for this bit of the parsing. The outputs with --debug and -vvvv (levels of verbosity) should if working correctly output the levels you want. I think we need some of the maintainers to look at the debug above and help with reading it...

vsoch picture vsoch on 12 Sep 2018

I've uploaded those build logs for 2.5.2-2 to http://neuro.debian.net/_files/_buildlogs/singularity-container/2.5.2 happen someone more knowledgeable could get a clue from build logs. I see that on ubuntu -Wl,-Bsymbolic-functions -Wl,-z seems to be used...

yarikoptic picture yarikoptic on 12 Sep 2018

FWiW it is also reproducible in Ubuntu (! not Debian, which is what we all love for its robust operation!;) ) docker:

$> docker run -it --rm neurodebian:nd16.04
root@bf987c74ce4e:/# apt-get update -qqq; apt-get install -y -qq eatmydata vim
...
root@bf987c74ce4e:/# eatmydata apt install singularity-container
...
root@bf987c74ce4e:/tmp# singularity pull shub://mwiens91/shub-debug
Progress |===================================| 100.0% 
Singularity: image-type (U=0,P=3566)> Called singularity_config_get_value on uninitialized config subsystem

ERROR  : Called singularity_config_get_value on uninitialized config subsystem
Singularity: image-type (U=0,P=3566)> Retval = 255

ABORT  : Retval = 255
Done. Container is at: /tmp/mwiens91-shub-debug-master-latest.simg
yarikoptic picture yarikoptic on 12 Sep 2018

Okay .... I'm not seeing anything jumping out at me yet. Your source _does_ at least look correct, in that the calls to singularity_config_init() aren't missing.

https://salsa.debian.org/hpc-team/singularity-container/blob/master/src/start.c#L72

And from the build log, it appears that libinternal_la-config_parser.o is getting added into the library. src/util/config_parser.c is where singularity_config_init() is defined.

The error is showing up because it's not seeing config_initialized as being set, when singularity_conf_get_value() is called. config_initialized _is_ a global static int value set in config_parser.c, and set to a value of 1 when singularity_config_init() is called.

@yarikoptic I think you may be onto something with the -Wl,-Bsymbolic-functions addition. It seems that can have some nasty side effects, and I'm not getting that used on my laptop (GCC 5.3.0, Slackware 14.2) when building.

jmstover picture jmstover on 12 Sep 2018

I think I can confirm that it is the -Wl,-Bsymbolic-functions leading to ... need to figure out the best way to strip (only) that option Ubuntu adds (defined in /usr/share/perl5/Dpkg/Vendor/Ubuntu.pm) to LDFLAGS without completely redefining it.

yarikoptic picture yarikoptic on 12 Sep 2018

Maybe modify configure.ac to do something like:
LDFLAGS=$(echo $LDFLAGS | sed -e 's/-Wl,-Bsymbolic-functions//')

After the # Setting rpath if necessary block.

jmstover picture jmstover on 12 Sep 2018

RTFM helps at times ;) There is a dedicated env variable to deal with such cases, so here is the "fix": https://salsa.debian.org/hpc-team/singularity-container/commit/c049302689fcb830ef4465b5b70c9ce6eed84826
building packages now, will upload 2.5.2-3 to (Neuro)Debians probably early tomorrow after minor re-verification

yarikoptic picture yarikoptic on 12 Sep 2018
👍2 🎉1

Might be outside the scope of NeuroDebian, but any idea of what the process is for this change to be propagated to Ubuntu's Singularity package here:

https://launchpad.net/ubuntu/+source/singularity-container

?

mwiens91 picture mwiens91 on 12 Sep 2018

For upcoming release they just need to sync from Debian, which they probably will do. For the existing releases (if any effected) some MOTU needs to provide the change/patch I guess... I am not first hand familiar with the process

yarikoptic picture yarikoptic on 12 Sep 2018

wow amazing work @yarikoptic @jmstover @mwiens91 !

vsoch picture vsoch on 12 Sep 2018

@mwiens91 could you confirm that on your travis setups which do use neurodebian the problem is gone? Although I see that us-ca mirror you are using still needs to sync (probably in 4 hours): http://neuro.debian.net/mirrors_status.html

yarikoptic picture yarikoptic on 12 Sep 2018

Confirmed that the error message no longer occurs:

https://travis-ci.com/mwiens91/saltant/jobs/144620574#L1048

mwiens91 picture mwiens91 on 12 Sep 2018
👍2

@mwiens91 Sounds good. I'll go a head and close this then.

@yarikoptic Thanks much for the work here.

jmstover picture jmstover on 12 Sep 2018

Solid work everyone, thanks again!

vsoch picture vsoch on 12 Sep 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

dtrudg picture dtrudg  路  3Comments
DrDaveD picture DrDaveD  路  4Comments
al3x609 picture al3x609  路  4Comments
GodloveD picture GodloveD  路  3Comments
chrisgorgo picture chrisgorgo  路  3Comments