Singularity: App name is converted to lowercase during build, case sensitive at runtime

Created on 29 Jan 2019 · 9Comments · Source: hpcng/singularity

Version of Singularity:

3.0.2

Expected behavior

When building from a definition file with mixed-case app name, case should be treated consistently at build time and run time. In my definition file, I have an app named "MyTest"

$cat mydef.def
...
%apprun MyTest
echo "Hello, it worked"
...

One of two behaviors is expected.

Behavior 1. All app names are converted to lower case (or otherwise case insensitive) at build time and at run time.

$singularity build myimage.sif mydef.def
$singularity run --app mYtEsT myimage.sif
Hello, it worked

Behavior 2. All app names are kept in original case at build time and at run time.

$singularity build myimage.sif mydef.def
$singularity run --app MyTest myimage.sif
Hello, it worked

Actual behavior

App names are converted to lower case at build time, but case sensitive at run time. No error is given at runtime, attempts to run are giving no output and return code 0. App runs when called with lowercase name.

$singularity build myimage.sif mydef.def
$singularity run --app MyTest myimage.sif
$echo $?
0
$singularity run --app mytest myimage.sif
Hello, it worked

Steps to reproduce behavior

Include an apprun section in a definition file with a mixed case name. Build and attempt to run with the matching case from the definition file. Running with lower case name runs application.

Bug NeedsMoreInfo

Source

jtullos-intel

All 9 comments

Hi, and Welcome!

I can only seem to replicate your Behavior 2.:

(The def file)

$ cat test.def

bootstrap: docker
from: alpine:3.8

%apprun MyTest
echo "hello world from MyTest"

$ sudo singularity build test.sif test.def

$ singularity run --app MyTest test.sif  # expected to work
hello world from MyTest

$ singularity run --app myTest test.sif  # expected to fail
Could not locate the container application: myTest

WestleyK on 29 Jan 2019

Using the test program you posted, here's what I'm getting:

$ cat test.def
bootstrap: docker
from: alpine:3.8

%apprun MyTest
echo "hello world from MyTest"

$ sudo /usr/local/bin/singularity build test.sif test.def
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:cd784148e3483c2c86c50a48e535302ab0288bebd587accf40b714fffd0646b3
Copying config sha256:b6a11a52f30fe2e64d4a9e89b9b042a0067a0fa9caaf4c9b511296d156857c26
 584 B / 584 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: test.sif
$ singularity run --app MyTest test.sif
Could not locate the container application: MyTest
$ singularity run --app myTest test.sif
Could not locate the container application: myTest
$ singularity run --app mytest test.sif
hello world from MyTest

jtullos-intel on 30 Jan 2019

Thats really weird... Can you try a couple more things:

Try to inspect the definition file in the container:

$ singularity inspect --deffile test.sif 

bootstrap: docker
from: alpine:3.8

%apprun MyTest
echo "hello world from MyTest"

Notice the caps in MyTest.

It may also be useful to build/run the container with -vd (verbose & debug) options:

$ sudo singularity -vd build test.sif test.def 

$ singularity -vd run --app MyTest test.sif

Also, What OS are you running on? I'm using Ubuntu 18.

WestleyK on 30 Jan 2019

@WestleyK @jtullos-intel

Parts of the apps work is not yet implemented in 3.X but were available in 2.X. Pinging @ikaneshiro to see where the status should be.

jscook2345 on 31 Jan 2019

👍1

Agreed, this is completely unexpected.

$ cat /etc/system-release
CentOS Linux release 7.6.1810 (Core)

Inspecting the container:

$ singularity inspect --deffile test.sif

bootstrap: docker
from: alpine:3.8

%post

cd /
. /scif/apps/mytest/scif/env/01-base.sh

cd /scif/apps/mytest/scif


cd /

New build:

$ sudo /usr/local/bin/singularity -vd build test.sif test.def
WARNING [U=0,P=5137]       sylabsToken()                 Authentication token file not found : Only pulls of public images will succeed
DEBUG   [U=0,P=5137]       func1()                       Registering plugin as type BuildPlugin
Build target already exists. Do you want to overwrite? [N/y] y
DEBUG   [U=0,P=5137]       Init()                        Entering image format intializer
DEBUG   [U=0,P=5137]       Init()                        Check for image format sif
DEBUG   [U=0,P=5137]       Init()                        sif format initializer returns: can't read first 2048 bytes: %!s(<nil>)
DEBUG   [U=0,P=5137]       Init()                        Check for image format sandbox
DEBUG   [U=0,P=5137]       Init()                        sandbox format initializer returns: not a directory image
DEBUG   [U=0,P=5137]       Init()                        Check for image format squashfs
DEBUG   [U=0,P=5137]       Init()                        squashfs format initializer returns: can't read first 2048 bytes: %!s(<nil>)
DEBUG   [U=0,P=5137]       Init()                        Check for image format ext3
DEBUG   [U=0,P=5137]       Init()                        ext3 format initializer returns: can't read first 2048 bytes: %!s(<nil>)
DEBUG   [U=0,P=5137]       func1()                       Running singularity_apps plugin: HandleSection() hook
DEBUG   [U=0,P=5137]       NewBundle()                   Created temporary directory for bundle /tmp/sbuild-575678330
INFO    [U=0,P=5137]       Full()                        Starting build...
DEBUG   [U=0,P=5137]       Get()                         Reference: alpine:3.8
DEBUG   [U=0,P=5137]       updateCacheSubdir()           Caching directory set to /root/.singularity/cache/oci
Getting image source signatures
Copying blob sha256:169185f82c45a6eb72e0ca4ee66152626e7ace92a0cbc53624fb46d0a553f0bd
 2.10 MiB / 2.10 MiB [======================================================] 0s
Copying config sha256:4aaec7e8d874331df56132f1c31d24b9bedd00e8588880ed80c0daaaa0b462bf
 585 B / 585 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
DEBUG   [U=0,P=5137]       func1()                       Running singularity_apps plugin: HandleBundle() hook
DEBUG   [U=0,P=5137]       createAllApps()               Creating mytest app in bundle
DEBUG   [U=0,P=5137]       BuildHandlePosts()            Running singularity_apps plugin: HandlePost() hook
DEBUG   [U=0,P=5137]       HandlePost()                  Building app[mytest] post script section
DEBUG   [U=0,P=5137]       Full()                        Inserting Metadata
DEBUG   [U=0,P=5137]       Full()                        Calling assembler
INFO    [U=0,P=5137]       Assemble()                    Creating SIF file...
INFO    [U=0,P=5137]       Full()                        Build complete: test.sif

And running:

$ singularity -vd run --app MyTest test.sif
VERBOSE [U=0,P=5177]       print()                       Set messagelevel to: 5
DEBUG   [U=0,P=5177]       init()                        PIPE_EXEC_FD value: 7
VERBOSE [U=0,P=5177]       init()                        Container runtime
VERBOSE [U=0,P=5177]       is_suid()                     Check if we are running as setuid
DEBUG   [U=0,P=5177]       init()                        Overlay seems supported by kernel
DEBUG   [U=0,P=5177]       init()                        Drop privileges
DEBUG   [U=1000,P=5177]    init()                        Read json configuration from pipe
DEBUG   [U=1000,P=5177]    init()                        Set child signal mask
DEBUG   [U=1000,P=5177]    init()                        Create socketpair for smaster communication channel
DEBUG   [U=1000,P=5177]    init()                        Wait C and JSON runtime configuration from scontainer stage 1
DEBUG   [U=1000,P=5183]    set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=1000,P=5183]    init()                        Spawn scontainer stage 1
VERBOSE [U=1000,P=5183]    priv_escalate()               Get root privileges
DEBUG   [U=0,P=5183]       set_parent_death_signal()     Set parent death signal to 9
DEBUG   [U=0,P=5183]       prepare_scontainer_stage()    Entering in scontainer stage 1
DEBUG   [U=1000,P=5183]    set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=1000,P=5183]    startup()                     Execute scontainer stage 1
DEBUG   [U=1000,P=5183]    Stage()                       Entering scontainer stage 1
DEBUG   [U=1000,P=5183]    Init()                        Entering image format intializer
DEBUG   [U=1000,P=5183]    Init()                        Check for image format sif
DEBUG   [U=1000,P=5177]    init()                        Receiving configuration from scontainer stage 1
DEBUG   [U=1000,P=5177]    init()                        Wait completion of scontainer stage1
VERBOSE [U=1000,P=5177]    priv_escalate()               Get root privileges
VERBOSE [U=0,P=5177]       create_namespace()            Create mount namespace
DEBUG   [U=0,P=5177]       init()                        Create RPC socketpair for communication between scontainer and RPC server
DEBUG   [U=0,P=5177]       init()                        Pass terminal control to child
VERBOSE [U=0,P=5177]       init()                        Spawn smaster process
DEBUG   [U=0,P=5189]       set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=5189]       init()                        Spawn scontainer stage 2
VERBOSE [U=0,P=5189]       create_namespace()            Create mount namespace
VERBOSE [U=0,P=5190]       init()                        Spawn RPC server
VERBOSE [U=1000,P=5177]    startup()                     Execute smaster process
VERBOSE [U=0,P=5190]       startup()                     Serve RPC requests
DEBUG   [U=1000,P=5177]    setupSessionLayout()          Attempting to use overlayfs (enable overlay = try)
DEBUG   [U=1000,P=5177]    setupOverlayLayout()          Creating overlay SESSIONDIR layout
DEBUG   [U=1000,P=5177]    addRootfsMount()              Mount rootfs in read-only mode
DEBUG   [U=1000,P=5177]    addRootfsMount()              Mounting block [squashfs] image: /home/jtullos/basic_test/test.sif
DEBUG   [U=1000,P=5177]    addKernelMount()              Checking configuration file for 'mount proc'
DEBUG   [U=1000,P=5177]    addKernelMount()              Adding proc to mount list
DEBUG   [U=1000,P=5177]    addKernelMount()              Checking configuration file for 'mount sys'
DEBUG   [U=1000,P=5177]    addKernelMount()              Adding sysfs to mount list
DEBUG   [U=1000,P=5177]    addDevMount()                 Checking configuration file for 'mount dev'
DEBUG   [U=1000,P=5177]    addDevMount()                 Adding dev to mount list
DEBUG   [U=1000,P=5177]    addHostMount()                Not mounting host file systems per configuration
VERBOSE [U=1000,P=5177]    addBindsMount()               Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1000,P=5177]    addBindsMount()               Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG   [U=1000,P=5177]    addHomeStagingDir()           Staging home directory (/home/jtullos) at /var/singularity/mnt/session/home/jtullos
DEBUG   [U=1000,P=5177]    addHomeMount()                Adding home directory mount [/var/singularity/mnt/session/home/jtullos:/home/jtullos] to list using layer: overlay
DEBUG   [U=1000,P=5177]    isLayerEnabled()              Using Layer system: overlay
DEBUG   [U=1000,P=5177]    addTmpMount()                 Checking for 'mount tmp' in configuration file
DEBUG   [U=1000,P=5177]    addScratchMount()             Not mounting scratch directory: Not requested
DEBUG   [U=1000,P=5177]    addLibsMount()                Checking for 'user bind control' in configuration file
DEBUG   [U=1000,P=5177]    addResolvConfMount()          Adding /etc/resolv.conf to mount list
DEBUG   [U=1000,P=5177]    addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG   [U=1000,P=5177]    create()                      Mount all
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting tmpfs to /var/singularity/mnt/session
DEBUG   [U=1000,P=5177]    mountImage()                  Mounting loop device /dev/loop0 to /var/singularity/mnt/session/rootfs
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting overlay to /var/singularity/mnt/session/final
DEBUG   [U=1000,P=5177]    setSlaveMount()               Set RPC mount propagation flag to SLAVE
VERBOSE [U=1000,P=5177]    Passwd()                      Checking for template passwd file: /var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=1000,P=5177]    Passwd()                      Creating passwd content
VERBOSE [U=1000,P=5177]    Passwd()                      Creating template passwd file and appending user data: /var/singularity/mnt/session/rootfs/etc/passwd
DEBUG   [U=1000,P=5177]    addIdentityMount()            Adding /etc/passwd to mount list
VERBOSE [U=1000,P=5177]    Group()                       Checking for template group file: /var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=1000,P=5177]    Group()                       Creating group content
DEBUG   [U=1000,P=5177]    addIdentityMount()            Adding /etc/group to mount list
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /dev to /var/singularity/mnt/session/final/dev
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /etc/localtime to /var/singularity/mnt/session/final/etc/localtime
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /etc/hosts to /var/singularity/mnt/session/final/etc/hosts
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /usr/local/etc/singularity/actions to /var/singularity/mnt/session/final/.singularity.d/actions
DEBUG   [U=1000,P=5177]    mountGeneric()                Remounting /var/singularity/mnt/session/final/.singularity.d/actions
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /proc to /var/singularity/mnt/session/final/proc
DEBUG   [U=1000,P=5177]    mountGeneric()                Remounting /var/singularity/mnt/session/final/proc
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting sysfs to /var/singularity/mnt/session/final/sys
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /home/jtullos to /var/singularity/mnt/session/home/jtullos
DEBUG   [U=1000,P=5177]    mountGeneric()                Remounting /var/singularity/mnt/session/home/jtullos
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /var/singularity/mnt/session/home/jtullos to /var/singularity/mnt/session/final/home/jtullos
DEBUG   [U=1000,P=5177]    mountGeneric()                Remounting /var/singularity/mnt/session/final/home/jtullos
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /tmp to /var/singularity/mnt/session/final/tmp
DEBUG   [U=1000,P=5177]    mountGeneric()                Remounting /var/singularity/mnt/session/final/tmp
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /var/tmp to /var/singularity/mnt/session/final/var/tmp
DEBUG   [U=1000,P=5177]    mountGeneric()                Remounting /var/singularity/mnt/session/final/var/tmp
DEBUG   [U=1000,P=5177]    mountGeneric()                Skipping mount /var/singularity/mnt/session/final/home/jtullos/basic_test, /var/singularity/mnt/session/final/home/jtullos already mounted
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /var/singularity/mnt/session/etc/resolv.conf to /var/singularity/mnt/session/final/etc/resolv.conf
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /var/singularity/mnt/session/etc/passwd to /var/singularity/mnt/session/final/etc/passwd
DEBUG   [U=1000,P=5177]    mountGeneric()                Mounting /var/singularity/mnt/session/etc/group to /var/singularity/mnt/session/final/etc/group
DEBUG   [U=1000,P=5177]    create()                      Chroot into /var/singularity/mnt/session/final
DEBUG   [U=0,P=5190]       Chroot()                      Change current directory to /var/singularity/mnt/session/final
DEBUG   [U=0,P=5190]       Chroot()                      Hold reference to host / directory
DEBUG   [U=0,P=5190]       Chroot()                      Called pivot_root on /var/singularity/mnt/session/final
DEBUG   [U=0,P=5190]       Chroot()                      Change current directory to host / directory
DEBUG   [U=0,P=5190]       Chroot()                      Apply slave mount propagation for host / directory
DEBUG   [U=0,P=5190]       Chroot()                      Called unmount(/, syscall.MNT_DETACH)
DEBUG   [U=0,P=5190]       Chroot()                      Changing directory to / to avoid getpwd issues
DEBUG   [U=1000,P=5177]    create()                      Chdir into / to avoid errors
DEBUG   [U=0,P=5189]       set_parent_death_signal()     Set parent death signal to 9
DEBUG   [U=0,P=5189]       prepare_scontainer_stage()    Entering in scontainer stage 2
DEBUG   [U=1000,P=5189]    set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=1000,P=5189]    startup()                     Execute scontainer stage 2
DEBUG   [U=1000,P=5177]    PostStartProcess()            Post start process
Could not locate the container application: MyTest
DEBUG   [U=1000,P=5177]    CleanupContainer()            Cleanup container
DEBUG   [U=1000,P=5177]    Master()                      Child exited with exit status 1

I can grant remote access to the system for debugging if that will help.

jtullos-intel on 31 Jan 2019

Finally I was able to replicate error! Only when I installed Singularity via RPM, (Not build from source)

(From Singularity installed through RPM)

$ singularity inspect --deffile test.sif 

bootstrap: docker
from: alpine:3.8

%post

cd /
. /scif/apps/mytest/scif/env/01-base.sh

cd /scif/apps/mytest/scif


cd /

(From Singularity installed from source)

$ singularity inspect --deffile test.sif 

bootstrap: docker
from: alpine:3.8

%apprun MyTest
echo "hello world from MyTest"

My temporary solution:

Install Singularity from source. or
Try remote builder (Documentation coming soon).

WestleyK on 31 Jan 2019

Interesting. I did build from source, cloned on January 18. I'll try updating and building again in the near future (not enough free time today).

jtullos-intel on 31 Jan 2019

👍1

@WestleyK That should not be an issue. Where did you get the RPM from? Can you ping me on slack? The behavior between RPMs and Source builds should be the same given they have the same git commit.

jscook2345 on 3 Feb 2019

@WestleyK what does "form RPM" mean? where did that RPM come from?