Singularity: Sourcing from /environment doesn't work

[fmorency]

Version of Singularity:

2.3 (neurodebian)

Expected behavior

File is sourced properly when running/executing/shelling the container. This was working in 2.2.1

Actual behavior

File is never sourced.

Steps to reproduce behavior

Create and bootstrap a container using source_environment_issue.zip

# singularity create --size 1000 test.img
# singularity bootstrap test.img test.def

and open a shell in the newly created container. Print the content of the value defined in the sh file

# singularity shell test.img
Singularity: Invoking an interactive shell within container...

Singularity test.img:/tmp/aaa> echo $SOMEVAR

Source the file from within the shell and print the content of the value again

Singularity test.img:/tmp/aaa> . /test/test.sh 
Singularity test.img:/tmp/aaa> echo $SOMEVAR
123

[gmkurtzer]

The /environment file has been superseded to a directory structure at /.singularity.d/env/*.

Sorry!

[vsoch]

hey @fmorency, if you want to redo this functionality with fewer files and a little cleaner syntax, you can pay around with the following:

BootStrap: debootstrap
OSVersion: trusty
MirrorURL: http://ca.archive.ubuntu.com/ubuntu/


%files
   ./test.sh /test

%environment
export SOMEVAR=123

and next time feel free to include your code here! It's a little scary always downloading and extracting a random zip from the internet :)

[vsoch]

and you probably want to define the variable first and export on the next line, I think that format is called a "bashism"

[fmorency]

I'm not sure why, but Singularity doesn't seem to like . /test/test.sh in the %environment section

[fmorency]

Nevermind my previous comment :D

[vsoch]

hey @fmorency did you get this working?

[fmorency]

Yes all good. You can close this issue.

[vsoch]

+1! Thanks @fmorency :)

[jccurtis]

Hi all, I'm sorry for bringing this back up but I'm having the same issue within the %environment section. I need to source a file to activate a software package that I compile in %post however source is not found because the %environment is not executed with bash. I receive the following error:

/.singularity.d/actions/exec: 12: /opt/megalib/bin/source-megalib.sh: source: not found

This script (source-megalib.sh) sources multiple other scripts. Is there some way to source it like in a bashrc file such that any singularity exec will have the environment setup properly?

I've also emailed @gmkurtzer about the issue so sorry for the double messaging Greg! 😄

[vsoch]

Hey @jccurtis ! I have two hacks, neither are perfect, but might help a bit. The first is the stupid way that I came up with quickly. You write a file to source, and have it trigger with the runscript.

BootStrap: docker
From: ubuntu:latest

%runscript
    . /envar.sh
    echo "Pizza is $PIZZA"
    echo "Pasta is $PASTA"

%post
    echo "PIZZA=PIE
          PASTA=JAZZ
          export PIZZA
          export PASTA" > /envar.sh

but that doesn't work for anything beyond the runscript. So an option would be to work with the image via the runscript. That's probably not ideal. Here is option 2:

BootStrap: docker
From: ubuntu:latest

%setup

mkdir -p $SINGULARITY_ROOTS/.singularity.d/env
echo "PIZZA=PIE
PASTA=JAZZ
export PIZZA
export PASTA" >> $SINGULARITY_ROOTFS/.singularity.d/env/80-custom.sh
chmod u+x $SINGULARITY_ROOTFS/.singularity.d/env/80-custom.sh

%runscript
    echo "Pizza is $PIZZA"
    echo "Pasta is $PASTA"

This one takes advantage of the fact that, if you look in exec (or other files under actions) they in fact do the source that you desire!

singularity exec pizza.img cat /.singularity.d/actions/exec
#!/bin/sh

for script in /.singularity.d/env/*.sh; do
    if [ -f "$script" ]; then
        . "$script"
    fi
done
exec "$@"

So all I did was write a file with some variables in it to that folder in %setup, and then it gets sourced:

singularity exec pizza.img env | grep PIZZA
PIZZA=PIE
singularity exec pizza.img env | grep PASTA
PASTA=JAZZ

and then running the image (showing the different action entrypoint)

./pizza.img 
Pizza is PIE
Pasta is JAZZ

and shell is present too:

singularity shell pizza.img 
Singularity: Invoking an interactive shell within container...

Singularity pizza.img:~/Desktop> echo $PIZZA
PIE

The one thing that is weird is that, although the variable is sourced in the environment, when I exec it doesn't show up as it should:

singularity exec pizza.img echo $USER
vanessa
singularity exec pizza.img echo $PIZZA

I added a sanity check to exec to see which files were getting sourced:

singularity exec pizza.img echo $PIZZA
/.singularity.d/env/01-base.sh
/.singularity.d/env/10-docker.sh
/.singularity.d/env/80-custom.sh
/.singularity.d/env/90-environment.sh
/.singularity.d/env/99-base.sh

and it looks like my custom one is, so it's probably just that my script is stupid :) Maybe something like this could work? You wouldn't need to do all the echo, you could just copy that file to be sourced in the %setup like:

%setup
mkdir -p $SINGULARITY_ROOTS/.singularity.d/env
cp source-megalib.sh $SINGULARITY_ROOTS/.singularity.d/env/80-source-megalib.sh

[jccurtis]

Thanks @vsoch! I'll hopefully give this a shot today. I think the reason your exec echo commands didn't work is because the variable is evaluated on the host before being passed to the image.

I'll be on travel for the next 2 weeks so my apologies for any delayed responses...

[jccurtis]

Hi @vsoch, I'm back from travel and determined to get this working. I added a symlink during %post after the megalib installation:

BootStrap: debootstrap
OSVersion: xenial
MirrorURL: http://us.archive.ubuntu.com/ubuntu/

%labels
AUTHOR [email protected]

%post
    apt-get update
    apt-get upgrade
    apt-get -y --force-yes install curl wget vim git dpkg-dev make cmake g++ gcc binutils ccache libx11-dev libxpm-dev libxft-dev libxext-dev gfortran libssl-dev libpcre3-dev libglu1-mesa-dev libgl1-mesa-dev libglew-dev libmysqlclient-dev libfftw3-dev graphviz-dev libavahi-compat-libdnssd-dev libldap2-dev python-dev libxml2-dev libkrb5-dev libgsl-dev libqt4-dev libpng-dev libjpeg-dev g++-multilib libc6-dev-i386 libxmu-dev libxmu-headers libxt-dev
    cd /
    git clone https://github.com/jccurtis/megalib /opt/megalib
    cd /opt/megalib
    bash setup.sh --release=dev --patch=yes --clean=yes --megalib-path=/opt/megalib
    chmod u+x /opt/megalib/bin/source-megalib.sh
    ln -s /opt/megalib/bin/source-megalib.sh /.singularity.d/env/80-source-megalib.sh
    cd /opt
    wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /opt/miniconda.sh
    bash /opt/miniconda.sh -b -p /opt/miniconda

%environment
    export PATH="/opt/miniconda/bin:$PATH"

The issue I am having now is that /.singularity.d/actions/exec runs with sh and not bash. I tried adding the shebang #!/bin/bash to the file in question: /opt/megalib/bin/source-megalib.sh but it still has issues with bash specific commands (source, [[ in if statements, etc). Any thoughts to resolve the bash vs sh sourcing? Thanks!

[vsoch]

hey @jccurtis !

What I had in mind was to write a script that is called by shell that calls bash to the first env file. Like this:

Bootstrap: docker
From: ubuntu:16.04

%labels
AUTHOR [email protected]

%setup
mkdir -p $SINGULARITY_ROOTS/.singularity.d/env
echo "MEGALIBDIR=/opt/megalib
ROOTDIR=/opt/megalib/external/root_v6.08.06
GEANT4DIR=/opt/megalib/external/geant4_v10.02.p03
bash /opt/megalib/config/env.sh --root=${ROOTDIR} --geant4=${GEANT4DIR} --megalib=${MEGALIBDIR}" >> $SINGULARITY_ROOTFS/.singularity.d/env/80-custom.sh
chmod u+x $SINGULARITY_ROOTFS/.singularity.d/env/80-custom.sh

%post
apt-get update
apt-get -y --force-yes install curl wget vim git dpkg-dev make cmake g++ gcc binutils ccache libx11-dev libxpm-dev libxft-dev libxext-dev gfortran libssl-dev libpcre3-dev libglu1-mesa-dev libgl1-mesa-dev libglew-dev libmysqlclient-dev libfftw3-dev graphviz-dev libavahi-compat-libdnssd-dev libldap2-dev python-dev libxml2-dev libkrb5-dev libgsl-dev libqt4-dev libpng-dev libjpeg-dev g++-multilib libc6-dev-i386 libxmu-dev libxmu-headers libxt-dev
cd /
git clone https://github.com/jccurtis/megalib /opt/megalib
cd /opt/megalib
bash setup.sh --release=dev --patch=yes --clean=yes --megalib-path=/opt/megalib
chmod u+x /opt/megalib/bin/source-megalib.sh    
cd /opt
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /opt/miniconda.sh
bash /opt/miniconda.sh -b -p /opt/miniconda

%environment
PATH="/opt/miniconda/bin:$PATH"
export PATH

so the env/80-custom.sh looks like this:

cat /.singularity.d/env/80-custom.sh 
MEGALIBDIR=/opt/megalib
ROOTDIR=/opt/megalib/external/root_v6.08.06
GEANT4DIR=/opt/megalib/external/geant4_v10.02.p03
bash /opt/megalib/config/env.sh --root=${ROOTDIR} --geant4=${GEANT4DIR} --megalib=${MEGALIBDIR}

and to make

singularity create --size 8000 mega.img
sudo singularity bootstrap mega.img Singularity

I would generally try to avoid nested layers of scripts like this with bash/sh expectations given not knowing how they will be called, but this might do the trick. it takes a REALLY long time to compile so I haven't tried this again from source, but I'm starting a build now and will see if it works without a hitch later.

[GodloveD]

Hey @jccurtis have you tried using the following syntax?

. /file/to/source

It should be supported by the bourne shell.

[vsoch]

I did try that in the example above (in the custom environment script) and it didn't work.

[jccurtis]

Full Disclaimer: I'm not an expert on unix shells but my understanding is as follows:

Hi @GodloveD, . sources the script with the current shell (in this case bourne/sh) and ignores the shebang line (link). This causes issues because /opt/megalib/bin/source-megalib.sh ends up sourcing scripts with bash specific commands (like [[ in if statements). Thus a syntax error is thrown 🙁

@vsoch, calling bash doesn't thrown an error because the script is called with bash in a new shell and exits successfully. The PATH and other VAR's are set within the new shell but not within the singularity exec environment where the bash command is called. The functionality would be to set the shell that singularity exec uses so the env scripts can be called like .<env>rc.

I'm making a workaround by digging through the env.sh script to figure out the PATH and other VAR's to set them with export in the %environment section. The PATHS are set in a particular order to make three software packages (geant4, root and megalib) work together. There are a bunch of options the default configuration doesn't use so I don't think the manual export's will be difficult.

Another work around would be to print the variables from the bash program and then pipe the output to sh so that the env is setup correctly. This would be prone to bugs and the execution of potentially insecure cmds so I wouldn't be keen to follow this path.

[vsoch]

I would take an approach that views the application as part of the image, and unwrap all the extra code to check the base os (we know what it is) and just export the needed variables in the %environment section.

[bermanmaxim]

I had a similar issue: my base docker image defines environment variables that I need to install pip modules during %post.

I took the approach of sourcing the environment scripts inside %post:

%post
for script in /.singularity.d/env/*.sh; do
    if [ -f "$script" ]; then
        . "$script"
    fi
done
pip install scikit-cuda
...

[chaitanyadgore]

How I can get singularity container on Ubantu 20.04. I tried this command in termial
$ sudo apt-get install singularity-container
I am facing following error.
E: Package 'singularity-container' has no installation candidate