Singularity: Setting a default Conda environment

[JonN17]

Version of Singularity:

What version of Singularity are you using? Run:

$ singularity version
2.6.1-dist

Expected behavior

What did you expect to see when you do...?

Two things.
1) Convert a docker file and have the conda environment running by default. Default conda environment works in just fine in docker but does not work in converted singularity image.

2) Build a singularity image from singularity recipe and be able to set a conda environment as the default.

Actual behavior

Specified / Preferred Conda environment works but isn't running by default.

What actually happend? Why was it incorrect?

Specified / Preferred Conda environment should be running by default.

Steps to reproduce this behavior

How can others reproduce this issue/problem?

Just try to set a default conda environment. Or convert a docker file with a default conda environment and watch it no longer be default.

I have a docker file that succesfully activates a conda environment when I shell into it.
When I convert that docker container file over to singularity it no longer activates that environment.
So the preferred feature is lost in conversion.

Since the conversion of docker image > singularity image failed to work I tried creating
a singularity image from a recipe.

All 3 of these methods were put into %environment, %runscript and %post for 9 different combinations.
. activate myenv
source activate myenv
conda activate myenv

All 3 of these methods were put into %environment, %runscript and %post for 9 different combinations.
exec . activate myenv
exec source activate myenv
exec conda activate myenv

There is a simple script in the attached tar.gz:

SINGULARITY_ONLY_ATTEMPTS/configureAndBuildContainers.sh

That is easy to configure and I used to create all of these combinations.

Both of these methods were put into %environment, %runscript and %post for 6 different combinations.

echo "source /usr/local/miniconda3/etc/profile.d/conda.sh" >> $SINGULARITY_ENVIRONMENT
echo "source activate myenv" >> $SINGULARITY_ENVIRONMENT

echo ". /usr/local/miniconda3/etc/profile.d/conda.sh" >> $SINGULARITY_ENVIRONMENT
echo ". activate myenv" >> $SINGULARITY_ENVIRONMENT

Multiple different ways of using an RC file
echo "" >> /.singularity_bashrc
echo ". activate myenv" >> /.singularity_bashrc
echo "source activate myenv" >> /.singularity_bashrc
echo "conda activate myenv" >> /.singularity_bashrc

OCI_CMD='/bin/bash'
OCI_ARGS='-rcfile /.singularity_bash'
SINGULARITY_OCI_RUN="${OCI_CMD} ${OCI_ARGS}"
exec "$SINGULARITY_OCI_RUN"

Recipes for both docker and singularity are attached. While I may be making a mistake with the singularity recipe, the docker conversion clearly is not working properly.

DEBUG.tar.gz

What OS/distro are you running

Ubuntu 18.04 Bionic

$ cat /etc/os-release

How did you install Singularity

Probably your standard apt-get method highlighted here:
https://singularity.lbl.gov/install-linux

I have a CentOS install with similar issues that also followed the documentation for install.

Write here how you installed Singularity. Eg. RPM, source.

Apt-get for ubuntu, source for CentOS. most of my work is done on ubuntu and I try to reproduce on CentOS.

[dtrudg]

Hello - the open-source release of Singularity 2.x is no longer supported. Some environment variable etc. handling and docker conversion has changed in 3.x so please could you try out your methods in 3.5 and we can then advise.

[JonN17]

singularity now at version 3.5.3 and we have the same issue.

Problem persists

Docker Conversion
When converting docker image to singularity image I still lose my conda environment settings. I have to manually activate the conda environment.

Recipe Method

All 3 of these methods were put into %environment, %runscript and %post for 9 different combinations.
. activate myenv
source activate myenv
conda activate myenv

This method was put into %environment, %runscript and %post for 3 different combinations.

echo "source /usr/local/miniconda3/etc/profile.d/conda.sh" >> $SINGULARITY_ENVIRONMENT
echo "source activate myenv" >> $SINGULARITY_ENVIRONMENT

the above was changed to ". activate myenv" for %post only variation as "source not found" was displayed.

And i tried playing with various versions of the below:

%runscript
exec echo "This is an Ubuntu Container with Miniconda with ENV!!"
OCI_CMD='/bin/bash'
OCI_ARGS='-rcfile /.singularity_bash'
SINGULARITY_OCI_RUN="${OCI_CMD} ${OCI_ARGS}"
exec "$SINGULARITY_OCI_RUN"

%post
echo "" >> /.singularity_bashrc
echo "## Activate rnaseq environment" >> /.singularity_bashrc
echo "conda activate myenv" >> /.singularity_bashrc

Thank you in advance for your help.

[cclerget]

Hi @JonN17 , there is an issue with the runscript :

%runscript
exec echo "This is an Ubuntu Container with Miniconda with ENV!!"

Everything after the first exec statement won't be executed, try with :

%runscript
    echo "This is an Ubuntu Container with Miniconda with ENV!!"
    exec /bin/bash --noprofile --init-file /.singularity_bash "$@"

Now if you want to force something when using singularity shell comand you could add to the environment :

%environment
    action="${0##*/}"
    if [ "$action" == "shell" ]; then
        if [ "${SINGULARITY_SHELL:-}" == "/bin/bash" ]; then
            set -- --noprofile --init-file /.singularity_bash
        elif test -z "${SINGULARITY_SHELL:-}"; then
            export SINGULARITY_SHELL=/bin/bash
            set -- --noprofile --init-file /.singularity_bash
        fi
    fi

%post
    echo "## Activate rnaseq environment" >> /.singularity_bashrc
    echo "conda activate myenv" >> /.singularity_bashrc

%runscript
    echo "This is an Ubuntu Container with Miniconda with ENV!!"
    exec /bin/bash --noprofile --init-file /.singularity_bash "$@"

With the above both singularity shell and singularity run commands should source conda environment automatically.

[JonN17]

Thank you so much for the feedback, unfortunately I get the following error:
Error:
/.singularity.d/actions/shell: 12: [: shell: unexpected operator

Recipe used:
```
Bootstrap: localimage
From: UbuntuMinicondaClean.simg

%environment
action="${0##*/}"
if [ "$action" == "shell" ]; then
if [ "${SINGULARITY_SHELL:-}" == "/bin/bash" ]; then
set -- --noprofile --init-file /.singularity_bash
elif test -z "${SINGULARITY_SHELL:-}"; then
export SINGULARITY_SHELL=/bin/bash
set -- --noprofile --init-file /.singularity_bash
fi
fi

%post
echo "## Activate rnaseq environment" >> /.singularity_bashrc
echo "conda activate myenv" >> /.singularity_bashrc

%runscript
echo "This is an Ubuntu Container with Miniconda with ENV!!"
exec /bin/bash --noprofile --init-file /.singularity_bash "$@"
````
EZ-Troubleshooting:
I've included both recipes in the attached tar.gz file. Just extract and run "buildBoth.sh" to build both of the singularity images. Takes about 2 minutes.

ez-package.tar.gz

[JonN17]

I'm trying to fix the error I get with your suggestion. Normally this error:

[: shell: unexpected operator

Means something is running in the "sh" environment not the "bash" environment. Sh has a less extensive syntax. So I tried modifying your suggested solution by adding "#!/bin/bash" in multiple places. I ran several variations to no avail. Below is just one of multiple versions. Again, this does not work.

```Bootstrap: localimage
From: UbuntuMinicondaClean.simg

%environment
#!/bin/bash
action="${0##*/}"
if [ "$action" == "shell" ]; then
if [ "${SINGULARITY_SHELL:-}" == "/bin/bash" ]; then
set -- --noprofile --init-file /.singularity_bash
elif test -z "${SINGULARITY_SHELL:-}"; then
export SINGULARITY_SHELL=/bin/bash
set -- --noprofile --init-file /.singularity_bash
fi
fi

%post
echo "#!/bin/bash" >> /.singularity_bashrc
echo "## Activate rnaseq environment" >> /.singularity_bashrc
echo "conda activate myenv" >> /.singularity_bashrc

%runscript
echo "This is an Ubuntu Container with Miniconda with ENV!!"
exec /bin/bash --noprofile --init-file /.singularity_bash "$@"```

[JonN17]

So removing the if statement completely gets rid of the errors but fails to activate the environment. I've tried this:

%environment
            set -- --noprofile --init-file /.singularity_bash

As well as this:

%environment
            export SINGULARITY_SHELL=/bin/bash
            set -- --noprofile --init-file /.singularity_bash

And neither works activates the environment when I shell in.

Any suggestions? Help?

[jmstover]

You have to use this in your $HOME/.bashrc or wherever to activate conda. Pretty sure it's exactly the same as with OpenFOAM... functions/aliases/etc... do not get shared down. In your ~/.bashrc have something like:

if [ -n "$SINGULARITY_CONTAINER" ]; then 
  echo "In container";
fi

In that if block, put your conda command. Then start it up with:

singularity shell --shell /bin/bash myimage.sif

Edit:
You can get more exact with checking the value of SINGULARITY_NAME or SINGULARITY_CONTAINER, etc... whatever works for you.

If were were wanting a runscript you could put the conda command in the %runscript section... and it would be loaded correctly. But for shell you're working against some shell security stuff in not passing everything that's defined down the chain.

[JonN17]

I converted the bash version of your "if statement" to a little more classical sh shell friendly version. It still fails to activate the environment. If I type:

conda activate myenv

it works just fine. All relevant files are attached above so you can reproduce what I did easily.

See the DEBUG.tar.gz attached above.

Your suggested solution converted from bash to sh compatibility below. I simply converted "==" to "=" in vim by typing:
%s/==/=/g

Recipe:

Bootstrap: localimage
From: UbuntuMinicondaClean.simg


%environment
    action="${0##*/}"
    if [ "$action" = "shell" ]; then
        if [ "${SINGULARITY_SHELL:-}" = "/bin/bash" ]; then
            set -- --noprofile --init-file /.singularity_bash
        elif test -z "${SINGULARITY_SHELL:-}"; then
            export SINGULARITY_SHELL=/bin/bash
            set -- --noprofile --init-file /.singularity_bash
        fi
    fi

%post
    echo "## Activate rnaseq environment" >> /.singularity_bashrc
    echo "conda activate myenv" >> /.singularity_bashrc

%runscript
    echo "This is an Ubuntu Container with Miniconda with ENV!!"
    exec /bin/bash --noprofile --init-file /.singularity_bash "$@"

[jmstover]

I have to type "exit" twice now to exit the container.

Then the conda command is dropping you into a subshell yes? So you do "conda activate myenv" and you're in a subshell that is using myenv ... That's what that reads like to me.

Is there a way you can make conda verbose like running: sh -x ??

~And how is the UbuntuMinicondaClean.simg file built?~ -- Ignore

[cclerget]

@JonN17 Forgot to add the conda profile script, here the updated recipe :

Bootstrap: localimage
From: UbuntuMinicondaClean.simg

%environment
    action="${0##*/}"
    if [ "$action" = "shell" ]; then
        if [ "${SINGULARITY_SHELL:-}" = "/bin/bash" ]; then
            set -- --noprofile --init-file /.singularity_bash
        elif test -z "${SINGULARITY_SHELL:-}"; then
            export SINGULARITY_SHELL=/bin/bash
            set -- --noprofile --init-file /.singularity_bash
        fi
    fi

%post
    echo "## Activate rnaseq environment" >> /.singularity_bash
    echo "source /usr/local/miniconda3/etc/profile.d/conda.sh" >> /.singularity_bash
    echo "conda activate myenv" >> /.singularity_bash

%runscript
    echo "This is an Ubuntu Container with Miniconda with ENV!!"
    exec /bin/bash --noprofile --init-file /.singularity_bash "$@"

[JonN17]

I greatly appreciate the feedback. Unfortunately, the updated solution still doesn't work. I'm trying to figure out how to make conda verbose.

Feel free to download the attached tar.gz and simply run

./buildAll.sh

That will automatically compile everything from scratch in 2 minutes.

Layer Calling Anaconda Environment:

Bootstrap: localimage
From: UbuntuMinicondaClean.simg

%environment
    action="${1##*/}"
    if [ "$action" = "shell" ]; then
        if [ "${SINGULARITY_SHELL:-}" = "/bin/bash" ]; then
            set -- --noprofile --init-file /.singularity_bash
        elif test -z "${SINGULARITY_SHELL:-}"; then
            export SINGULARITY_SHELL=/bin/bash
            set -- --noprofile --init-file /.singularity_bash
        fi
    fi

%post
    echo "## Activate rnaseq environment" >> /.singularity_bash
    echo "source /usr/local/miniconda4/etc/profile.d/conda.sh" >> /.singularity_bash
    echo "conda activate myenv" >> /.singularity_bash

%runscript
    echo "This is an Ubuntu Container with Miniconda with ENV!!"
    exec /bin/bash --noprofile --init-file /.singularity_bash "$@"

Base Layer:

Bootstrap: docker
From: ubuntu

%runscript
    exec echo "This is an Ubuntu Container with Miniconda!!"

%files

%environment
   CONDA_INSTALL_PATH="/usr/local/miniconda3"
   CONDA_BIN_PATH="/usr/local/miniconda3/bin"
   export PATH="$CONDA_BIN_PATH:$PATH"

%labels

%post
   echo "The post section is where you can install, and configure your container."
   # Installing commonly used apps
   apt-get update && apt-get -y install python3 git wget htop vim python-pip 
   # Making Data Directory
   mkdir -p /BIFICS
   # download and install Anaconda
   wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
   chmod +x Miniconda3-latest-Linux-x86_64.sh
   CONDA_INSTALL_PATH="/usr/local/miniconda3"
   ./Miniconda3-latest-Linux-x86_64.sh -b -p $CONDA_INSTALL_PATH
   export PATH="/usr/local/miniconda3/bin:$PATH" # can't create environments without this
   which conda
   ls $CONDA_INSTALL_PATH | head
   #conda update -n base -c defaults conda # this breaks and I don't know why.  "which conda" works fine.
   # Installing conda environments
   conda create --name myenv

%runScript

latest-2020-03-04.tar.gz

[cclerget]

@JonN17 Looks like it works for shell command only, you can check this once in the container with shell command that CONDA_ environment variables are set, it doesn't show you the expected prompt but seems to work

[JonN17]

Got it working. Thank you so much!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I really appreciate this!

[cclerget]

Great ! Closing the issue, feel free to ping to re-open it if needed.

[mestia]

@JonN17 would you mind posting the working solution?
Thank you

[wumpus]

A lot of HPC people use conda -- this bug report is the best documentation I could find on using Singularity + conda. Not the easiest to follow, however! Is it possible for this to become part of the formal documentation? @cclerget