Singularity: environment variables with spaces incorrectly interpreted

[gkiar]

When I provide an environment variable to singularity exec things work as expected -- until there is a space in the variable value. In that case, Singularity chokes and tells me that the characters following the space are "invalid variable names" rather than continuing to recognize them as part of the value.

Version of Singularity:

$ singularity version
singularity version 3.5.3-1.el7

Expected behavior

$ SINGULARITYENV_MY_VAR="libinterflop.so -f arg1" singularity exec --cleanenv my-image.simg sh -c 
 'echo ${MYVAR}'
libinterflop.so -f arg1

Actual behavior

$ SINGULARITYENV_MY_VAR="libinterflop.so -f arg1" singularity exec --cleanenv my-image.simg sh -c 
 'echo ${MYVAR}'
/.singularity.d/actions/exec: 11: export: -f: bad variable name

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[dtrudg]

@gkiar - we'll have to look into this some more. What is the source of the container image, or does this occur with all?

For me there is no error:

$ SINGULARITYENV_MY_VAR="libinterflop.so -f arg1" singularity exec --cleanenv alpine_latest.sif echo ${MY_VAR}

Note that the value of MY_VAR doesn't show, as ${MY_VAR} is interpreted by the host shell outside of the container, but this will work:

$ SINGULARITYENV_MY_VAR="libinterflop.so -f arg1" singularity exec --cleanenv alpine_latest.sif sh -c 'echo ${MY_VAR}'
libinterflop.so -f arg1

[gkiar]

Hey @dctrud - thanks for the quick response! The container I'm using can be built with:

singularity pull docker://gkiar/dipy_tracking:v0.4.0

And the specific command/environment variable I used to find this issue is as follows:

$ SINGULARITYENV_VFC_BACKENDS="libinterflop_mca.so -m rr" singularity exec --cleanenv /home/gkiar/images/dipy_tracking_v0.4.0.sif python3.6 -c "print(sum([0.0001 for _ in range(1000)]))"
/.singularity.d/actions/exec: 11: export: -m: bad variable name

[prioux]

Is that docker image public? I can't pull it to test.

Wild guess: the singularity init scripts in /.singularity.d that are built for docker images have a little bug that doesn't trigger for other singularity images.

[gkiar]

Oops - apologies, there was a typo in my container pull command; it's an _ not a -. I've updated the comment. Thanks!

[prioux]

Found the bug...

[gkiar]

Also, for reference, the expected behaviour is observed when using docker:

$ docker run -ti -e VFC_BACKENDS="libinterflop_mca.so -m rr" --entrypoint python3.6 gkiar/dipy_tracking:v0.4.0 -c "print(sum([0.0001 for _ in range(1000)]))"
python3.6: verificarlo loaded backend libinterflop_mca.so
python3.6: interflop_mca: loaded backend with precision = 53 and mode = rr
0.10000000000000014

(Note, if you run it your number will be slightly different than mine)

[prioux]

When pulling from docker, singularity creates /.singularity.d/10-docker2singularity.sh which contains the assignments of environment variables that are defined by the docker container. It looks like this in your case

#!/bin/sh
export PATH="/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
export LIBRARY_PATH=${LIBRARY_PATH:-"/usr/lib/gcc/x86_64-linux-gnu/4.7:/usr/lib/llvm-3.5/lib:"}
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH:-"/usr/local/lib:"}
export PYTHONPATH=${PYTHONPATH:-"/usr/local/lib/python3.4/site-packages/:"}
export CC=${CC:-"verificarlo --exclude-file /tmp/python-vfc-exclude.txt"}
export CXX=${CXX:-"verificarlo"}
export FC=${FC:-"verificarlo"}
export LD=${LD:-"verificarlo"}
export LDSHARED=${LDSHARED:-"verificarlo -shared"}
export VFC_BACKENDS=${VFC_BACKENDS:-"libinterflop_mca.so -m rr"}

That script is meant to set all variables as set in docker, with the option to overrides any of them from the launch environment of the singularity container.

One of your variables in docker contains spaces, which is OK (last line). But if you supply it on the command-line, it gets replaced in the statement, which becomes:

export VFC_BACKENDS=libinterflop_mca.so -m rr

Notice the missing quotes here...

[prioux]

I think I can find a workaround using quotes inside the value...

[gkiar]

Thanks, @prioux ! I suspected it was something like that but was less successful at sleuthing than you. I have played with many different variable quote sequences, including wrapping the contents of " quotes with escaped \' quotes, etc..., all to no avail, but perhaps you'll have more success there as well. Thanks for your help!

[prioux]

Actually it's hard... it seems the "/bin/sh" use by singularity might be a bit trickier than standard bash?

[prioux]

I haven't found a quoting solution yet.

[dtrudg]

@prioux - /bin/sh varies in different operating systems, and hence different containers. Assume no bashisms, only standard POSIX sh behaviour. I'm busy elsewhere right now, but @gkiar try just escaping the spaces:

"libinterflop_mca.so\ -m\ rr"

[gkiar]

@dctrud no dice with escaping the spaces as you suggested. Will report back if I find some escape pattern that works...

[prioux]

It seems the problem is in the export statement itself. exporting the bariables after substituting their values work fine. Here's an example in a singularity container, after invoking /bin/sh to be in the same limited POSIX shell:

Singularity> export A="a -b -c"
Singularity> export B=${A:-"x -y -z"}
/bin/sh: 44: export: -b: bad variable name
Singularity> B=${A:-"x -y -z"}         
Singularity> export B
Singularity> echo $B
a -b -c

[prioux]

(deleted a message, example was bad, sorry)

[prioux]

I think the easiest solution is to make sure double quotes are around the values everywhere:

export VFC_BACKENDS="${VFC_BACKENDS:-"libinterflop_mca.so -m rr"}"

I checked, it seems to work even when the values provided at run time have double quotes in them. And tt works even if the constant value inside is also delimited with double quotes.

[prioux]

(That was a singularity patch I was suggesting... no workaround on the command line yet, because export X=${Y:-Z} is broken...)

[prioux]

Ahhhrrgh!

OK, so I think I know what's happening. As a sysadmin I've been bitten by this before. The /bin/sh in your container, @gkiar, happens to be "/bin/dash". dash is known to have subtle problems. Did you build this docker container out of ubuntu? Then in your Dockerfile, just add a few directives to replace the symlink "/bin/sh" which points to dash so that it points instead to "/bin/bash".

After much checking, I discovered that bash does all the variable substitutions perfectly fine.

[prioux]

Yep, @gkiar, your container is built out of ... wait for it ... 'Ubuntu 14.04'. That's 6 years old and at the time dash was still very much used back then.

[gkiar]

Thanks @prioux - I've rebuilt my docker container with your suggestion (ln -sf /bin/bash /bin/sh) and verified that using sh before this replicated the error and following it the error was resolved. I am pulling+rebuilding it with Singularity now and will comment when I know how that turns out.

It's still a bug from the singularity side of things, but this would be a passable workaround for me to continue moving along with my work 🙂

[gkiar]

To follow up @dctrud @prioux : the Singularity build-from-docker with my replaced /bin/sh did work, in case others are stuck on an older version of Singularity before their institution installs the patched version from #5110

Thanks!!
Greg