Singularity: LD_LIBRARY_PATH not correct when using a converted Docker image

[samcmill]

Version of Singularity:

3.1.1

Expected behavior

Environment inside the container reflects the container environment.

Actual behavior

Environment inside the container reflects the container environment except for LD_LIBRARY_PATH, which mirrors the host environment.

Steps to reproduce behavior

Use this simple Dockerfile to build a Docker container image:

FROM centos:7

ENV FOO=foo
ENV PATH=/foo/bin:$PATH
ENV LD_LIBRARY_PATH=/foo/lib:$LD_LIBRARY_PATH

sudo docker build -t foo -f Dockerfile .

Convert the Docker container image into a Singularity image: singularity build foo.sif docker-daemon://foo:latest

Check the environment inside the container:

$ echo $LD_LIBRARY_PATH
/bar/lib
$ singularity exec foo.sif printenv FOO PATH LD_LIBRARY_PATH
foo
/foo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/bar/lib:/.singularity.d/libs

LD_LIBRARY_PATH inside the container contains /bar/lib, which is the value on the host, rather than /foo/lib which is the value defined in the Dockerfile. Note that PATH and FOO are correct (match the Dockerfile).

If LD_LIBRARY_PATH is unset on the host, then the container environment is correct:

$ unset LD_LIBRARY_PATH
$ singularity exec foo.sif printenv FOO PATH LD_LIBRARY_PATH
foo
/foo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/foo/lib::/.singularity.d/libs

If the container image is converted using the singularityware/docker2singularity container, then the behavior is also correct:

$ sudo docker run -t --rm --cap-add SYS_ADMIN -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/output singularityware/docker2singularity foo:latest
$ echo $LD_LIBRARY_PATH
/bar/lib
$ singularity exec foo.simg printenv FOO PATH LD_LIBRARY_PATH
foo
/foo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/foo/lib::/.singularity.d/libs

When using --cleanenv the behavior is also correct:

# echo $LD_LIBRARY_PATH
/bar/lib
root@c60a4a54666d:/tmp# singularity exec --cleanenv foo.sif printenv FOO PATH LD_LIBRARY_PATH
foo
/foo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/foo/lib::/.singularity.d/libs

[cclerget]

Hi @samcmill , will look at the issue, this is actually due to a change some month ago to solve another issue.

[samcmill]

Version 3.2.0 is impacted as well.

[bilke]

I stumbled on this as well. The change was introduced here: #2669.

The variable contents are replaced when the variable already exist on the host. See /.singularity.d/env/10-docker2singularity.sh in the container:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH:-"/usr/local/petsc/lib:/usr/local/vtk/lib:/usr/local/boost/lib:/usr/local/openmpi/lib:/usr/local/ucx/lib:"}

The minus (-"/usr/local/...) is a substitution operator.

@samcmill My advice for the moment is to don't rely on LD_LIBRARY_PATH inside the container. Passing from outside is necessary for interoperability with e.g. OpenMPI. Use the ldconfig option in hpccm!

@cclerget Maybe it would be better to have an option to merge LD_LIBRARY_PATH from outside with the one from inside the container?

[samcmill]

@cclerget I don't understand why the resolution to https://github.com/sylabs/singularity/issues/2668 wasn't SINGULARITYENV?

$ SINGULARITYENV_LISTEN=3334 singularity exec docker://eggdrop printenv LISTEN
3334

Merging environment variables between the host and the container seems like it would be very difficult and not intuitive. I think the principle of least astonishment would say that

1. variables defined in the container should have precedence when conflicting with the host
2. host variables without an analogue in the container should be passed through as is
3. a mechanism is available to override 1 if the user specifically requests it (e.g., SINGULARITYENV).

[AdamSimpson]

@samcmill My advice for the moment is to don't rely on LD_LIBRARY_PATH inside the container. Passing from outside is necessary for interoperability with e.g. OpenMPI. Use the ldconfig option in hpccm!

Is there actually a situation that results in the host LD_LIBRARY_PATH(or PATH) being valid in the container? Host libraries passed into the container environment are dumped into /.singularity.d/libs, as far as I'm aware.

[cclerget]

@samcmill Before the patch of #2669, it wasn't possible to override like you did, it works now because you used a patched version :). Because SINGULARITYENV_ is trimmed to become LISTEN and Singularity know it must pass it to container when used in conjunction with -e flag. Without -e flag LISTEN and SINGULARITYENV_LISTEN are finally the same LISTEN variable inside container, so there is no way actually to distinguish them when env files in /.singularity.d/env/ are evaluated at execution.

@bilke Unfortunately there is no simple way to do that actually because the image must be mounted and env files must be evaluated to determine the value of environment variables set in container image.

The immediate solution could be to left untouch SINGULARITYENV_ and pass them to shell/exec/run scripts, but a reliable solution solution in the long term would be to embed container environment variables in image metadata that could be read directly, to determine early what environment variables could be overridden.

[sashayakovtseva]

Hello @samcmill,
Looks like this one is fixed in #4398 and then #4412
Can you confirm and close the issue if it is not relevant any more? Thanks :)

[samcmill]

@sashayakovtseva I confirmed this is fixed in master.