Nix: Any way for LD_LIBRARY_PATH to *not* break nix-env's absolute paths for dynamic libs

[rrnewton]

Maybe this is naive of me, but I thought there was some way on linux to link a dynamic library with an absolute path which would not be affected by LD_LIBRARY_PATH.

I'm on an ubuntu 14.04 system on a shared account where, for whatever reason, LD_LIBRARY was set to include some extra directories. That causes nix-env to segfault:

$ nix-env -q
Segmentation fault (core dumped)
$ ldd `which  nix-env`
        linux-vdso.so.1 =>  (0x00007ffde2da7000)
        libnixexpr.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixexpr.so (0x00007f72a54ac000)
        libgc.so.1 => /nix/store/f24zx1r39kalz01q9kw7zcg1ngj7w2db-boehm-gc-7.2f/lib/libgc.so.1 (0x00007f72a5243000)
        libnixmain.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixmain.so (0x00007f72a5031000)
        libnixstore.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixstore.so (0x00007f72a4d6f000)
        libnixutil.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixutil.so (0x00007f72a4b3d000)
        libnixformat.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixformat.so (0x00007f72a4933000)
        libstdc++.so.6 => /nix/store/kxac9yv62zff2pj6k1j4d6hiz8jdi2f3-gcc-5.3.0-lib/lib/libstdc++.so.6 (0x00007f72a45b9000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f72a42b3000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f72a409c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f72a3cd7000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f72a3ad3000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f72a38b5000)
        /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x0000$f72a573a000)
        libsqlite3.so.0 => /nix/store/1w64w28j9vz57g2k5bivh2wpk8q9j6vj-sqlite-3.12.2/lib/libsqlite3.so.0 (0x00007f72a35e0000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f72a33d0000)
        libcurl.so.4 => /nix/store/9whcgafma7473b4xpyb88hbhi8w5414z-curl-7.47.1/lib/libcurl.so.4 (0x00007f72a3160000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f72a2f3e000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f72a2b62000)
        libnghttp2.so.14 => /nix/store/vv2n25vgck007mjph2ay8g4ab58igzp2-nghttp2-1.9.2-lib/lib/libnghttp2.so.14 (0x00007f72a293$000)
        libssh2.so.1 => /nix/store/y8jcz5g52265i18kr63d6lim4k1d7jgn-libssh2-1.7.0/lib/libssh2.so.1 (0x00007f72a2711000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f72a24b2000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f72a2299000)

Whereas simply unsetting LD_LIBRARY_PATH makes nix-env run fine:

$ LD_LIBRARY_PATH= nix-env -q
bash-4.3-p42

That's an easy fix, sure. But shouldn't nix-env be robust against LD_LIBRARY_PATH if possible?

[rrnewton]

I'm not even sure what it means when ldd reports <absolute path> => <absolute path> in its output. That's what it does for ld-linux-x86-64.so.

Likewise, here's the ldd printout for bash, this time without any LD_LIBRARY_PATH shenanigans:

   $ ldd /nix/store/d20f169ryps7ds2qak0r5n1f4hhxr80h-bash-4.3-p42/bin/bash
    linux-vdso.so.1 =>  (0x00007fffa6a72000)
    libdl.so.2 => /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/libdl.so.2 (0x00007f0bb1705000)
    libc.so.6 => /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/libc.so.6 (0x00007f0bb1363000)
    /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f0bb1909000)

readelf -d makes it look like libc.so is in fact a bare name, not an absolute path:

Dynamic section at offset 0xb0020 contains 26 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

What I wanted to see was the nix binary using only its own ld-linux.so and pointing to its own libs exclusively, with absolute paths. Since bash is a core package built from source, it should specify these linker options in the derivation (and not need PatchElf), right?

[vcunat]

This belongs to nixpkgs, really, but let's leave it here.

The way we've always done this:

• we set the loader to an absolute path (that's the ld-linux*.so*),
• the usual short so-names are used, e.g. libdl.so.2, and they're searched in a list of absolute RPATHs coded into the ELFs;
• $LD_LIBRARY_PATH can be used to add directories where to search before RPATH (IIRC we use a nixpkgs-specific patch for the loader to change the priority here).

Recently there was a suggestion on ML to use absolute-path so-names instead of RPATH, but I've seen no code yet and I suspect that way might be (significantly) more complicated to implement than what we're doing nowadays.

[rrnewton]

Ok, so it sounds like it's a known problem that LD_LIBRARY_PATH can break executables in /nix/store.

I spent some time reading about rpath, but I wasn't having luck controlling rpath or the loader directly from my own nix build (where I'm going through gcc-wrapper).

If we're always running a custom load, what the heck does it mean that ld-linux-x86-64.so claims to point to an Ubuntu system version under /lib64? E.g.:

    /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f0bb1909000)

[vcunat]

If we're always running a custom load, what the heck does it mean that ld-linux-x86-64.so claims to point to an Ubuntu system version under /lib64?

I'd think that ldd shows this one wrong. IIRC running ldd from a different Linux can show information different from what really happens during running. Best use ldd from nixpkgs.

[copumpkin]

@rrnewton just because it's known doesn't mean it's good or that we want it to persist. Is there another ticket tracking this issue? If not, I'd just leave it open.

[vcunat]

There are some places where we rely on $LD_LIBRARY_PATH. We would have to adapt those.

$ git grep LD_LIBRARY_PATH | wc -l
271

[rrnewton]

@copumpkin, yes indeed it would be great to fix it. I really want to be able to tell people "run this command; it's nix so I can guarantee it will do the same thing for you as for me". But in a multi-user environment sitting on top of another distro... we just haven't fully been able to realize that dream. This is one the things we trip over.

[vcunat]

I don't think distros normally define $LD_LIBRARY_PATH by default. (Well, NixOS does.) On non-NixOS there's also often problem with libGL, which is related.

Really, it's difficult to balance what the user wants there, e.g. we've been arguing whether to use /etc/foo from the host OS – IMHO basically noone wants nix stuff _completely_ disregard config from the host machine (includes also ~/.* and env vars).

[rrnewton]

Yeah, but on shared machines where people can't install in /usr then users end up with libraries installed in their home directories and then they set their LD_LIBRARY_PATH to make it work. This is probably a terrible thing. But there have to be some non-zero number of people in the world who set an LD_LIBRARY_PATH in their bashrc or profile.

I'm not sure what other people want regarding the isolation of nix on other distros, but whenever we've "mixed" the two worlds, things go south really fast. (E.g. install a compiler via nix and build something outside of nix with it.)

[rrnewton]

I'm perfectly happy if the manual is just updated to say loudly and clearly "unset LD_LIBRARY_PATH before calling any Nix-built binary".

[vcunat]

Yeah, we could document that.

[amosbird]

Can I add a post install hook that change all RUNPATH to RPATH? I have customized toolchain that relies on LD_LIBRARY_PATH while I still want to use binaries from nix-env like cmake and ninja.

[vcunat]

@amosbird: I think the easiest way for you is to wrap the cmake/ninja executables to unsed that variable for them.

[amosbird]

@vcunat Isn't that variable masking going to be inherited by subprocesses like toolchain binaries using LD_LIBRARY_PATH?

[vcunat]

I firmly believe that (exported) variables are inherited in the state you have them, so if you unset/change some, they'll be inherited that way.

[amosbird]

@vcunat yes, and that will break the toolchain's binaries.

[vcunat]

"that way" means _unset_, i.e. the way almost everyone uses the toolchain.

[vcunat]

Ah, and you want it the other way, probably. Linkify: https://github.com/NixOS/nixpkgs/issues/16767