Meson: rpath doesn't get set with find_library('foo', dirs:'/path/to/lib')

[rhd]

Hi Jussi,

If you build a shared object with meson, the rpath get sets correctly (or so the docs say), but when I use find_library() and give it a search path, the rpath doesn't get set. This is causing me lot of pain.

I'm currently doing stuff like this:

thirdparty_root = '/usr/local/thirdparty-1.6.0'
thirdparty_lib = [thirdparty_root + '/lib/linux-x64-all',
    thirdparty_root + '/lib/linux-x64-gcc4.8'
]
thirdparty_rpath = thirdparty_lib[0] + ':' + thirdparty_lib[1]

thirdparty_dep = declare_dependency(
    include_directories: thirdparty_inc,
    dependencies: [
        find_library('lib1', dirs: thirdparty_lib),
        find_library('lib2', dirs: thirdparty_lib),
        find_library('lib3', dirs: thirdparty_lib)
    ]
)

Then doing this:

executable('foo', 'foo.cpp',
   ...,
    link_args: '-Wl,-rpath,@0@:@1@'.format(thirdparty_rpath, boost_rpath),
)

Am I doing it wrong?

[jpakkane]

You can not set rpath yourself for the build dir (you can set an rpath that will be set after 'ninja install'). This is because Meson needs to do some Magic to the object files while they are in the build tree.

The established Unix convention is that you should not use rpath for finding public libraries, only for your private ones (like if you have stuff in /usr/lib/foobarlib/libfoobarprivate.so). For libraries that are public but not in the standard location (this includes things like /usr/local/lib) you need to set the LD_LIBRARY_PATH environment variable.

[rhd]

Hi Jussi,

This is one area where cmake and meson differ and IMO cmake does the "better thing".

By default, cmake will jam in the rpath values and strip them out for an install while giving you the option to keep them. This prevents the need for LD_LIBRARY_PATH and just works.

Meson is the opposite - it doesn't provide the rpath by default and allows you to add them during the install. This would cause 'ninja test' to fail on a system that wasn't properly setup even though the shared objects were there and everything built fine.

Are there any downsides to the cmake way? I know you mention some "magic".

What I'm currently doing is putting in the rpath in the link_args - is this going to cause me problems?

My other issue is that meson will strip out the rpath values during install - I really wish there was an option to not do that.

Thanks for taking the time to talk about these issues with me!

[jpakkane]

The last time I checked, CMake has two properties for rpath. One for build tree and one after install. CMake stores internal dirs in the rpath so binaries can be run directly from the build tree. They are removed upon ninja install. Meson does the same thing (I stole it directly).

To set the rpath to be used after install, just set install_rpath keyword argument on your targets.

Setting rpath yourself is very much undefined behaviour. It might work. It might not work. It might stop working at any time for any reason.

Setting builddir rpath has not been supported for a reason, which is that it is very easy to shoot yourself in the foot with it. At deployment time it can cause wacky stuff that is painful to debug. The simple and reliable ways to do dependencies is to either have a staging dir where you put your stuff (such as ~/devroot) and set LD_LIBRARY_PATH. The second is to embed your deps as Meson subprojects with wrap. That is obviously not possible for most deps.

If there is need, we could add a new keyword argument such as build_rpath that has a list of dirs that would be added to the list of rpath dirs. However I highly recommend you avoid rpaths if at all possible. It is one of those things that, unless you are extremely careful, will jump up and bite you in the ass when you least expect it.

[rhd]

Hi,

The last time I checked, CMake has two properties for rpath. One for build tree and one after install. CMake stores internal dirs in the rpath so binaries can be run directly from the build tree. They are removed upon ninja install. Meson does the same thing (I stole it directly).

Yes, it does. cmake goes one step further and links in rpath from external libs too. Meson doesn't. I'll follow up tomorrow with a specific example.

Setting rpath yourself is very much undefined behaviour. It might work. It might not work. It might stop working at any time for any reason.

:\

Setting builddir rpath has not been supported for a reason, which is that it is very easy to shoot yourself in the foot with it.

I agree - except I am on a system with a known configuration that must be setup the same way for things to work. In my specific case, the danger of shooting myself in the foot is limited...

If there is need, we could add a new keyword argument such as build_rpath that has a list of dirs that would be added to the list of rpath dirs.

The the project I'm working on, this would be pretty perfect. The code would look like:

lib1_lib_dir = '/path/to/lib1'
lib2_lib_dir = '/path/to/lib2'
exe = executable('blah', src, 
    build_rpath:[lib1_lib_dir, lib2_lib_dir], 
    install_rpath:[lib1_lib_dir, lib2_lib_dir], # would be nice if it could use the 'build_rpath' setting, but I'll live :)
)

However I highly recommend you avoid rpaths if at all possible.

Agreed, I wish I could avoid shared objects all together, but in this case, I can't.

I'll follow up again tomorrow with what I'm seeing cmake do.

Thanks!

[rhd]

Here are the bits from cmake files.

top level:

set(LIB_ROOT /usr/local/lib-3.6.0)
set(LIB_INCLUDE_DIRS ${LIB_ROOT}/include/common
 ${LIB_ROOT}/include/inc ${LIB_ROOT}/include/posix
 ${LIB_ROOT}/include/gcc)
set(LIB_LIBRARY_DIRS ${LIB_ROOT}/lib/linux-x64-all
 ${LIB_ROOT}/lib/linux-x64-gcc4.8)
set(LIB_LIBRARIES lib1 lib2)

sub folder

include_directories(${LIB_INCLUDE_DIRS}) 
link_directories(${LIB_LIBRARY_DIRS})
add_executable(foo main.cpp) 

And this outputs (during the link phase):

/usr/local/gcc-4.8.1/bin/g++    -std=c++11    
    CMakeFiles/foo.dir/main.cpp.o  -o foo  
    -L/usr/local/lib-3.6.0/lib/linux-x64-all  
    -L/usr/local/lib-3.6.0/lib/linux-x64-gcc4.8  
    -rdynamic -Wl,-Bstatic -Wl,-Bdynamic -llib1 -llib2 
    -Wl,-rpath,/usr/local/lib-3.6.0/lib/linux-x64-all:/usr/local/lib-3.6.0/lib/linux-x64-gcc4.8

This allows the unit tests to work without any extra work. The rpath gets stripped during install (which is changeable via an option)

[jpakkane]

What does it do for libraries obtained via FindPackage or pkg-config?

[rhd]

The top level

set(BOOST_ROOT /usr/local/boost_1_58_0)
find_package(Boost COMPONENTS program_options iostreams filesystem system REQUIRED)

Generates this: -Wl,-rpath,/usr/local/boost_1_58_0/lib (other paths stripped off).

I'm not sure about pkg-config.

[shlevy]

Note that this issue makes meson-build binaries efffectively unusable without further wrapping in nixpkgs, since by design nix doesn't install all libraries into a common prefix and instead puts each library in its own prefix. Is there any workaround from the perspective of a package manager maintainer?

[nirbheek]

Maintainers can edit the build files and set the install_rpath: kwarg on the relevant binaries. Unfortunately that's the only workaround available right now.

For the general case I suppose we would want to set the RPATH when linking to any libraries that aren't already in the linker's search path. This would mean people wouldn't need to manually set install_rpath: when linking to libraries from the same project that are installed into locations outside the linker search path.

I can only see advantages for this, any reason why we shouldn't do this?

[rhd]

You'd also need to maintain the build_rpath otherwise ninja test won't work.

This would bring parity to that cmake does and would be a very welcome change.

[jtojnar]

What would be the implications of this patch:

https://github.com/NixOS/nixpkgs/commit/259a74a166f09a688f82b3978eed2af2e8002001

[nirbheek]

That patch would result in the build-time rpaths being kept in the final binary. You can check by looking at the output of readelf -d <binary>.

[nirbheek]

That patch would result in the build-time rpaths being kept in the final binary; including those created by meson for running binaries uninstalled. You can check by looking at the output of readelf -d <binary>.

I think we should definitely auto-set RPATH when find_library contains dirs:, otherwise we're building a binary that cannot be run.

[jtojnar]

I mean, are there any downsides? Other than more directories to crawl through when loading a library. cc https://github.com/NixOS/nixpkgs/pull/29784#discussion_r140977322

[nirbheek]

No real downsides (besides affecting build reproducibility), as long as the build directory is truly temporary and will be gone afterwards.

[yegortimoshenko]

There is a downside as far as I understand. One could use install_rpath to supply a custom rpath to a directory, for example:

executable('app', 'main.cpp',
    dependencies: [ Xrandr ],
    install: true,
    install_rpath: '../custom-lib-path',
)

With patch @yorickvP proposed that I want(ed) to incorporate in NixOS/nixpkgs#29784, install_rpath would never be actually installed (or rather, appended) to rpath.

I still believe that with this patch, Meson behavior is much more sane (and no project I've seen in the wild actually uses install_rpath this way).

[jpakkane]

install_rpath: '../custom-lib-path',

This should be $ORIGIN/../custom-libpath. Rpath handling is special in many ways.

[yorickvP]

I don't remember patches. Leaving the meson build dir in the rpath is a security problem if other people can write to them or change environment variables in them (don't get sanitized by sudo like LD_LIBRARY_PATH). Ideally, meson would only remove it's own rpaths and not the ones added by our gcc wrapper(?).

[morrisonlevi]

Chiming in a bit late, but regarding:

For libraries that are public but not in the standard location (this includes things like /usr/local/lib) you need to set the LD_LIBRARY_PATH environment variable.

I'd like to invoke CITATION NEEDED. Everything I've read from knowledgeable sources says that LD_LIBRARY_PATH is designed for temporary usage, not permanent usage. Here are some that explicitly call LD_LIBRARY_PATH bad:

• LD_LIBRARY_PATH considered harmful (Although this one does seem to misunderstand the related variable LD_RUN_PATH, the rest looks correct)
• Why LD_LIBRARY_PATH is bad

Typically on gcc you would add non-standard software locations to -L and append it to the -Wl,-rpath, preferably with -Wl,--enable-new-dtags. I definitely think meson should make this simple to get right.

Edit: and, incidentally, using LD_LIBRARY_PATH temporarily in the build directory is appropriate and using rpath there would be inappropriate, at least as far as I understand the meson-specific issue.

[NickeZ]

If you have public libraries in non-standard locations (like /usr/local/lib) you should probably do echo /usr/local/lib > /etc/ld.so.conf.d/local.conf (at least on debian-based systems). This is the whole purpose of having ldconfig read files in a special .d folder.

[jtojnar]

If you have public libraries in non-standard locations (like /usr/local/lib) you should probably do echo /usr/local/lib > /etc/ld.so.conf.d/local.conf (at least on debian-based systems). This is the whole purpose of having ldconfig read files in a special .d folder.

This is not a possibility on NixOS, where every library is stored in its own directory like /nix/store/hash-pkgname. We could add a wrapper script that would set the LD_LIBRARY_PATH variable but handling this on meson level would be much more practical.

[yegortimoshenko]

@NickeZ There is a reason why ELF DT_RPATH exists and libraries don't just append an entry to ld.so.conf on install. The way Meson treats DT_RPATH is totally broken. This is a workaround at best and doesn't resolve the actual problem.

[yegortimoshenko]

This is not a possibility on NixOS, where every library is stored in its own directory like /nix/store/hash-pkgname. We could add a wrapper script that would set the LD_LIBRARY_PATH variable but handling this on meson level would be much more practical.

@jtojnar Currently most NixOS derivations that use Meson just use patchelf and restore correct DT_RPATH for every dynamic executable or library in the output, just like we do with proprietary stuff. Luckily we don't have to use LD_LIBRARY_PATH, it's a very bad idea to do so and will break in FHS environments like Steam runtime because quite a few applications override it.

[NickeZ]

I'm not sure how you can use rpath in meson. But I would say that the use case should be strictly limited to libraries that are compiled within the same project. Meson shouldn't support public dependencies in non-standard locations because this is not related to building a project. Definitely not through the build file syntax.

I think package managers that needs this funcitonality should run some post-build command to add rpaths.

Anyways this isn't supported on windows binaries so it would give meson a platform dependent feeling.

[yegortimoshenko]

@NickeZ This is a bug. It's specific to platforms that use ELF (which doesn't include Darwin). Solving it has no security repercussions whatsoever: if you believe that it does, you should also wipe rpath in Mach-O binaries, so it's a bug either way.

https://github.com/mesonbuild/meson/blob/db34a3a7017d0096faa8d3f020efd078ad8a65e1/mesonbuild/scripts/depfixer.py#L287-L318

The problem here is that this overwrites existing rpath instead of appending install_rpath to it. (Moreover, it seems that if I set install_rpath to be long enough, this will fail).

This should not be a post-build fixup as it currently is: flags should be passed to the compiler to append to existing rpath. Also it doesn't make sense that install_rpath doesn't do anything on Darwin as Mach-O has @rpath. Sorry, but overwriting ELF headers is a dirty hack.

No other build system has this problem, because no other build system touches ELF headers. If build system ignores compile/linker flags, it doesn't do its job properly. Remember SCons? :-)

[NickeZ]

I'm not sure about the current behavior, there may be a bug or not.

What I mean is that the CMake behavior quoted below would be wrong to implement in the build system. There is no reason to expect that the end user will have, for example, boost in the same path that it was on the build machine. To facilitate package maintainers I supposed you could add an environment variable or an argument to meson to override the built-in behavior. But IMO paths shouldn't be hard coded into the binaries by default. With the only exception of libraries that the application itself maintains.

The top level

set(BOOST_ROOT /usr/local/boost_1_58_0)
find_package(Boost COMPONENTS program_options iostreams filesystem system REQUIRED)

Generates this: -Wl,-rpath,/usr/local/boost_1_58_0/lib (other paths stripped off).

I'm not sure about pkg-config.

[NickeZ]

Maybe it could be as simple as by default don't insert any rpath that begins with / and have some environment variable/command line option that instead inserts an rpath for every single dynamic library. Wouldn't that solve the nix os use case?

[jpakkane]

No other build system has this problem, because no other build system touches ELF headers.

Meson's rpath fixup handling is stolen verbatim from CMake.

That is, inside the build directory Meson reserves the right to set up rpath however it likes. These internal entries are stripped on install so only those set explicitly with install_rpath remain. If we don't do the equivalent thing on OSX then it's a bug and should be fixed.

[shlevy]

@jpakkane Cmake definitely does not have this issue. We need to distinguish rpath entries pointing to system libraries and rpath entries pointing to temporary locations created by the build. The former should not be stripped.

[rhd]

cmake definitely jams in the rpath for out-of-source libraries when using FindPackage as shown here

[morrisonlevi]

That is, inside the build directory Meson reserves the right to set up rpath however it likes.

Can anyone explain why this bit is necessary?

[jpakkane]

Can anyone explain why this bit is necessary?

Because we want to be able to run executables directly from the build dir without doing ninja install or anything like that.

[morrisonlevi]

[W]e want to be able to run executables directly from the build dir without doing ninja install [...]

Okay. Why do we need to use special rpaths different from the installed rpath?

[jpakkane]

Because they point to the inside of the build dir. Usually something like $ORIGIN/../foo/libdir/.

[morrisonlevi]

Hmm. This doesn't jive with my previous experience with other systems. Why are dependencies located inside the build directory? Aren't they typically found on the system through pkg-config or something similar? Or are these libraries part of the project?

[rhd]

@morrisonlevi you have to consider the case where meson builds the shared_library and it lives in the build directory. Without rpath mods, any app that links against it wouldn't work.

[morrisonlevi]

In that case why don't we mimic the install structure in the build structure? Then the $ORIGIN/../lib rpath works in either case?

I think understand what meson is doing but don't understand why it's done that way. It seems like a meson-created issue, not an inherent one. Does that make sense?

[jpakkane]

This is identical to what CMake does. This is also what Autotools does, the "executables" and "libraries" it produces are actually shell scripts that set up magic envvars to mimic what we do with rpath. Having the build directory layout mirror the final install layout makes things hard and some projects will not work at all, mostly because they have always been built in-source so the directory layout is buried deep inside their settings.

[rhd]

This issue isn't about meson managed shared objects. It's about shared objects that 1) aren't built by meson and 2) aren't in standard library search paths. In this case, meson kind of leaves you hanging - which deviates from cmake.

[morrisonlevi]

Let's assume I have the latest and greatest version of libfoo installed to /opt/libfoo/2.0. It doesn't provide .pc files for pkg-config. How do we get meson to find it? As far as I understand it I'd need to hard-code find_library('libfoo', dirs='/opt/libfoo/2.0') and then it still won't include /opt/libfoo/2.0/lib in the rpath, so I'd need to then edit the executable() call to add it to install_rpath:. Is that right? Did I get something wrong or miss a step? Edit: I assume we can set C_INCLUDE_PATH + LIBRARY_PATH; what about LD_RUN_PATH? Does it get ignored?

[rhd]

@morrisonlevi you hit the nail on the head. That's what this issue is about. cmake automatically adds the rpath for you - at least it used to when I first opened this issue - I haven't used cmake in a long time (thanks to meson).

[NickeZ]

@morrisonlevi "find_library('libfoo', dirs='/opt/libfoo/2.0')" is clearly insane if the build files should be reusable for different distributions. "executable() call to add it to install_rpath" is even worse in my opinion. Why should the build system care about where the library are when it is time to run an executable? But in very opinionated environments like nixOS it is clearly beneficial if the paths to all the libraries are put in as rpath in the executables. But this has to be done in a way so that the meson.build files still are usable for other distributions (i.e. command line flags or environment variables).

[shlevy]

@morrisonlevi @NickeZ No one suggested we should put custom library paths into the meson build files. The point is we set up pkgconfig path or library search path to include the non-standard directories.

[shlevy]

To be clear, the meson builds have no problem building and linking on NixOS. They just fail at runtime due to the superfluous rpath stripping.

[jpakkane]

That's what this issue is about. cmake automatically adds the rpath for you - at least it used to when I first opened this issue

We changed our behaviour to do the same just this last release. The commit to do that is this one.

That is only applied inside the build dir, it is stripped on install like the other ones.

[morrisonlevi]

A bit of background: I install and maintain a wide range of packages for HPC clusters. It is absolutely routine to install software into non-standard locations because people need different versions of the same program/library. Some software provides .pc files so we add them to our PKG_CONFIG_PATH, but not everything does and not every build tool uses them anyway. So when we install a program that needs dependency libfoo we add things to LIBRARY_PATH, LD_RUN_PATH, CPATH/C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, etc.

Meson breaks this. We intend the DT_RPATH/DR_RUNPATH to be set when we set LD_RUN_PATH. Shouldn't the build system honor that? If meson mangles these during build that's fine as long as 1) it works and 2) on install it puts them the way they ought to be. For CMake I do something like:

cmake -DCMAKE_INSTALL_RPATH="$LD_RUN_PATH" [...]

I don't think meson has anything as an equivalent on the command-line, does it?

[rhd]

We changed our behaviour to do the same just this last release. The commit to do that is this one.

That is only applied inside the build dir, it is stripped on install like the other ones.

Oh, I didn't know this. This is great. So it seems the only thing people (myself included) are missing is the ability for meson to keep the non-standard system rpaths after a ninja install.

[shlevy]

FWIW, on nixpkgs we just patch meson now and everything works as desired https://github.com/NixOS/nixpkgs/commit/8f95aef531de13e6fffc62a31b4106f745d647ec

[yegortimoshenko]

@shlevy This is just a workaround. We should probably install install_rpath via patchelf.

On Meson: it is currently missing something like CMake's -DCMAKE_SKIP_BUILD_RPATH=ON flag, which I guess would be a solution most people would agree with.

[morrisonlevi]

Based solely on description CMAKE_INSTALL_RPATH_USE_LINK_PATH seems to be the behavior that @rhd is interested in when they opened this ticket. I would also like to see an equivalent to CMAKE_INSTALL_RPATH which gives the builder full control over the final rpath.

[jtojnar]

Would there be any disadvantage to doing the following? As I understand it, the whole X business is to ensure the RPATH is not contaminated with the install_rpath when running the app from build directory but I do not see what harm it could do – even if the install_rpath existed on the file system (e.g. /usr/local/lib/deja-dup) the $ORIGIN paths would precede it.

diff --git a/mesonbuild/compilers/compilers.py b/mesonbuild/compilers/compilers.py
index 3f088b0f..71a5fe8e 100644
--- a/mesonbuild/compilers/compilers.py
+++ b/mesonbuild/compilers/compilers.py
@@ -834,12 +834,10 @@ class Compiler:
         # Build_rpath is used as-is (it is usually absolute).
         if build_rpath != '':
             paths += ':' + build_rpath
-        if len(paths) < len(install_rpath):
-            padding = 'X' * (len(install_rpath) - len(paths))
-            if not paths:
-                paths = padding
-            else:
-                paths = paths + ':' + padding
+        if not paths:
+            paths = install_rpath
+        else:
+            paths = paths + ':' + install_rpath
         args = ['-Wl,-rpath,' + paths]
         if get_compiler_is_linuxlike(self):
             # Rpaths to use while linking must be absolute. These are not
diff --git a/mesonbuild/scripts/meson_install.py b/mesonbuild/scripts/meson_install.py
index 985b0e94..665f523b 100644
--- a/mesonbuild/scripts/meson_install.py
+++ b/mesonbuild/scripts/meson_install.py
@@ -353,7 +353,6 @@ def install_targets(d):
         if is_elf_platform() and os.path.isfile(outname):
             try:
                 e = depfixer.Elf(outname, False)
-                e.fix_rpath(install_rpath)
             except SystemExit as e:
                 if isinstance(e.code, int) and e.code == 0:
                     pass

[jpakkane]

Would there be any disadvantage to doing the following? As I understand it, the whole X business is to ensure the RPATH is not contaminated with the install_rpath when running the app from build directory

The disadvantage would be that it would break everything. The X business is not there to prevent install rpath from being in the build dir, it is to ensure the build dir rpath is not there after an install.

The final product must not contain any traces of the build dir. This is a hard, non-negotiable requirement from Linux distros. If rpath on binaries is not perfectly clean, the package will get autorejected.

[arnout]

I don't think this is fixed.

Here is a reproduction scenario:

wget http://downloads.sourceforge.net/project/libpng/zlib/1.2.8/zlib-1.2.8.tar.xz
tar xf zlib-1.2.8.tar.xz
cd zlib-1.2.8
./configure --prefix=/tmp/mesonfail
make
make install

cd ..

cat > fail.c <<EOF
#include <zlib.h>
#include <stdio.h>
#include <string.h>

int main()
{
   printf("%s = %s\n", zlibVersion(), ZLIB_VERSION);
   return !!strcmp(zlibVersion(), ZLIB_VERSION);
}

EOF

cat > meson.build <<EOF
project('mesonfail', ['c'])
zdep = dependency('zlib', version : '=1.2.8')
exe = executable('fail', 'fail.c', dependencies: zdep, install: true)
EOF

rm -rf build
mkdir build
cd build
PKG_CONFIG_LIBDIR=/tmp/mesonfail/lib/pkgconfig meson --prefix=/tmp/mesonfail ..
ninja -v install
./fail && echo OK # OK
/tmp/mesonfail/bin/fail || echo FAIL # FAIL unless zlib 1.2.8 is installed in /usr/lib

[xclaesse]

I'm my pr #4321 i use find_library() with 'dirs:' and the rpath seems to work on Linux but not on Mac OSX.

[aduskett]

Buildroot has had a workaround for over a year now in regards to the fix_rpath on install bug:
https://gitlab.com/buildroot.org/buildroot/blob/master/package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch

It seems like a trivial fix to implement, and I am not sure why it hasn't been done yet:
1) It's simple.
2) It works properly.
3) It's only 8 lines of code, 3 of which are comments, one of which is a print statement.

[dankegel]

@arnout, thank you for the easy to reproduce test case.

My attempt at a fix is https://github.com/mesonbuild/meson/pull/7103, now pending review.

You can rescue your test case by applying https://github.com/mesonbuild/meson/pull/7103
and then supplying the right rpath in one of two ways:
1) by adding LDFLAGS when running ninja, e.g.

 rm -rf build
 mkdir build
 cd build
+LDFLAGS="-Wl,-rpath=/tmp/mesonfail/lib" \
 PKG_CONFIG_LIBDIR=/tmp/mesonfail/lib/pkgconfig ~/src/meson/meson.py --prefix=/tmp/mesonfail ..
 ninja -v install
 ./fail && echo OK # OK

2) if you don't want to do that, you could supply the rpath by editing zlib.pc, e.g.

 Version: 1.2.8

 Requires:
-Libs: -L${libdir} -L${sharedlibdir} -lz
+Libs: -L${libdir} -L${sharedlibdir} -lz -Wl,-rpath=${libdir}
 Cflags: -I${includedir}