Conda-build: On OSX, library install name should contain @rpath

[dfroger]

In order to make is easier for user to link executable (not packaged with conda) with library
installed with conda, library install names shoud contain @rpath.

Current situtation

From what I understand looking at pkgs/hdf5-1.8.13-1/lib/libhdf5.dylib and
pkgs/hdf5-1.8.13-1/bin/h5dump here is the current situation, sorry if
I missunderstand something, or miss other source of informations:

On OS X, for a conda pakage foo containing a library libfoo.dylib and an executable xfoo,
conda install foo yields to something like (actually, the the binary files are just
extracted from the package, but it doesn't matter for the example):

$ ROOT=$PWD
$ mkdir -p $ROOT/miniconda/pkgs/foo/bin $ROOT/miniconda/pkgs/foo/lib

$ cd $ROOT/miniconda/pkgs/foo/lib
$ echo 'int foo(int a, int b){return a+b;}' > foo.c
$ clang foo.c -dynamiclib -install_name libfoo.dylib -o libfoo.dylib

$ cd $ROOT/miniconda/pkgs/foo/bin
$ echo 'int foo(int,int); int main(){return foo(2,4);}' > xfoo.c
$ clang xfoo.c -L../lib -lfoo  -o xfoo
$ install_name_tool -change libfoo.dylib '@loader_path/../lib/libfoo.dylib' xfoo

Now, the directory libfoo.dylib and xfoo moved for example as:

$ mkdir -p $ROOT/miniconda/envs/envname/lib $ROOT/miniconda/envs/envname/bin
$ ln $ROOT/miniconda/pkgs/foo/lib/libfoo.dylib $ROOT/miniconda/envs/envname/lib/libfoo.dylib
$ ln $ROOT/miniconda/pkgs/foo/bin/xfoo $ROOT/miniconda/envs/envname/bin/xfoo

safely, because in xfoo, the install name of the dependent shared library fooin relative to where xfoo is.
foo and xfoo are rellocatables.

Problem

A user developping an executable yfoo needing the foo library will want to reuse the library libfoo.dylib
provided by conda, due to the easiness and flexibility of the installation with conda.

However, this require to run install_name_tool -change on the created executable:

$ mkdir $ROOT/perso ; cd $ROOT/perso
$ echo 'int foo(int,int); int main(){return foo(2,4);}' > yfoo.c
$ clang yfoo.c -L$ROOT/miniconda/envs/envname/lib -lfoo  -o yfoo
$ install_name_tool -change libfoo.dylib "$ROOT/miniconda/envs/envname/lib/libfoo.dylib" yfoo

Proposal

The library install name should contain @rpath, so conda install foo yields to something like this:

$ mkdir -p $ROOT/miniconda_bis/pkgs/foo/bin $ROOT/miniconda_bis/pkgs/foo/lib

$ cd $ROOT/miniconda_bis/pkgs/foo/lib
$ echo 'int foo(int a, int b){return a+b;}' > foo.c
$ clang foo.c -dynamiclib -install_name '@rpath/libfoo.dylib' -o libfoo.dylib

$ cd $ROOT/miniconda_bis/pkgs/foo/bin
$ echo 'int foo(int,int); int main(){return foo(2,4);}' > xfoo.c
$ clang xfoo.c -L../lib -lfoo  -o xfoo
$ install_name_tool -change '@rpath/libfoo.dylib' '@loader_path/../lib/libfoo.dylib' xfoo

$ mkdir -p $ROOT/miniconda_bis/envs/envname/lib $ROOT/miniconda_bis/envs/envname/bin
$ ln $ROOT/miniconda_bis/pkgs/foo/lib/libfoo.dylib $ROOT/miniconda_bis/envs/envname/lib/libfoo.dylib
$ ln $ROOT/miniconda_bis/pkgs/foo/bin/xfoo $ROOT/miniconda_bis/envs/envname/bin/xfoo

So that the user can links with libfoo.dylib using rpath:

$ mkdir $ROOT/perso_bis ; cd $ROOT/perso_bis
$ echo 'int foo(int,int); int main(){return foo(2,4);}' > yfoo.c
$ clang yfoo.c -L$ROOT/miniconda_bis/envs/envname/lib -Wl,-rpath,$ROOT/miniconda_bis/envs/envname/lib -lfoo -o yfoo

[asmeurer]

I didn't know about @rpath on OS X. It looks like http://stackoverflow.com/questions/9263256/can-you-please-help-me-understand-how-mach-o-libraries-work-in-mac-os-x and man dyld (at the bottom) are good references.

Can you explain more how your example works? Does it require compiling foo with certain compiler flags? If so, then that's a bit of a show stopper, as we don't want to require that a package be compiled with specific flags just so that it can be made into a valid conda package.

[dfroger]

The install_name of foo can be set at its creation with the compiler flag -install_name, or after its creation with install_name_tool -id and install_name_tool -change.

Here are others references that may be useful:

• https://wincent.com/wiki/@executable_path,_@load_path_and_@rpath
• https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html

I don't have very experience with these things, I've just searched informations on the web. Here I summarise what I've understand, hope it can helps others. I may be wrong and I would be happy if experienced users could confirm/fix this.

• When a shared library is created, the install_name of the library is written in the header of the library.
• When linking an executable A (or a shared library A) with a shared library B, the install_name of B is copied into the header of the executable A, and become the install_name in A of the dependant shared library B (-L/path/to/lib/dir to find the shared library B at link time)
• When the executable A is executed, the dependent shared library B is searched accoring to its install_name in A

The install_name in both library B or executable A can be changed after their creation, see the section usefull commands bellow.

The motivation to not just set the install_name of a library to its absolute path at the library creation time, is because in a typical ./configure ; make ; make install process:

• during the make:
  
  
  
  • library would be created with install_name /path/to/build/dir/lib/libfoo.dylib
  
  
  • executable is linked with library, and install_name in executable would be /path/to/build/dir/lib/libfoo.dylib
  
  
• during the make install:
  
  
  
  • library and executable are move to /path/to/install/dir/[lib,bin]
  
  
• then:
  
  
  
  • when executable is executed, it would search library in /path/to/build/dir/lib/libfoo.dylib, and would not find it.
  
  
  • creating new libraries that links to /path/to/install/dir/lib/libfoo.dylib would inherit the wrong install_name /path/to/build/dir/lib/libfoo.dylib, and fails to be executed too.
  
  

So we want options to set install_name to relative paths. It also allow to package libraries and executables that works in the environment where they have been compiled and in the environment where they have been installed, and be installed in different places (relocation).

If we have the following directories and files:

/path/to/lib/libfoo.dylib  # install_name is ../lib/libfoo.dylib
/path/to/bin/xfoo          # install_name of dependent libfoo.dylib library is ../lib/libfoo.dylib

Invoking xfoo from /path/to/bin will search libfoo.dylib in /path/to/bin../lib/libfoo.dylib, and
will succeed.

Invoking xfoo from /path/to will search libfoo.dylib in /path/to/../lib/libfoo.dylib, and
will fails.

OSX provides several options to fix this:

• setting the install_name to @executable_path/../lib will always replace @executable_path with
  /path/to/bin../lib/libfoo.dylib, wherever the executable is invoked from.
• @loader_path is the same thing, but works also in the case when shared library A is loaded and searches
  for a dependant shared library B.
• @rpath allow to give to the executable whatever path we want to search library in. For exemple,
  we may have three libraries libfoo.dylib in different directories, which use @rpath in their install_name:

/path/to/lib/libfoo.dylib # install_name is @rpath/libfoo.dylib
/another/path/to/lib/libfoo.dylib # install_name is @rpath/libfoo.dylib
/path/to/bin/xfoo # install_name is @rpath/libfoo.dylib

At the creation of xfoo, the value of @rpath can be written in the executable
(to specify which of the three library to use)
with -Wl,-rpath=/path/to/lib, so that when xfoo is invoked, it search libfoo.dylib
in /path/to/lib/libfoo.dylib.

Then, we can use install_name_tool to change the rpath in xfoo to /another/path/to/lib,
or to something begining with @loader_path or @executable_path.

Note that environment variable can shortcuts the use of the install_name of the dependent
shared library, typically using DYLD_LIBRARY_PATH.

When linking a executable the a shared library, I didn't find any clang/dyld options to specify a dependent shared library install_name different that the one in the shared library linked.

Usefull commands

Setting the install_name of a libary at its creation:

clang <sources> -dynamiclib -install_name <install name> -o lib<name>.dylib

Print install name of a shared library:

otool -D <library>

Lists dependent dynamics libraries and its install name:

otool -L <excutable or library>

Show the rpath:

otool -l <executable or library> # look at the section LC_RPATH

Change the install_name of a library:

install_name_tool -id /new/install/name /path/to/lib<name>.dylib

Change install_name of a dependent library:

install_name_tool -change old/path/to/libonwhichfoodepends.so new/path/to/libonwhichfoodepends.so libfoo.dylib

install_name_tool also has options to modify or add or delete rpath in a executable or shared library.

[asmeurer]

Thank you for the write up. That is very helpful.

So what change are you suggesting to make to conda-build?

[dfroger]

So the change I'm suggesting to make to conda-build is that conda-build use:

install_name_tool -id @rpath/libfoo.dylib path/to/libfoo.dylib

instead of:

install_name_tool -id libfoo.dylib path/to/libfoo.dylib

I think (or hope) it should not break anything in conda, and make it easier to link external program with libraries distributed with conda.

[asmeurer]

I see. I'd need to play around with this a bit to see if it breaks anything.

[asmeurer]

I think if we make that change we also need to add to the logic that changes the dependent install names to look for install names that start with @rpath.

[dfroger]

I think we only need to change the install name of library, and that install name of dependent library are already good, because install name are used at link time, and dependant install name are used at load time.

Simillary, install name are not used at load time, and dependant install name are not used at link time

Install name of $ROOT/miniconda/pkgs/foo/lib/libfoo.dylib would change from libfoo.dylib to @rpath/libfoo.dylib, so that when linking an executable with libfoo.dylib, with can use -rpath=$ROOT/miniconda/pkgs/foo/lib, and when the executable will load the library, it will search it in $ROOT/miniconda/pkgs/foo/lib/libfoo.dylib (the dependant install name) and find it, instead of searching it as libfoo.dylib, which require to set environment variable, or to change the dependant install name with install_name_tool -change. We could also use -rpath=@executable_path/... or -rpath=@loader_path/... if we want relative path.

Install name of dependant library libfoo.dylib in $ROOT/miniconda/pkgs/foo/bin/xfoo is @loader_path/../lib/libfoo.dylib and is good. xfoo will find libfoo.dylib because using @loader_path../lib/libfoo.dylib, install name @rpath/libfoo.dylib in libfoo.dylib does not play any role. install name @rpath/libfoo.dylib plays a role only at link time, not at load time.

Or if $ROOT/miniconda/pkgs/foo/lib/libbar.dylib tries to load $ROOT/miniconda/pkgs/foo/lib/libfoo.dylib, dependant install name @load_path/./libfoo.dylib (or something like that) is good to for the same reason. install name of libbar.dylib would be @rpath/libbar.dylib (as for libfoo.dylib), so it can be linked by external executable.

Note that we know in advance the relative path between $ROOT/miniconda/pkgs/foo/bin/xfoo and $ROOT/miniconda/pkgs/foo/lib/lifoo.dylib, which will never change, so install name of the dependant library libfoo.dylib in xfoo is relative, so both can be moved together (relocation).

By contrast, we don't know the path of the external executable the user want to link libfoo.dylib with, so we can not use a fixed relative path for the install name. So we need to let the user the possibilty to specify at link time the value of the dependant install name of libfoo.dylib in the executable, this is what @rpath is for.

Sorry for the long comment again...

[dfroger]

Indeed you're absolutely right in your last comment about the function osx_ch_link :-) I stupidely missundersand the comment, sorry.

i'm experimenting building packages with these changes, and will propose a pull request when it's ready.

[asmeurer]

See https://github.com/conda/conda-build/pull/312 (still WIP).

[ddale]

Any estimate on when this issue will be resolved? I have a project (http://github.com/praxes/sweep2png) for which I can't create conda packages, although I was able to do so a couple months ago. Or should I try with an older version of conda-build?

[asmeurer]

Hmm, why doesn't it work? Nothing should have changed in master regarding this. The issues this fixes have existed since the beginning.

[ddale]

Hmm. Maybe its a different issue. Four months ago, I was able to build this recipe, run the test, and upload the result to https://binstar.org/praxes/sweep2png/files. Now, when I try to build the recipe, it fails during the test:

[...]
 cp sweep2png /Users/darren/.conda/envs/_build/bin/sweep2png
number of files: 1
install_name_tool -change libpng15.15.dylib @loader_path/../lib/libpng15.15.dylib /Users/darren/.conda/envs/_build/bin/sweep2png

BUILD END: sweep2png-master-0
TEST START: sweep2png-master-0
Fetching package metadata: ............
Solving package specifications: 
The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    sweep2png-master           |                0         1.1 MB

The following NEW packages will be INSTALLED:

    sweep2png: master-0

Fetching packages ...
sweep2png-mast 100% |################################| Time: 0:00:00  58.35 MB/s
Extracting packages ...
[      COMPLETE      ] |##################################################| 100%
Linking packages ...
[      COMPLETE      ] |##################################################| 100%
+ sweep2png Si_5grains_mesh.log map/Si_70 0.7 cubic
dyld: Library not loaded: @loader_path/../lib/libpng15.15.dylib
  Referenced from: /Users/darren/.conda/envs/_test/bin/sweep2png
  Reason: image not found
/Users/darren/conda-bld/test-tmp_dir/run_test.sh: line 3: 25224 Trace/BPT trap: 5       sweep2png Si_5grains_mesh.log map/Si_70 0.7 cubic
TESTS FAILED: sweep2png-master-0

[ddale]

This was my mistake, I must have introduced a problem in my conda recipe. Sorry for the noise. Do you want me to delete unhelpful comments?

[asmeurer]

Don't worry about it.

[denis-bz]

Is @rpath/./libnetcdf.7.dylib with an absolute .dylib version an error ?
After conda install -c conda-forge pynco on macos 10.10 I get

ncks --version  # many nc* same
dyld: Library not loaded: @rpath/./libnetcdf.7.dylib
  Referenced from: /opt/local/py/miniconda2/bin/ncwa
  Reason: image not found
Trace/BPT trap

otool -L $conda/bin/ncwa  # macos
@rpath/./libnetcdf.7.dylib (compatibility version 10.0.0, current version 10.0.0)
@rpath/./libhdf5_hl.10.dylib (compatibility version 11.0.0, current version 11.1.0)
@rpath/./libhdf5.10.dylib (compatibility version 11.0.0, current version 11.1.0)
...

which don't exist:

ls -l $DYLD_FALLBACK_LIBRARY_PATH/libnetcdf*.dylib
1290160 May 20 12:04 /opt/local/py/miniconda2/lib/libnetcdf.13.dylib
     18 May 20 12:04 /opt/local/py/miniconda2/lib/libnetcdf.dylib -> libnetcdf.13.dylib

This is likely my stupid error (TL;DR all of the above),
or should it be @rpath/./libnetcdf.dylib instead of libnetcdf.7.dylib ?

Thanks, cheers

[mingwandroid]

@denis-bz, you have a simple version mismatch here. Probably mixing conda-forge and defaults or something like that?

There's nothing wrong with having the .7. in the LC_LOAD_LYLIB command.

I have no idea why you're messing about with DYLD_FALLBACK_LIBRARY_PATH since it doesn't do anything anymore for the vast majority of users and usecases (since SIP).

[denis-bz]

@mingwandroid, the problem is that conda loaded files with @rpath/./libnetcdf.7.dylib,
which doesn't exist; the only libnetcdf*.dylib on the machine are the 2 I listed.
(DYLD_LIBRARY_PATH is a red herring, historical, unsetting it has no effect.)

[mingwandroid]

Please file a bug at http://github.com/conda-forge/pynco-feedstock but before you do, do your homework so you don't waste peoples' time.

This is a packaging error by the look of it and nothing to do with this issue at all (which you'd have known if you didn't treat it as TL;DR).