Crystal: (Homebrew Install) Link fails if Crystal & its dependencies are not in /usr/lib or /usr/local/lib

[halostatue]

I am using Crystal through homebrew, and I install into ~/.brew, which means that the garbage collector is in $(brew --prefix)/lib/libgc.dylib (➡️ ~/.brew/lib/libgc.dylib).

I can only compile if I add --link-flags -L$(brew --prefix)/lib to my crystal build command line:

crystal build program.cr --link-flags -L$(brew --prefix)/lib

The problem is at src/compiler/crystal/codegen/link.cr:85 in that there appears to be no way for you to do anything _except_ support /usr/lib and /usr/local/lib without specifying.

Ideally, this will look for a file that contains additional configuration defaults (possibly project-specific). If this file were something like crystal-config, I would look for:

• config/crystal-config
• ./.crystal-config
• ./.config/crystal-config
• ~/.crystal-config
• /etc/crystal/config

Maybe with version distinction at the ~/.config, ~/, and /etc/crystal level. Looking for /etc should not be a fixed value, either, because if Crystal were installed in /opt, it should be looking in /opt/etc _first_. In my homebrew case, if I do brew --prefix crystal-lang, I get the result ~/.brew/opt/crystal-lang/, under which there is an etc directory. Perhaps looking for a configuration relative to the crystal binary might be a good choice.

[ysbaddaden]

The code you link are the default paths, and they are indeed appended to the link command, but the system linker will still use the paths you define to its environment variables or configuration files.

For example on OSX, you may set:

LIBRARY_PATH=~/.brew/lib:$LIBRARY_PATH

[halostatue]

What I’m saying is that the default paths included _are insufficient_ because it assumes a particular installation location for Crystal, its dependencies, and system dependencies. Should Windows become a priority at some point, having /usr/lib and /usr/local/lib as the defined values will completely fail. I _think_ that if I were to use Crystal under Nix or Guix, I would run into the same problem (without patching Crystal’s link.cr and recompiling).

I don’t know what the correct solution is, but telling users to set LIBRARY_PATH is not a solution. (It’s a fine interim workaround, but it’s not a solution, as it would _also_ change any other language’s library search path which I may not want by default.)

[RX14]

The library paths specified by crystal are in addition to the default ones that the linker should search. In a properly configured system, crystal shouldn't have to specify any linker paths at all. Crystal is not assuming a default install location, the default linker should find the libraries. The lack of all the paths where libraries reside in LIBRARY_PATH is a configuration error, and should not be accounted for by crystal. NixOS modifies LIBRARY_PATH in it's linker wrapper to fix this, as it should.

[halostatue]

Well, I'm not sure where the bug rests, but Crystal is the only brew-installed language I have that can't find its own link-time dependencies in my Homebrew install.

I realize this may not be a priority for you, but it makes Crystal unusable for me.

[asterite]

If it helps, I remember at some point noticing that our homebrew formula might not work if homebrew is not configured with the default paths. I can't remember why, now.

[asterite]

Hmmm... reading the comments above I agree with @ysbaddaden and @RX14 : Crystal just passed -lgc to the linker and it expects a properly configured system. Maybe other things installed via homebrew don't need to execute a link command?

[halostatue]

Pretty sure that’s not the case, since some of them are also built on LLVM as you are. It may be that “they” (that is, whoever made the Homebrew formula) are doing something within their Homebrew config that makes it work correctly.

I understand having to provide -L$(brew --prefix)/lib for tarballs that I download (or Rubygems that I install and want to use my homebrew versions of libraries rather than system ones, for example). I even understand that this is a use-case that you didn’t think of. I can’t understand that you think that this is not a “properly configured system”. My system is properly configured, and I expect languages that have link times to be aware that, if they are not installed in a “normal” place (e.g., /usr/local/…), they should ensure that their sibling directories are included as appropriate so that they can actually work _out of the box_. This is how /opt is supposed to work. This is how installing Homebrew in a directory different than /usr/local/ is supposed to work.

Idris provides idris --link, which on my system shows idris --link -L/Users/austin/.brew/Cellar/idris/0.12.1/share/x86_64-osx-ghc-8.0.1/idris-0.12.1/rts -lidris_rts -lpthread, which means it knows a bit more about its install environment.

One (probably easy) fix would be to modify the Homebrew formula so that it modifies link.cr to include the Homebrew prefix. This will only work as long as Crystal is delivered by source instead of a bottle, but it would be a start until you make your installation relocatable.

[RX14]

One solution would be making the crystal installed executable a wrapper which sets the correct LIBRARY_PATH before invoking crystal. That's much cleaner to me.

[halostatue]

@RX14, that would be fine. I haven’t yet learned Idris to be able to tell you how it’s able to determine its link parameters, but it is fairly smart.

[asterite]

Mmm... but imagine you want to use the XML module. Crystal uses libxml2 for that, and it expects it to be in the library path: in the system's library path, not in crystal's library path (Crystal doesn't provide libxml2, nor it provides any C library you could possibly link to). So what would we do in that case?

[ysbaddaden]

As crystal developers, we consider installing brew to a non standard directory but not declaring that special directory to the system to be a "non properly configured system" because installed libraries can't be found by the system linker, and there isn't much that the crystal compiler can do about it (we won't add hacks for every single package manager).

That being said, since this is a Homebrew configuration I guess the brew install script should be capable to configure the bin/crystal binstub to include the brew libdir to the LIBRARY _PATH environment variable? That should avoid the non standard brew installation issues.

[fibric]

And that would help use linuxbrew.sh, IMO.
Because I'm having troubles installing it as stated in https://github.com/Linuxbrew/homebrew-core/issues/716

[halostatue]

@ysbaddaden Pretty much. And maybe the bug should have been filed against the Homebrew formula instead, but as I’ve worked with other languages that _do_ some sort of sibling-directory detection (e.g., if you install into /opt/…, it would find /opt/lib; if you install into /opt/crystal/…, it would find /opt/crystal/lib…).

I don’t actually want a special fix for _Homebrew_, just a sane addition to the sane default behaviour, or something that package managers can write to for configuration whether they use a precompiled version or compile it themselves (which is why I suggested the configuration option at the beginning of the post). Something that might be expressed sort of as:

def lib_flags
  library_path = [ Filename.expand_path("../../lib", $0), "/usr/local/lib", "/usr/lib" ]
  library_path.delete_if { |e| !Filename.exist?(e) }
  …

The code is more or less how I’d write it in Ruby (such that if $0 == "~/.brew/bin/crystal", so expanding ../../lib, would be ~/.brew/lib/). I’ve also written a minor bug fix into your code there, because the resulting list should be -L$(brew --prefix)/lib -L/usr/local/lib -L/usr/lib so that /usr/local/lib is searched _before_ /usr/lib, which is the opposite of what you have now and not what most people would expect (if I install a different version of libxml2 in /usr/local/ than is in /usr/lib, it will be properly found).

BTW, I _think_ this could have been avoided if bdw-gc (used in Homebrew) had used pkg-config because you have code for handling that, and it looks pretty sane, too.

[ysbaddaden]

Expanding ../../lib from $0 is a bold assumption. It should consider $0 to be a link, but the bigger problem is that in official tarballs and packages we distribute, it would point to /opt/crystal/embedded/bin/crystal and the linker won't find any libraries except for the embedded ones, that the binstub already added to LIBRARY_PATH.

pkg-config should help, but it doesn't seem to be used. Since this is a special directory installation, maybe it requires ~/.brew/lib/pkgconfig to be appended to PKG_CONFIG_PATH?

[masukomi]

just to add to this:
I'm also using homebrew. Using Crystal 0.18.7 (2016-07-27) I compiled with

crystal build path/to/file.cr --link-flags "/usr/local/opt/libevent/lib/libevent.a /usr/local/opt/libpcl/lib/libpcl.a /usr/local/opt/bdw-gc/lib/libgc.a"

gave the binary to another mac user and they got

dyld: Library not loaded: /usr/local/opt/bdw-gc/lib/libgc.1.dylib
  Referenced from: /Users/sskates/projects/utilities/changelog_manager
  Reason: image not found

[jhass]

it'd be -static and -l before each path.

[ysbaddaden]

@jhass specifying the static libraries should tell the linker to _use_ them directly so it never has to search for shared libraries, shouldn't it?

[jhass]

I'm not 100% sure, I don't trust it anymore about that. But it won't replace the -lfoos Crystal inserts anyway, so the dynamic links would happen either way, no?

[joshgoebel]

Not sure if this is related but when trying to install Crystal from Homebrew I get this:

./bin/crystal build --release  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib                   │src
dyld: Library not loaded: /usr/local/opt/bdw-gc/lib/libgc.1.dylib                                                             │t
  Referenced from: /private/tmp/crystal-lang-20170124-40239-k8zgde/crystal-0.20.5/boot/embedded/bin/crystal                   │test
  Reason: image not found                                                                                                     │test.html
/private/tmp/crystal-lang-20170124-40239-k8zgde/crystal-0.20.5/boot/bin/crystal: line 102: 40380 Abort trap: 6           "$INS│tpow
TALL_DIR/embedded/bin/crystal" "$@"                                                                                           │webdav
make: *** [.build/crystal] Error 134

My brew is also in .brew and this has caused no issues in the past with anything else. Played around with DYLD_FALLBACK_LIBRARY_PATH but not sure where I'd need to put it in the brew for it to work.

[mohd-akram]

I'm also having trouble with this. I would like to link Crystal against /opt/local/lib for MacPorts (I have LIBRARY_PATH set to it), but it still links against /usr/lib. Perhaps Crystal should read LIBRARY_PATH from the environment and fallback to the hardcoded values?

[Overload119]

Has there been any progress on this?

[2018-04-14 08:07:03][ERROR: action.script] dyld: Library not loaded: /usr/local/opt/bdw-gc/lib/libgc.2.dylib

I receive a similar error when trying to pass this binary (which uses openssl) to someone else.

[Birch-san]

There are a few problems being discussed here, so I'll weigh in on this one:

when trying to pass this binary (which uses openssl) to someone else:

dyld: Library not loaded: /usr/local/opt/bdw-gc/lib/libgc.2.dylib

====

I've written a script that you can use upon a dynamically-linked Crystal binary, to find all its environment-specific dynamic dependencies and copy them to a lib folder.
It then rewrites the binary and its dylibs into relative links.

Thus, you can use this to make portable distributions that you can give to your friends. They won't need to install brew dependencies or anything.

https://gist.github.com/Birch-san/e84cfa3b93ffa104af2bd9a047d14109

The program we'll be preparing for distribution:

# write some trivial Crystal source code into hello.cr
echo 'puts "Hello world!"' > hello.cr
# build a Crystal binary with dynamic linkage
crystal build hello.cr

# try it out
./hello
Hello world!

Before we make our binary portable, let's take some observations:

# look at all those dynamic links pointing into /usr/local
otool -L hello
hello:
    /usr/local/opt/bdw-gc/lib/libgc.1.dylib (compatibility version 5.0.0, current version 5.3.0)
    /usr/local/opt/libevent/lib/libevent-2.1.6.dylib (compatibility version 7.0.0, current version 7.2.0)
    /usr/lib/libpcre.0.dylib (compatibility version 1.0.0, current version 1.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
    /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)

Prepare hello for distribution:

./make_portable.sh hello

After making our binary portable:

# dynamic links are now specified relative to a runtime search path
otool -L hello lib/*.dylib
hello:
    @rpath/libgc.1.dylib (compatibility version 5.0.0, current version 5.3.0)
    @rpath/libevent-2.1.6.dylib (compatibility version 7.0.0, current version 7.2.0)
    /usr/lib/libpcre.0.dylib (compatibility version 1.0.0, current version 1.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
    /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
# and we've also fixed up the links in the local copies of the dylibs
lib/libcrypto.1.0.0.dylib:
    @rpath/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
lib/libevent-2.1.6.dylib:
    @rpath/libevent-2.1.6.dylib (compatibility version 7.0.0, current version 7.2.0)
    @rpath/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
lib/libgc.1.dylib:
    @rpath/libgc.1.dylib (compatibility version 5.0.0, current version 5.3.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)

# we've told our hello binary to expand @rpath at runtime to @loader_path/lib
# and @loader_path is the directory from which ./hello executes
# watch how it expands to point to a local dependency
DYLD_PRINT_RPATHS=1 ./hello
RPATH successful expansion of @rpath/libgc.1.dylib to: dist/lib/libgc.1.dylib
RPATH successful expansion of @rpath/libevent-2.1.6.dylib to: dist/lib/libevent-2.1.6.dylib
RPATH successful expansion of @rpath/libcrypto.1.0.0.dylib to: dist/lib/libcrypto.1.0.0.dylib
Hello world!

# more generally, we can see all dyld loads (not just ones based on expanding @rpath)
DYLD_PRINT_LIBRARIES=1 ./hello
dyld: loaded: dist/hello
dyld: loaded: /usr/lib/libpcre.0.dylib
dyld: loaded: dist/lib/libgc.1.dylib
dyld: loaded: /usr/lib/libSystem.B.dylib
dyld: loaded: dist/lib/libevent-2.1.6.dylib
dyld: loaded: /usr/lib/libiconv.2.dylib
…
dyld: loaded: dist/lib/libcrypto.1.0.0.dylib
Hello world!

=====

My script makes one big assumption, which is: it's bad to link to libraries in /usr/local (and then distribute that binary). In other words: I try to repair links which point into the brew's default library location.

But it sounds like some people have brew installed in locations other than /usr/local, or are linking to things in /usr/lib.
So, get_env_specific_direct_dependencies () may need some refining, (or tuning for your environment).

[Birch-san]

The root of the problem in my opinion is that libraries distributed/compiled by brew are created with an absolute path as their install name.

Let's take a look at one of the libraries that Crystal binaries typically link against, libgc.dylib:

# ask libgc.dylib what its install name is (the LC_ID_DYLIB load command):
otool -D /usr/local/Cellar/bdw-gc/7.6.6/lib/libgc.dylib
/usr/local/Cellar/bdw-gc/7.6.6/lib/libgc.dylib:
/usr/local/opt/bdw-gc/lib/libgc.1.dylib

Its install name is an absolute path, /usr/local/opt/bdw-gc/lib/libgc.1.dylib. So this is the path that gets written into our Crystal binary's LC_LOAD_DYLIB command when we dynamically link to libgc.dylib.
Works fine if people install our Crystal application via brew, but kills any other distribution method.

Really it'd be nice if brew could distribute/build libraries such that they had relative install names like @rpath/libgc.1.dylib. And that's easy; it can be done trivially at build-time (gcc -install_name '@rpath/whatever.dylib'), or post-build using install_name_tool -id '@rpath/whatever.dylib'.

@rpath-relative install names would mean that all brew libraries would link _to each other_ using relative paths, and that our Crystal binary could also link to any brew library using an @rpath-relative path.

Our Crystal binary can choose what we want to expand @rpath to (this is a _list_ of expansions, so it gives you a lot of flexibility), and with this paradigm our only distribution job would be to ensure that we've copied libraries somewhere on the rpath, and included that folder in our release.

[RX14]

@Birch-san Have you made an issue on homebrew upstream? This sounds like something they can fix or investigate.

Perhaps someone can patch the homebrew formula to always provide the correct LIBRARY_PATH to user too. Or at least start the discussion with the homebrew devs about how they configure the environment. I don't have OSX so it's hard for me to contribute much to homebrew...

Anyway, thanks for the super detailed analysis marisa-san!

[Birch-san]

@RX14 yes, I think I've seen this happen enough times that it's worth raising with Homebrew.
Issue opened:
https://github.com/Homebrew/brew/issues/4371

[Birch-san]

Perhaps someone can patch the homebrew formula to always provide the correct LIBRARY_PATH to user too. Or at least start the discussion with the homebrew devs about how they configure the environment. I don't have OSX so it's hard for me to contribute much to homebrew...

Ah, this brings us back to the original topic. OP has a _build-time_ linking failure. this is actually completely unrelated to the dyld: Library not loaded problem (and the solution I proposed to that), which is a _run-time_ dynamic linking failure (caused by distributing the software in a non-portable way).

My thoughts on the build-time environment problem:

====

It's weird that the crystal linker is hard-coded with a few favourite library search paths. What if you disagreed with the order of precedence? What about Windows support?

I suspect the reason you default to these is because you're guessing where libgc, libevent and libcrypto were installed.

There's a cheat: you could read your own load commands:

# find the directory containing the libgc dylib to which crystal is linked
otool -L `which crystal` | awk '/\/libgc\./ {print $1}' | awk -F'/' 'sub(FS $NF,x)'

Doing it recursively is harder (and necessary to find libcrypto, which is a dependency of libevent).

Or you could ask pkg-config (though that may need its env configuring, which gives you the same problem).

Or you could ship those libraries _inside_ crystal's directory (so you can find them relative to a known location).

Or you could say there's nothing crystal-specific about those libraries, so really the user should tell you where they're installed.

If brew exists, then adding -L$(brew --prefix)/lib automatically might not be completely insane. But beware: it's hard for the user to remove that if they dislike it.

====

I wouldn't recommend editing LIBRARY_PATH. It's an env var, so it affects the whole process tree. When really you just want to talk to one subprocess. --link-flags -L has a much clearer guarantee of "which subprocesses will this be passed to". More importantly: LIBRARY_PATH has lower precedence than -L flags, so you can't use it to say "prefer this nightly build"

In any case, it's best not to create a permanent "environment management" burden for the user.

====

In a properly configured system, crystal shouldn't have to specify any linker paths at all

Build is most sane when it's configured in a project-specific manner, rather than being sensitive to environment. Relying on environment gets non-deterministic in ways that are hard to catch — i.e. when the same environment is shared (and updated) by a variety of projects, or when multiple versions of the same library exist in your search path.

So, I wouldn't say that "your system is only correctly configured once your LIBRARY_PATH is populated". I'd prefer an empty LIBRARY_PATH.

NixOS modifies LIBRARY_PATH in it's linker wrapper to fix this, as it should.

Okay, interesting. It could be a harsh reality of developing wrappers: environment is a more convenient way to affect subprocesses than building an API and routing the options.

====

I like the idea of a ~/.crystal-config. And maybe project-specific overrides (at ./.crystal-config). I think it actually solves all our problems:

• sensible defaults can be instantiated here instead of being hardcoded
• user doesn't need to pass in explicit flags on every build

To tie it all together: I reckon the brew formula should create for you a ~/.crystal-config, which specifies one default library search path: $(brew --prefix)/lib.

Other platforms could instantiate it with /usr/lib and /usr/local/lib as a "reasonably sensible" effort.

[RX14]

I don't know why the crystal compiler has any hardcoded library paths either, it looks like a hack to me. Crystal really shouldn't have to worry about the library path or where libraries are at all. We should be able to tell the system linker what libraries we want linked, either through using pkg-config or -lfoo, and which ones we want to be static/dynamic.

I consider it not crystal's job to tell the system linker where to find stuff. If homebrew installs libraries to a path which is nonstandard, it should be homebrew that works out how to inform the linker. So homebrew absolutely should add to LIBRARY_PATH. And it should do it globally, in /etc/profile or wherever, not just in a wrapper for crystal. But I think the homebrew developers wont accept that for whatever reason, so a wrapper specifically for crystal seems the way forward.

A global crystal config file is weird and unneccesary. Does the linker have a global config file? The C compiler? Make? No, they get their configuration from the environment, like has always been done. And why should crystal be different. Crystal shouldn't try and reinvent the wheel, it should try and be as similar as traditional C compilers in operation as possible. That makes crystal much much easier to understand for sysadmins, and distro packagers. These kinds of people shouldn't have to learn how crystal is special, at least not more than possible.

[matthewmcgarvey]

Just ran into this today. I thought it would be cool to distribute Alfred workflows written in Crystal

[rdp]

Seems working here, what's a test file (or steps) that repro's it for me? Thanks.