Rust-bindgen: Undefined references for native library

[fpagliughi]

I'm attempting to get the stock bindgen tutorial working from:
https://github.com/fitzgen/bindgen-tutorial-bzip2-sys

I have several Linux development machines running various flavors of Mint 17 & 18 (Ubuntu 14.04 and 16.04). The weird thing is that everything works fine on one machine, but on _all_ the others machines, the linker is reporting undefined references in the native library whenever I try to create an executable.

This can be seen in an attempt to build and then run the tests for the tutorial:

$ git clone https://github.com/fitzgen/bindgen-tutorial-bzip2-sys.git
Cloning into 'bindgen-tutorial-bzip2-sys'...
remote: Counting objects: 28, done.
remote: Total 28 (delta 0), reused 0 (delta 0), pack-reused 28
Unpacking objects: 100% (28/28), done.
Checking connectivity... done.

$ cd bindgen-tutorial-bzip2-sys/
$ cargo build -j4
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading lazy_static v0.2.9
   Compiling void v1.0.2
   Compiling libloading v0.4.2
   Compiling vec_map v0.8.0
   Compiling log v0.3.8
   Compiling unicode-xid v0.0.4
   Compiling term v0.4.6
   Compiling bitflags v0.9.1
   Compiling glob v0.2.11
   Compiling rustc-serialize v0.3.24
   Compiling libc v0.2.31
   Compiling lazy_static v0.2.9
   Compiling peeking_take_while v0.1.2
   Compiling cfg-if v0.1.2
   Compiling strsim v0.6.0
   Compiling unicode-width v0.1.4
   Compiling regex-syntax v0.4.1
   Compiling utf8-ranges v1.0.0
   Compiling ansi_term v0.9.0
   Compiling bitflags v0.8.2
   Compiling unreachable v1.0.0
   Compiling term_size v0.3.0
   Compiling memchr v1.0.1
   Compiling atty v0.2.2
   Compiling clang-sys v0.19.0
   Compiling thread_local v0.3.4
   Compiling textwrap v0.8.0
   Compiling aho-corasick v0.6.3
   Compiling nom v3.2.0
   Compiling clap v2.26.2
   Compiling syntex_pos v0.58.1
   Compiling regex v0.2.2
   Compiling syntex_errors v0.58.1
   Compiling cexpr v0.2.2
   Compiling syntex_syntax v0.58.1
   Compiling env_logger v0.4.3
   Compiling aster v0.41.0
   Compiling quasi v0.32.0
   Compiling syntex v0.58.1
   Compiling quasi_codegen v0.32.0
   Compiling bindgen v0.26.3
   Compiling bindgen-tutorial-bzip2-sys v0.1.0 (file:///home/fmp/tmp/bindgen-tutorial-bzip2-sys)
    Finished dev [unoptimized + debuginfo] target(s) in 65.0 secs

$ cargo test
   Compiling bindgen-tutorial-bzip2-sys v0.1.0 (file:///home/fmp/tmp/bindgen-tutorial-bzip2-sys)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps/bindgen_tutorial_bzip2_sys-845fbd2219e8ea65.0.o" "-o" "/home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps/bindgen_tutorial_bzip2_sys-845fbd2219e8ea65" "/home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps/bindgen_tutorial_bzip2_sys-845fbd2219e8ea65.crate.allocator.o" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps" "-L" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "bz2" "-Wl,-Bstatic" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libtest-30d4d638be038598.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgetopts-c58b62d995849f6d.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libterm-359c3de19d4def40.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-29fb1fb52cf57377.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9cbadc6554202be9.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-42efcbbdeb607d44.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-618c266cf9124966.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_system-3c10208cdd7e61cb.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-16f3b02b9a976b94.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-bc70b4efeaeb398c.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_unicode-a0ad42dc8f5856aa.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-32f648f7f7567c6c.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-21491ce3d14f1ef2.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-b9713bd7f605c0b2.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util"
  = note: /home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps/bindgen_tutorial_bzip2_sys-845fbd2219e8ea65.0.o: In function `bindgen_tutorial_bzip2_sys::tests::round_trip_compression_decompression':
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:21: undefined reference to `BZ2_bzCompressInit'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:38: undefined reference to `BZ2_bzCompress'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:49: undefined reference to `BZ2_bzCompressEnd'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:58: undefined reference to `BZ2_bzDecompressInit'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:74: undefined reference to `BZ2_bzDecompress'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:86: undefined reference to `BZ2_bzDecompressEnd'
          clang: error: linker command failed with exit code 1 (use -v to see invocation)


error: aborting due to previous error

error: Could not compile `bindgen-tutorial-bzip2-sys`.

To learn more, run the command again with --verbose.

I can't figure out why it's working on one computer but none of the others.
Thanks

[fitzgen]

Is bzip2 installed on the system? And is the shared library on the system's usual search path for shared libraries?

Aside: @emilio do you think we should move the tutorial into this repo to test it for breakage in CI and keep it up to date and all that?

[fpagliughi]

Yes, the dev package for bzip2 is installed via the package manager, apt-get.

$ apt list --installed | egrep '(bzip2|bz2)'
bzip2/xenial,now 1.0.6-8 amd64 [installed]
libbz2-1.0/xenial,now 1.0.6-8 amd64 [installed]
libbz2-dev/xenial,now 1.0.6-8 amd64 [installed]

and I can link to it with a C program:

$ clang -o bz2tst bz2tst.c -lbz2

[emilio]

Aside: @emilio do you think we should move the tutorial into this repo to test it for breakage in CI and keep it up to date and all that?

That sounds fine to me.

[fpagliughi]

It appears that this may be due to the #[link_name = "..."] attributes which are being specified for each of the native functions in the bindings.rs file, like:

extern "C" {
    #[link_name = "_Z18BZ2_bzCompressInit"]
    pub fn BZ2_bzCompressInit(strm: *mut bz_stream,
                              blockSize100k: ::std::os::raw::c_int,
                              verbosity: ::std::os::raw::c_int,
                              workFactor: ::std::os::raw::c_int)
     -> ::std::os::raw::c_int;
}

If I manually remove them or comment them all out, then the tests seem to link and run properly, like:

extern "C" {
    //#[link_name = "_Z18BZ2_bzCompressInit"]
    pub fn BZ2_bzCompressInit(strm: *mut bz_stream,
                              blockSize100k: ::std::os::raw::c_int,
                              verbosity: ::std::os::raw::c_int,
                              workFactor: ::std::os::raw::c_int)
     -> ::std::os::raw::c_int;
}

Why is it mangling the names like that?
Is it required by some other compiler or operating system?
Is there a way to turn it off, or conditionally specify it on targets where it's necessary?

Thanks

[lilianmoraru]

Yep, same issue here, also because of wrong link_name.
The interesting thing is that at home it works, at work it doesn't...
Maybe has to do with the version of clang that I use(and the one that Rust uses)?
With bindgen(I've written the bindings by hand before, and it works just fine, but for maintainability I ported the code to bindgen) it was completely failing to compile(not finding stddef.h) when I did not have clang installed.

At home I have clang-6.0, at work clang-4.0(the one I installed after it was failing to compile because of the stddef.h header).

[fitzgen]

Maybe has to do with the version of clang that I use(and the one that Rust uses)?

Rust doesn't use libclang (although it is using LLVM) -- I think that is a red herring.

bindgen is definitely affected by what version of libclang is being used, however. Right now, we test with 3.8, 3.9, and 4.0. 5.0 has been known to work well in the past, but no one has taken the time to set up our CI with it yet. Last I heard 6.0 was still trunk, has that changed?

The other thing to double check is that you have the LIBCLANG_PATH env var properly set, to be sure that clang-sys is using the correct libclang.

[fpagliughi]

OK. It looks like this might be the root cause of the issue.

All of my Linux machines have multiple versions of clang, including 3.6, 3.8, and 3.9. Even the machine that "works" has v3.6, but for some reason it... works.

The clang-sys README states that it supports Cargo features like "clang_3_5", "clang_3_6", and so on up to "clang_5_0" to force it to us the specified version or newer, and that:

If you do not enable one of these features, the API provided by libclang 3.5 will be available by default.

So perhaps it is (somewhat inconsistently?) seeking the oldest libclang that it can find?

I tried to see if I could set the Cargo features, but had no luck. Admittedly I have no idea what I'm doing.

But if I set the LIBCLANG_PATH to v3.9, that appears to work, like:

LIBCLANG_PATH=/usr/lib/llvm-3.9/lib cargo build

[fpagliughi]

Yes, it appears that the old clang version is causing problems, although somewhat inconsistently which made it difficult to diagnose. This issue appears to be a repeat of #528, and possibly a few others.

Setting LIBCLANG_PATH seems the correct fix, but this places a burden on users with a silent failure when things go wrong.

There is even a work-around already in place for older clang's in the bindgen::Builder to fix this: trust_clang_mangling(false) as in:

let bindings = bindgen::Builder::default()
    .trust_clang_mangling(false)
    .header("wrapper.h")
    .generate()
    .expect("Unable to generate bindings");

I hope this is harmless if we're wrapping a straight C library, which shouldn't require mangling?

I guess this issue can be closed, but it might be nice for bindgen to issue a warning on build if it detects a clang library that is unsupported.

Thanks for the help and this awesome tool.

[fitzgen]

libclang versioning is tricky, but we have hopes of eventually side-stepping the issue: https://github.com/rust-lang-nursery/rust-bindgen/issues/918

I guess this issue can be closed, but it might be nice for bindgen to issue a warning on build if it detects a clang library that is unsupported.

We do have a warning, but only in the executable, not in the library: https://github.com/rust-lang-nursery/rust-bindgen/blob/master/src/main.rs#L46

Would love to receive a PR that moves it somewhere shared by both the executable and library.