Rust: Linking issue with Rust 1.37.0

[vitvakatu]

See build log.

It looks like name mangling doesn't work correctly when linking with a dynamic library which is also depends on libstd.

The application structure is the following:

1. Large dylib crate java_bindings, containing 400+ dependencies and is dynamically linked to libstd.dylib.
2. exonum-java binary application, depends on java_bindings and is dynamically linked to libstd.dylib.

The compilation works well on Rust 1.36.0, but doesn't work on 1.37.0 and nightly versions (such as rustc 1.39.0-nightly (c6e9c76c5 2019-09-04)).

To reproduce:

git clone https://github.com/exonum/exonum-java-binding
cd exonum-java-binding/exonum-java-binding/core/rust/exonum-java
rustup default nightly
cargo build

Sorry for such heavy project, couldn't reproduce on an easier example.

[popzxc]

This bug is can also be reproduced with a small project: link (be sure to use a linked revision, since project is under active development, and upstream may change).

If I try to compile examples/node with 1.38.0, I get errors similar to the listed in the issue.
With 1.36.0 everything works fine.

[AlexKornitzer]

Just finished bisecting and the culprit commit affecting my project is https://github.com/rust-lang/rust/commit/185dcebbff7f3c18fb94be82f76bdfd620cfb096 which appears to be the same changes affecting #66265 and #65610. @alexcrichton is there any chance we can revert this? or is there a way to move this issue forwards?

[alexcrichton]

Sorry I don't know much about that change, I didn't author it, I'd recommend bringing it up on the PR and/or with the release team to figure out a revert.

[AlexKornitzer]

Thank you, appologies for incorrectly @'ing you, I shall do just that.

[michaelwoerister]

Is there are strong reason for using Rust dylibs instead of rlibs or cdylibs for these projects? Rust dylibs are rather underspecified and the concrete set of symbols exported from them is an implementation detail.

[popzxc]

Yes, if the built library should be usable by both Rust and other programming language.

[AlexKornitzer]

@michaelwoerister so I need to use dylibs over rlibs and cdylibs for a couple of reasons:

1. Can't use rlibs as they are statically linked and this takes too much space especially as I use a plugin system and hence have a shared library of common code.
2. From what I understand I cannot use cdylibs because I need to expose rust primitives such as traits, etc. and these cannot be exposed through a C ABI. (I also acknowledge that plugins are limited due to lack of a stable dylib abi but I am happy with that limit atm)

Very happy to be corrected if any of the above is incorrect.

[michaelwoerister]

@AlexKornitzer It sounds like your project should not be affected by the PR in question. Can you share a reproduction?

[AlexKornitzer]

@michaelwoerister annoyingly it is private code, let me see if I can reproduce it in a generic project.

[vitvakatu]

Btw, we can confirm that reverting the PR in question with locally build Rust compiler solves the problem on Rust 1.37 and Rust 1.39.

cc @skletsun

[AlexKornitzer]

@michaelwoerister here is a relatively streamlined build that will cause linker errors. Commenting out the Consul crate (https://github.com/AlexKornitzer/dylib-errors/blob/1ab7cdbddebec3353a80a40d7b8ae0899f6ac581/shared/src/lib.rs#L2) will cause the majority of the errors to go away (no idea why!), I believe the NNG errors are due to related C issues, but am unsure. I will make a branch without NNG and see if the ones caused by Consul go away.

The below builds on 1.36.0 but not above:
https://github.com/AlexKornitzer/dylib-errors

Build output:
https://gist.github.com/AlexKornitzer/0f56318716048b992e6ac61f8e2992da

[michaelwoerister]

Thank you, @AlexKornitzer! That looks like a reasonably small reproduction case. If you can reduce it further, as you mentioned, that would be even better. Let's nominate this issue for priority assignment. Looks like it might be a proper regression.

[AlexKornitzer]

Right @michaelwoerister, that has now been done so from looking over the issues, I have a C based linking error and a Rust based error, which I have split into two branches:

1. error/nng: This appears to be a C based linking error, maybe an issue with how the NNG library is exposing its binding? (https://gitlab.com/neachdainn/nng-rs/blob/master/src/options/private.rs#L184). I assume @Zoxc would be able to trivially answer this?

2. error/consul: This appears to be a Rust based linking error, I have no idea what the root cause could be but most of the undefined references seem to be to core rust functions, would not even know what to guess for that one :P

Hopefully these are reduced enough, the consul one is fairly large in regards to deps but as I am not sure what is causing it I have no idea how to isolate that further.

Thanks again for looking into this.

[AlexKornitzer]

@michaelwoerister has identified the issue for branch 2 (error/consul) and has suggested opening a new issue that is more targeted to its root cause, hence #67276

[AlexKornitzer]

So I have tried @Zoxc solution as mentioned here (https://github.com/rust-lang/rust/issues/65610#issuecomment-545680868) for branch 1 (error/nng) and the appears to work. In https://gitlab.com/neachdainn/nng-sys/blob/master/src/bindings.rs#L665 I added #[link(name = "nng")] and the link errors I had there went away, so is it now intended behaviour that all binding libraries built statically must decorate their externs in order to support dylibs?

[AlexKornitzer]

As I accidentality high-jacked this issue I just wanted to check to see if either of the above address the original issue, and I can confirm this is the case, it is suffering from the c link issue and the share generics issue.

Building with nightly --release we get:

This leaves us with missing C exports which can be fixed using the link attribute. So if anyone can confirm on how we should be using the link attribute, that would be great :)

[pnkfelix]

triage: this has been prioritized as P-high, it sounds like progress is being made (or at least sub-issues like #67276 are being filed), so I'm going to remove the nomination label

[AlexKornitzer]

Hey @pnkfelix, where are we at with this? Is anyone able to clarify what should be done for the C linking issue, and/or the progress on #67276 (or is this no longer considered to be a regression)?

[alekseysidorov]

I have refreshed @popzxc bug example:
https://github.com/alekseysidorov/exonum-python-backend/tree/bug_demo

This demonstration works fine in release mode but fails in debug with error.

  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/aleksey/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.1b8nq8hqazx4o4mu.rcgu.o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.1vao7b0rtod5wp1t.rcgu.o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.1ywq4e7nzcoqwhg1.rcgu.o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.320yqv3a1ambp0ox.rcgu.o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.3gr63relyw5f0u48.rcgu.o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.3rso5tov9uvbwd75.rcgu.o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.4bjj1o9bwh9jxsrc.rcgu.o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.4j1xth9tuuqlrtlm.rcgu.o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.4mvhyvubfe3zn1mu.rcgu.o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633.4v3x1v15x4lnl9en.rcgu.o" "-o" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/node-2e56978a8e2d5633" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps" "-L" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/build/backtrace-sys-703ba8c28de722ae/out" "-L" "/usr/lib/x86_64-linux-gnu" "-L" "/usr/lib" "-L" "/usr/lib/x86_64-linux-gnu/" "-L" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/build/clear_on_drop-5818bc3ecf10a8fc/out" "-L" "/home/aleksey/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps" "-lrust_interface" "-Wl,-Bstatic" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libexonum_cli-1c82bb460a10dbdd.rlib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/librpassword-dd59d7d2ea084fe9.rlib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libexonum_supervisor-dc285bfe3a377f0b.rlib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libstructopt-bff0ace8a057a961.rlib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libclap-2e129f80c5d90332.rlib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libvec_map-16d645c47f25e618.rlib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libtextwrap-8d243dca064bb376.rlib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libunicode_width-2a6cfc3f3a68993f.rlib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libstrsim-2f472800f2976f07.rlib" "/home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libansi_term-0716f2697a5e0bd1.rlib" "-Wl,--start-group" "-L" "/home/aleksey/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bdynamic" "-lstd-74488c47a41eb313" "-Wl,--end-group" "-Wl,-Bstatic" "/home/aleksey/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-4ed27d8420cb4abc.rlib" "-Wl,-Bdynamic" "-lsodium" "-lrocksdb" "-lsnappy" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /usr/bin/ld: /home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libexonum_cli-1c82bb460a10dbdd.rlib(exonum_cli-1c82bb460a10dbdd.exonum_cli.ams5x799-cgu.1.rcgu.o): in function `alloc::vec::Vec<T>::extend_desugared':
          /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/liballoc/vec.rs:2115: undefined reference to `core::ptr::write'
          /usr/bin/ld: /home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libexonum_cli-1c82bb460a10dbdd.rlib(exonum_cli-1c82bb460a10dbdd.exonum_cli.ams5x799-cgu.1.rcgu.o): in function `<alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter':
          /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/liballoc/vec.rs:1994: undefined reference to `core::ptr::write'
          /usr/bin/ld: /home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libexonum_cli-1c82bb460a10dbdd.rlib(exonum_cli-1c82bb460a10dbdd.exonum_cli.ams5x799-cgu.14.rcgu.o): in function `std::path::Path::join':
          /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/path.rs:2228: undefined reference to `<&T as core::convert::AsRef<U>>::as_ref'
          /usr/bin/ld: /home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libexonum_cli-1c82bb460a10dbdd.rlib(exonum_cli-1c82bb460a10dbdd.exonum_cli.ams5x799-cgu.15.rcgu.o): in function `exonum::node::NodeConfig::read_secret_keys':
          /home/aleksey/.cargo/git/checkouts/exonum-4bba1b30cad600e3/4b45f0e/exonum/src/node/mod.rs:275: undefined reference to `<&T as core::convert::AsRef<U>>::as_ref'
          /usr/bin/ld: /home/aleksey/Projects/bitfury/exonum-python-backend/target/debug/deps/libexonum_cli-1c82bb460a10dbdd.rlib(exonum_cli-1c82bb460a10dbdd.exonum_cli.ams5x799-cgu.3.rcgu.o): in function `core::ptr::real_drop_in_place':
          /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175: undefined reference to `<std::sync::mutex::Mutex<T> as core::ops::drop::Drop>::drop'

Command to run is:
cargo run -- run-dev -a target/python

P.S. Sorry for huge size of example.

[AlexKornitzer]

So with the dylib part fixed by @michaelwoerister :raised_hands: we are now only left with the issue of linking C code through dylibs.

Originally I thought I could just decorate the extern blocks with #[link(name = "EXAMPLE")] but the file in question is created through bindgen and it intentionally does not set this attr in favour of rustc-link-lib (https://github.com/rust-lang/rust-bindgen/issues/104). So is something strange happening with cargo/rustc or is the way that the binding files are being created incorrect? or is link actually the way to go?

Without insight from the rust team we will not know how we should proceed with this issue.

Again the small isolated example is here: https://github.com/AlexKornitzer/dylib-errors/tree/error/nng and should give the following (latest nightly):

  = note: /sbin/ld: /home/alex/Developer/dylib-errors/target/debug/deps/server-cd4f976a7d66f192.50nccdz6es266d8g.rcgu.o: in function `nng::options::private::HasOpts::setopt_ms':
          /home/alex/.cargo/registry/src/github.com-1ecc6299db9ec823/nng-0.5.1/src/options/private.rs:184: undefined reference to `nng_setopt_ms'
          collect2: error: ld returned 1 exit status

Surely with nng-sys being built statically the above should not happen as the extern blocks should be linked into nng so the server should never need to link against it?

Also the above huge (@alekseysidorov) example is affected by the same issue, in its case with references to rocksdb which is also using bindgen.

[nagisa]

So is something strange happening with cargo/rustc or is the way that the binding files are being created incorrect? or is link actually the way to go?

As per https://github.com/rust-lang/rust/issues/65610 you are required to annotate the extern blocks with #[link(name=...)] of the originating library to get correct behaviour.

[AlexKornitzer]

@nagisa understood, but as these files are the output of bindgen does that mean it needs to change how it outputs extern blocks, as noted above?

[nagisa]

Yes. What bindgen does currently works in most cases but will fail in some other fairly obscure cases and may also lead to various spurious failures that seemingly happen because of unrelated changes.