Rust: Bootstrap fails due to system LLVM libffi link error

[MagaTailor]

Trying to bootstrap rust on aarch64 Linux using --llvm-root fails:

error: linking with `cc` failed: exit code: 1
note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc.0.o" "-o" "aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc" "-Wl,--gc-sections" "-pie" "-Wl,-O1" "-nodefaultlibs" "-L" "/usr/lib/llvm-3.8/lib" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_driver-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_passes-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_borrowck-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_lint-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_resolve-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_typeck-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_plugin-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_privacy-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_trans-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_platform_intrinsics-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_incremental-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "syntax_ext-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_mir-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_const_eval-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_metadata-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_save_analysis-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_const_math-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "getopts-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "flate-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_back-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rbml-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_data_structures-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "syntax-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "serialize-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "log-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "term-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "fmt_macros-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "arena-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "rustc_llvm-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "graphviz-fe3cdf61" "-L" "../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-l" "std-fe3cdf61" "-l" "rt" "-l" "dl" "-l" "tinfo" "-l" "pthread" "-l" "z" "-l" "m" "-l" "stdc++" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "util" "-Wl,-rpath,$ORIGIN/../lib" "-Wl,-rpath,/usr/local/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,--enable-new-dtags" "-s" "-l" "compiler-rt"

note: ../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_float'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint64'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_pointer'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint32'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_void'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint8'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_call'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_sint16'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_type_double'
../aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_llvm-fe3cdf61.so: undefined reference to `ffi_prep_cif'
collect2: error: ld returned 1 exit status

error: aborting due to previous error
../mk/target.mk:220: recipe for target 'aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc' failed
make: *** [aarch64-unknown-linux-gnu/stage0/lib/rustlib/aarch64-unknown-linux-gnu/bin/rustc] Error 101

For some reason the correct llvm configuration is not being detected and -lffi has to be passed manually for the bootstrap to succeed.

[brson]

In-tree, we configure LLVM without -lffi so maybe there's a discrepancy when we encounter an LLVM with it compiled in.

[MagaTailor]

Yup, the LLVM I was bootsrapping against was clearly built with LLVM_ENABLE_FFI.

[jethrogb]

Running into the same isse. Shouldn't llvm-config output this requirement?

Btw for anyone reading this: a quick fix is to add:

#[link(name = "ffi")] extern {}

to src/librustc_llvm/lib.rs.

[japaric]

I'm also experiencing this on x86_64 Ubuntu 16.04 using the llvm-dev package and --llvm-root.

[MagaTailor]

The title should be edited then, and btw, the quickest non-permanent solution looks like this:
RUSTFLAGS='-C link-args=-lffi'

[japaric]

It seems this is a duplicate of #29694

[japaric]

Recently, I have been annoyed by this a lot. I've been boostraping rustc against system llvm a lot in different Docker containers and I have to remember to apply @jethrogb patch/fix to every rust branch/worktree I work on and then I have to be careful to not commit this fix to the branch I'm working on. :disappointed:

Could we simply commit @jethrogb's fix into master? I tested bootstrapping rust against our llvm fork with that fix merged into master (5a5736db916ac30ca67945bbf0aee41ced1fcf05) and at least, on Linux, it bootstraped without problems.

P.S. @petevine's RUSTFLAGS solution didn't work for me.

[MagaTailor]

@japaric That's all there's to it, and it definitely works for the Makefiles bootstrap.

[alexcrichton]

This is of course not intentional, and fixes are always welcome! The trick here is detecting whether we're building against an LLVM that needs libffi. For example the in-tree LLVM doesn't and we shouldn't link to it as a result.

[cuviper]

FWIW, Fedora's LLVM is built with FFI, but this is usually fine because libLLVM.so links to libffi.so itself. However, with llvm-static.rpm installed, llvm-config starts giving static link options (but not -lffi), and ldd -r librustc_llvm-*.so indeed shows a few undefined symbol: ffi_....

[cuviper]

IOW I blame llvm-config, not that blame solves anything.
(I also don't like that it doesn't let me _choose_ shared linkage even when static libraries are present...)

[userwithuid]

@cuviper Regarding 'choose shared over static llvm':
If I understand correctly, since 3.9 llvm-config will prefer the combo-dynlib by default over any static libs present. And even if you don't have the combo, but say shared _and_ static component libs installed, there are now options to specifically choose --link-static or --link-shared.

Relevant llvm commit explains it: https://github.com/llvm-mirror/llvm/commit/93083d498658f4c6da98e8950bd9f58c76080299

Doesn't fix this bug here, external llvm + ffi enabled + intentionally linking llvm static libs still triggers it, just thought it would be nice to know.

(BTW, the ffi-issue seems to be sort of a known shortcoming of llvm-config --system-libs. It only statically lists the LLVMSupport sysdeps, but ffi is the (currently only?) sysdep from another component - LLVMExecutionEngine)

[cuviper]

@userwithuid Thanks, I wasn't aware of that! I'll try that out, and maybe we should expose those explicit link options somehow too.

[ishitatsuyuki]

Please tell me the current situation; rustbuild is enabled by default, the old workarounds may not work.

[MagaTailor]

Yeah, last time I tried it didn't even build due to some completely unfamiliar link errors. (edit: I meant exactly this)

If it's still a problem in the new build system, please open a more specific issue.

[ishitatsuyuki]

I'm quite sure that this is still a problem; please don't blindly close it.

[jethrogb]

Yes this should remain open. @MagaTailor I don't understand what you mean, this issue is quite specific.

[hussamT]

I went ahead and filed a new bug 39880.

[hussamT]

@csmoe Is this an issue again in post 1.31.1 releases? I have version 1.31.1 installed and did not see this issue when compiling rust against system llvm 7.0.1.

[cuviper]

@hussamT IIRC it was only an issue with linking external LLVM _statically_, and even then only if that build is configured with -DLLVM_ENABLE_FFI:BOOL=ON. At least on Fedora rawhide, I still get:

$ rpm -q llvm
llvm-7.0.1-1.fc30.x86_64
$ llvm-config --link-static --system-libs
-lz -lrt -ldl -ltinfo -lpthread -lm

... missing -lffi, even though some of the static libs do use it. But this is really an LLVM problem, as noted in https://github.com/rust-lang/rust/issues/34486#issuecomment-257139448, so I'm not sure we need this Rust issue open except as a "known issue, see elsewhere" tracker.

Our official Fedora rust builds are linked dynamically (--enable-llvm-link-shared), and this is fine.