Rust: Build with pure LLVM toolchain on Linux

Created on 3 Oct 2019  ·  15Comments  ·  Source: rust-lang/rust

Hello,

I'm creating a Linux distribution that does not use GCC and use LLVM/Clang and their libraries by default (libc++, libc++abi, compiler-rt and libunwind). No libstdc++ or gcc installed. The distribution is based on musl as well.

I'd like to build rust with that pure LLVM toolchain and since nobody did that before I'm having few issues and unable to find many resource on this topic. Alpine Linux also build with musl but they still use GCC instead.

What I've tried:

  1. I've downloaded the prebuilt musl image from the official binaries and installed in a temporary location.
wget https://static.rust-lang.org/dist/rust-1.37.0-x86_64-unknown-linux-musl.tar.gz
tar xf rust-1.37.0-x86_64-unknown-linux-musl.tar.gz
cd rust-1.37.0-x86_64-unknown-linux-musl && ./install --prefix= --destdir=/tmp/bootstrap-rust
  1. I've downloaded rust sources and configured the config.toml before build:
wget http://static.rust-lang.org/dist/rustc-1.37.0-src.tar.gz
tar xf rustc-1.37.0-src.tar.gz

The config.toml:

cat > rustc-1.37.0-src/config.toml <<EOF
[build]
rustc = "/tmp/bootstrap-rust/bin/rustc"
cargo = "/tmp/bootstrap-rust/bin/cargo"
python = "/bin/python"
build = 'x86_64-unknown-linux-musl'
host = ['x86_64-unknown-linux-musl']
target = ['x86_64-unknown-linux-musl']
vendor = true

[install]
prefix = ""

[rust]
channel = "stable"
rpath = false
codegen-tests = false
verbose-tests = false
default-linker = "clang"
llvm-libunwind = true

[dist]
src-tarball = false

[llvm]
static-libstdcpp = false
ninja = true
use-libcxx = true
link-shared = true

[target.x86_64-unknown-linux-musl]
cc = clang
cxx = clang++
linker = clang
llvm-config = "/bin/llvm-config"
crt-static = false
EOF
  1. Then trying to build using ./x.py build will abort not finding -lgcc_s (which unsurprisingly, does not exist).
error: linking with `clang` failed: exit code: 1
  |
  = note: "clang" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-L" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.0.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.1.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.10.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.11.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.12.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.13.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.14.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.15.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.2.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.3.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.4.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.5.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.6.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.7.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.8.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.9.rcgu.o" "-o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.813shqno7rs2e7d.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/deps" "-L" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-9a431e54f29d011f.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-541cee8c5b6f056a.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace-9b9843a21c950d24.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace_sys-f4f15dbb47e9d653.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-53359f68302a289b.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-dfa7f1d62fd88f75.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-c6c7bf94cfe1287c.rlib" "/tmp/rustcihQO53/libunwind-c02db0b699a2e957.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-ffad3faca04b6ecf.rlib" "/tmp/rustcihQO53/liblibc-63baa27af2c30c40.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-c55e6587bce48104.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-e2f6ada331b532fd.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-06ad1fc9bd858d00.rlib" "-Wl,--end-group" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-2ec319ee584bfdc0.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc"
  = note: ld.lld: error: unable to find library -lgcc
          clang-8: error: linker command failed with exit code 1 (use -v to see invocation)


error: aborting due to previous error

error: Could not compile `syn`.
warning: build failed, waiting for other jobs to finish...
error: aborting due to previous error

error: Could not compile `proc-macro2`.

What should I do to force linking against libunwind, libc++abi rather than GCC equivalents?

A-LLVM A-rustbuild C-feature-request
Source
picture markand
👍1

Most helpful comment

Configure llvm with -DCLANG_DEFAULT_RTLIB=compiler-rt so that clang will link against libclang_rt.builtin*.a instead of -lgcc or -lgcc_s.

Alternatively, use link option -rtlib=compiler-rt.

Since clang 9, --unwindlib={none,libunwind} (-DCLANG_DEFAULT_UNWINDLIB) is available. This option can be used to avoid -lgcc_s or -lgcc_eh. I don't remember the status before r356508, but with --rtlib=compiler-rt, by default no unwind lib is linked in. If you use clang 8 and somehow some object files require a unwind lib, you may need explicit -Wl,-lunwind.

-DCLANG_DEFAULT_CXX_STDLIB=libc++ if you want to use libc++ by default. On most Linux distributions, the default is libstdc++. libc++abi.so is linked because libc++.so is a linker script (INPUT(libc++.so.1 -lc++abi)) that adds libc++abi.so. Your compiler driver is clang, not clang++, so C++ stdlib will not be linked.

picture MaskRay on 3 Oct 2019
👍3

All 15 comments

Configure llvm with -DCLANG_DEFAULT_RTLIB=compiler-rt so that clang will link against libclang_rt.builtin*.a instead of -lgcc or -lgcc_s.

Alternatively, use link option -rtlib=compiler-rt.

Since clang 9, --unwindlib={none,libunwind} (-DCLANG_DEFAULT_UNWINDLIB) is available. This option can be used to avoid -lgcc_s or -lgcc_eh. I don't remember the status before r356508, but with --rtlib=compiler-rt, by default no unwind lib is linked in. If you use clang 8 and somehow some object files require a unwind lib, you may need explicit -Wl,-lunwind.

-DCLANG_DEFAULT_CXX_STDLIB=libc++ if you want to use libc++ by default. On most Linux distributions, the default is libstdc++. libc++abi.so is linked because libc++.so is a linker script (INPUT(libc++.so.1 -lc++abi)) that adds libc++abi.so. Your compiler driver is clang, not clang++, so C++ stdlib will not be linked.

MaskRay picture MaskRay on 3 Oct 2019
👍3

Thanks for the answer,

Obviously, my LLVM and clang compiler are already built with those options and that's why by default clang++ will link to libc++, libunwind and use compiler-rt by default without any switches.

See:

/tmp # echo "int main() {}" > main.cpp

/tmp # clang++ -v main.cpp
Vanilla Linux clang version 8.0.1 (tags/RELEASE_801/final) (based on LLVM 8.0.1)
Target: x86_64-unknown-linux-musl
Thread model: posix
InstalledDir: /bin
 "/bin/clang-8" -cc1 -triple x86_64-unknown-linux-musl -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model pic -pic-level 2 -pic-is-pie -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /lib/clang/8.0.1 -internal-isystem /bin/../include/c++/v1 -internal-isystem /usr/local/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /lib/clang/8.0.1/include -fdeprecated-macro -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length 151 -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/main-39d04f.o -x c++ main.cpp -faddrsig
clang -cc1 version 8.0.1 based upon LLVM 8.0.1 default target x86_64-linux-musl
ignoring nonexistent directory "/usr/local/include"
ignoring duplicate directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /bin/../include/c++/v1
 /include
 /lib/clang/8.0.1/include
End of search list.
 "/bin/ld.lld" -pie --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib/ld-musl-x86_64.so.1 -o a.out /bin/../lib/Scrt1.o /bin/../lib/crti.o /lib/clang/8.0.1/lib/linux/crtbeginS.o -L/bin/../lib -L/lib -L/usr/lib /tmp/main-39d04f.o -lc++ -lm /lib/clang/8.0.1/lib/linux/libclang_rt.builtins-x86_64.a -lc /lib/clang/8.0.1/lib/linux/libclang_rt.builtins-x86_64.a /lib/clang/8.0.1/lib/linux/crtendS.o /bin/../lib/crtn.o

/tmp # ldd a.out
        /lib/ld-musl-x86_64.so.1 (0x7f5979fe7000)
        libc++.so.1 => /lib/libc++.so.1 (0x7f5979f24000)
        libc++abi.so.1 => /lib/libc++abi.so.1 (0x7f5979ee7000)
        libunwind.so.1 => /lib/libunwind.so.1 (0x7f5979ed7000)
        libc.so => /lib/ld-musl-x86_64.so.1 (0x7f5979fe7000)

What I suspect is that -lgcc_s is probably "hardcoded" in the bootstrap tools and forced while re-building rustc with this bootstrap itself.

markand picture markand on 3 Oct 2019
👍1

What I suspect is that -lgcc_s is probably "hardcoded" in the bootstrap tools and forced while re-building rustc with this bootstrap itself.

Correct, IIRC gcc libs were hardcoded in multiple places in the past (don't really know about how it currently is).
You can replace gcc_s here with unwind but then your toolchain will require compatible libunwind installed in the system https://github.com/rust-lang/rust/blob/ad7c55e1fc55d9af4787b285cec1c64e3480ae84/src/libunwind/lib.rs#L25

mati865 picture mati865 on 3 Oct 2019

For the moment, I'm fooling the build process by creating a fake libgcc_s.so which simply contains:

INPUT(-lunwind)

It's a bit hackish but I went further into the build process now.

markand picture markand on 3 Oct 2019

I got further in the build with the fake libgcc_s.so file and the proposal to change gcc_s in lib.rs file from libunwind. But it looks like something is still not linked to LLVM's libunwind because after stage1 I get:

    Finished release [optimized] target(s) in 8m 14s
Assembling stage1 compiler (x86_64-unknown-linux-musl)
Building stage1 std artifacts (x86_64-unknown-linux-musl -> x86_64-unknown-linux-musl)
warning: rustc-1.37.0-src/src/tools/miri/Cargo.toml: the cargo feature `default-run` is now stable and is no longer necessary to be listed in the manifest
error: process didn't exit successfully: `rustc-1.37.0-src/build/bootstrap/debug/rustc -vV` (exit code: 127)
--- stderr
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_Backtrace: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_DeleteException: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_RaiseException: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetDataRelBase: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetIPInfo: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetLanguageSpecificData: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetRegionStart: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetTextRelBase: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_SetGR: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_SetIP: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetIP: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_FindEnclosingFunction: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_interface-f6372f68ca4c37fc.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_borrowck-7373c9341a309f71.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_mir-5c1c1819075ebd26.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_save_analysis-846e375bc77c7d8e.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_codegen_utils-65d9103bc435188d.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_metadata-64d587906dd5eba7.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_FindEnclosingFunction: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_GetIP: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_Backtrace: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libtest-dc489fd1ade7cf99.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax-29c4688773e43a24.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_target-1fdfcbe13b408e9a.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_errors-bcce3650c698371e.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax_pos-91929928d8d8df07.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libarena-40fb647db2c6c16f.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_data_structures-b3d1b53deed1cc21.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libgraphviz-c924c23f873259cc.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_cratesio_shim-451cc189886af978.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libserialize-ded14d9c4374ae5b.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_lint-cb9cab5a661e0fc4.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_traits-768f999b81f7c152.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_resolve-24af22f041ddbc99.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_privacy-d66f878948e250ee.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_typeck-0c55c94c2fe00bc3.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_plugin-aaf21d589d6f9a96.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_passes-230e54df788c1f06.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_codegen_ssa-ffbc4d353d0a6eba.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_incremental-93604673bdae5cca.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_allocator-61b49b916bd03a67.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax_ext-894cd6a808d68b68.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libfmt_macros-2f4f0c96b22175f2.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_fs_util-eae473008a93e789.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libterm-143f9562b76da2bf.so: _Unwind_Resume: symbol not found
markand picture markand on 3 Oct 2019

@markand could you paste here output of ldd rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so?

mati865 picture mati865 on 3 Oct 2019

There are lot of missing symbols.

# ldd build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so
        ldd (0x7f56a02c9000)
Error loading shared library librustc_interface-f6372f68ca4c37fc.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_borrowck-7373c9341a309f71.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_mir-5c1c1819075ebd26.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_save_analysis-846e375bc77c7d8e.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_codegen_utils-65d9103bc435188d.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_metadata-64d587906dd5eba7.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc-cb39e474659e15c8.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libtest-dc489fd1ade7cf99.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libsyntax-29c4688773e43a24.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_target-1fdfcbe13b408e9a.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_errors-bcce3650c698371e.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libsyntax_pos-91929928d8d8df07.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libarena-40fb647db2c6c16f.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_data_structures-b3d1b53deed1cc21.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libgraphviz-c924c23f873259cc.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_cratesio_shim-451cc189886af978.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libserialize-ded14d9c4374ae5b.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libstd-7ee15cbd53b7b254.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
        libc.so => ldd (0x7f56a02c9000)
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _Unwind_Resume: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _ZN4core9panicking5panic17h8fe75b76845d641eE: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _ZN5alloc5alloc18handle_alloc_error17heeed78525e569dfeE: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _ZN5alloc7raw_vec17capacity_overflow17h7261c2dd34ad30e0E: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: __rust_alloc: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: __rust_alloc_zeroed: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: __rust_dealloc: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: __rust_realloc: symbol not found

It clearly isn't linked to libunwind as readelf shows.

# readelf -d build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so

Dynamic section at offset 0x12e530 contains 38 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [librustc_interface-f6372f68ca4c37fc.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_borrowck-7373c9341a309f71.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_mir-5c1c1819075ebd26.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_save_analysis-846e375bc77c7d8e.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_codegen_utils-65d9103bc435188d.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_metadata-64d587906dd5eba7.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc-cb39e474659e15c8.so]
 0x0000000000000001 (NEEDED)             Shared library: [libtest-dc489fd1ade7cf99.so]
 0x0000000000000001 (NEEDED)             Shared library: [libsyntax-29c4688773e43a24.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_target-1fdfcbe13b408e9a.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_errors-bcce3650c698371e.so]
 0x0000000000000001 (NEEDED)             Shared library: [libsyntax_pos-91929928d8d8df07.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarena-40fb647db2c6c16f.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_data_structures-b3d1b53deed1cc21.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgraphviz-c924c23f873259cc.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_cratesio_shim-451cc189886af978.so]
 0x0000000000000001 (NEEDED)             Shared library: [libserialize-ded14d9c4374ae5b.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstd-7ee15cbd53b7b254.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW
 0x0000000000000007 (RELA)               0x15e68
 0x0000000000000008 (RELASZ)             42552 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          1085
 0x0000000000000017 (JMPREL)             0x204a0
 0x0000000000000002 (PLTRELSZ)           120 (bytes)
 0x0000000000000003 (PLTGOT)             0x12f6f8
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000006 (SYMTAB)             0x238
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0x6810
 0x000000000000000a (STRSZ)              63061 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x4b58
 0x0000000000000004 (HASH)               0x4fa8
 0x000000000000000c (INIT)               0x127328
 0x000000000000000d (FINI)               0x127330
 0x0000000000000000 (NULL)               0x0
markand picture markand on 3 Oct 2019

There are lot of missing symbols.

This is expected, at least when it comes to missing Rust libraries. You'd have to set LD_LIBRARY_PATH to avoid it but it doesn't matter right now.

It clearly isn't linked to libunwind as readelf shows.

I'd expect it to be linked at this point but I can be wrong.

IIRC libunwind doesn't provide _Unwind_Resume on it's own but relies on libgcc or compiler-rt. I assume that's why it's missing.
Can you confirm your libunwind or one of the libs it depends on provides _Unwind_Resume symbol?

I haven't been playing with non-trunk LLVM for a bit but I recall there were issues around libc++abi and libunwind. Hoping that @MaskRay will confirm or deny my guess.

mati865 picture mati865 on 3 Oct 2019

Thanks again for the quick answer, libunwind.so has _Unwind_Resume symbol, yes.

$ nm /lib/libunwind.so.1.0 | grep Unwind_Resume
000000000000bca0 T _Unwind_Resume
000000000000c360 T _Unwind_Resume_or_Rethrow
markand picture markand on 3 Oct 2019

Hello,

This is the list of built libraries that have undefined symbols to LLVM's libunwind, I've tried adding #[link(name = "unwind")] in their respective lib.rs but with no luck. Can I edit their Cargo.toml to force linking to it? I'm not yet confident with Rust tooling yet.

Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_target-1fdfcbe13b408e9a.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_metadata-64d587906dd5eba7.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_mir-5c1c1819075ebd26.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_data_structures-b3d1b53deed1cc21.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_interface-f6372f68ca4c37fc.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_typeck-0c55c94c2fe00bc3.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_FindEnclosingFunction: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_GetIP: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_Backtrace: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_codegen_utils-65d9103bc435188d.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_resolve-24af22f041ddbc99.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_privacy-d66f878948e250ee.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_codegen_ssa-ffbc4d353d0a6eba.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_plugin-aaf21d589d6f9a96.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_cratesio_shim-451cc189886af978.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libtest-dc489fd1ade7cf99.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax_pos-91929928d8d8df07.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_passes-230e54df788c1f06.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_lint-cb9cab5a661e0fc4.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libterm-143f9562b76da2bf.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax_ext-894cd6a808d68b68.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_traits-768f999b81f7c152.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libserialize-ded14d9c4374ae5b.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_allocator-61b49b916bd03a67.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_borrowck-7373c9341a309f71.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_incremental-93604673bdae5cca.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax-29c4688773e43a24.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libgraphviz-c924c23f873259cc.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_fs_util-eae473008a93e789.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libfmt_macros-2f4f0c96b22175f2.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_save_analysis-846e375bc77c7d8e.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libarena-40fb647db2c6c16f.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_errors-bcce3650c698371e.so: _Unwind_Resume: symbol not found
markand picture markand on 16 Oct 2019

I have an theory.
Unless you patched Clang it won't link libunwind by default. In https://github.com/rust-lang/rust/issues/65051#issuecomment-537870171 it worked because it was pulled by libcxx(abi?).
If that's correct then LDFLAGS="-Wl,-lunwind" mentioned by MaskRay should help.

mati865 picture mati865 on 16 Oct 2019

Yes it is pulled by libc++.so. In fact we have the following input file (FreeBSD does something similar with compiler-rt).

# cat /lib/libc++.so
INPUT(libc++.so.1 -lc++abi -lunwind)

I have not seen the LDFLAGS proposal (maybe the message was edited)? Anyway sorry, I'll try immediately :-)

markand picture markand on 16 Oct 2019

So the LDFLAGS didn't work and I've also rebuilt clang with the option to use libunwind by default with no luck. So I've rebuilt musl library to force inclusion of libunwind symbols into it. A bit hackish but I went further... Once I get a native rustc I'll revert my custom libc.so.

Now it fails because of an unmatched function definition.

TARGET = Some("x86_64-unknown-linux-musl")
OPT_LEVEL = Some("2")
HOST = Some("x86_64-unknown-linux-musl")
CXX_x86_64-unknown-linux-musl = Some("clang++")
CXXFLAGS_x86_64-unknown-linux-musl = Some("-ffunction-sections -fdata-sections -fPIC --target=x86_64-unknown-linux-musl")
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("false")
running: "clang++" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=x86_64-unknown-linux-musl" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=x86_64-unknown-linux-musl" "-I//include" "-std=c++11" "-stdlib=libc++" "-D__STDC_CONSTANT_MACROS" "-D__STDC_FORMAT_MACROS" "-D__STDC_LIMIT_MACROS" "-DLLVM_COMPONENT_AARCH64" "-DLLVM_COMPONENT_AMDGPU" "-DLLVM_COMPONENT_ARM" "-DLLVM_COMPONENT_ASMPARSER" "-DLLVM_COMPONENT_BITREADER" "-DLLVM_COMPONENT_BITWRITER" "-DLLVM_COMPONENT_HEXAGON" "-DLLVM_COMPONENT_INSTRUMENTATION" "-DLLVM_COMPONENT_INTERPRETER" "-DLLVM_COMPONENT_IPO" "-DLLVM_COMPONENT_LINKER" "-DLLVM_COMPONENT_LTO" "-DLLVM_COMPONENT_MCJIT" "-DLLVM_COMPONENT_MIPS" "-DLLVM_COMPONENT_MSP430" "-DLLVM_COMPONENT_NVPTX" "-DLLVM_COMPONENT_POWERPC" "-DLLVM_COMPONENT_RISCV" "-DLLVM_COMPONENT_SPARC" "-DLLVM_COMPONENT_SYSTEMZ" "-DLLVM_COMPONENT_WEBASSEMBLY" "-DLLVM_COMPONENT_X86" "-o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage0-codegen/x86_64-unknown-linux-musl/release/build/rustc_llvm-73aad4fac499ea8f/out/../rustllvm/PassWrapper.o" "-c" "../rustllvm/PassWrapper.cpp"
cargo:warning=../rustllvm/PassWrapper.cpp:917:3: error: no matching function for call to 'thinLTOResolvePrevailingInIndex'
cargo:warning=  thinLTOResolvePrevailingInIndex(Ret->Index, isPrevailing, recordNewLinkage);
cargo:warning=  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=//include/llvm/LTO/LTO.h:49:6: note: candidate function not viable: requires 4 arguments, but 3 were provided
cargo:warning=void thinLTOResolvePrevailingInIndex(
cargo:warning=     ^
cargo:warning=1 error generated.
exit code: 1

Is rust 1.37 compatible with llvm 9? I'll try with rust 1.38 instead.

markand picture markand on 17 Oct 2019

Is rust 1.37 compatible with llvm 9? I'll try with rust 1.38 instead.

LLVM 9 support was added in version 1.38.

mati865 picture mati865 on 17 Oct 2019

I've finally built a version using a patched version of my musl. Now it builds fine itself with my native compiler.

Thanks for the help, though I think something should be changed to bootstrap Rust easier when “non traditional” toolchains are used but I have no knowledge in rust internals to help unfortunately.

markand picture markand on 28 Oct 2019
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

wthrowe picture wthrowe  ·  3Comments
dtolnay picture dtolnay  ·  3Comments
overvenus picture overvenus  ·  3Comments
modsec picture modsec  ·  3Comments
pedrohjordao picture pedrohjordao  ·  3Comments