Rust: Cannot cross-compile aarch64-musl on x86_64 linux

[asonix]

Cross compiling fails with the linker being unable to find __addtf3, __multf3 and __subtf3

code:

asonix@asonix-boxes-16-04:~/hello-world-rust$ cat src/main.rs 
fn main() {
    println!("Hello, world!");
}

I'm compiling the default hello-world application, and it works fine for aarch64-unknown-linux-gnu in addition to working fine for arm-unknown-linux-musleabihf. I installed musl the same way for both aarch64 and for armhf, so I'm not sure what the issue would be.

Furthermore, I can compile a simple hello-world application written in C with the musl standard library just fine.

Note: I have encountered this issue both in Arch Linux with the 4.14.4 kernel and Ubuntu 16.04 with the 4.4.0 kernel.

version info for the output below, although this happens with nightly and stable:

asonix@asonix-boxes-16-04:~/hello-world-rust$ rustc --version --verbose
rustc 1.22.1 (05e2e1c41 2017-11-22)
binary: rustc
commit-hash: 05e2e1c41414e8fc73d0f267ea8dab1a3eeeaa99
commit-date: 2017-11-22
host: x86_64-unknown-linux-gnu
release: 1.22.1
LLVM version: 4.0

output:

asonix@asonix-boxes-16-04:~/hello-world-rust$ cargo build --target=aarch64-unknown-linux-musl --verbose
   Compiling hello-world v0.1.0 (file:///home/asonix/hello-world-rust)
     Running `rustc --crate-name hello_world src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=df195ab9189e537e -C extra-filename=-df195ab9189e537e --out-dir /home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps --target aarch64-unknown-linux-musl -C ar=aarch64-linux-gnu-ar -C linker=aarch64-linux-gnu-musl-gcc -L dependency=/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps -L dependency=/home/asonix/hello-world-rust/target/debug/deps`
error: linking with `aarch64-linux-gnu-musl-gcc` failed: exit code: 1
  |
  = note: "aarch64-linux-gnu-musl-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-nostdlib" "-Wl,--eh-frame-hdr" "-Wl,-(" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/crt1.o" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/crti.o" "-L" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps/hello_world-df195ab9189e537e.hello_world0.rust-cgu.o" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps/hello_world-df195ab9189e537e.hello_world1.rust-cgu.o" "-o" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps/hello_world-df195ab9189e537e" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps/hello_world-df195ab9189e537e.crate.allocator.rust-cgu.o" "-Wl,--gc-sections" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "-L" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps" "-L" "/home/asonix/hello-world-rust/target/debug/deps" "-L" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liballoc_system-751808ba756769d5.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/librand-3d7b10e850a67e89.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libpanic_unwind-8cb97051d8238386.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liballoc-78c21267a2dc15c1.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd_unicode-0e1b544c94586415.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libunwind-25cc9b024a02d330.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-d42e80cee81b06ce.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libcore-0c5e3d6c117f8c44.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-bd7cc5ada1e908e0.rlib" "-static" "-Wl,-Bdynamic" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/crtn.o" "-Wl,-)"
  = note: /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib(std-827ddea93328ae2d.std0.rust-cgu.o)(.debug_info+0x71d3): R_AARCH64_ABS64 used with TLS symbol _ZN3std10sys_common11thread_info11THREAD_INFO7__getit5__KEY17hffb73c8d664d75eeE
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib(std-827ddea93328ae2d.std0.rust-cgu.o)(.debug_info+0x185a7): R_AARCH64_ABS64 used with TLS symbol _ZN3std2io5stdio12LOCAL_STDOUT7__getit5__KEY17hc4c662569625ecfaE
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib(std-827ddea93328ae2d.std0.rust-cgu.o)(.debug_info+0x24159): R_AARCH64_ABS64 used with TLS symbol _ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h933b3f2f97fe548cE
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib(std-827ddea93328ae2d.std0.rust-cgu.o)(.debug_info+0x24191): R_AARCH64_ABS64 used with TLS symbol _ZN3std9panicking18update_panic_count11PANIC_COUNT7__getit5__KEY17h5de4a458c39aaf87E
          /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-d42e80cee81b06ce.rlib(vfprintf.o): In function `fmt_fp':
          vfprintf.c:(.text.fmt_fp+0x178): undefined reference to `__addtf3'
          vfprintf.c:(.text.fmt_fp+0x20c): undefined reference to `__multf3'
          vfprintf.c:(.text.fmt_fp+0x240): undefined reference to `__subtf3'
          vfprintf.c:(.text.fmt_fp+0x250): undefined reference to `__addtf3'
          vfprintf.c:(.text.fmt_fp+0x278): undefined reference to `__addtf3'
          vfprintf.c:(.text.fmt_fp+0x284): undefined reference to `__subtf3'
          vfprintf.c:(.text.fmt_fp+0x33c): undefined reference to `__subtf3'
          vfprintf.c:(.text.fmt_fp+0x344): undefined reference to `__multf3'
          vfprintf.c:(.text.fmt_fp+0x4e8): undefined reference to `__multf3'
          vfprintf.c:(.text.fmt_fp+0x548): undefined reference to `__subtf3'
          vfprintf.c:(.text.fmt_fp+0x550): undefined reference to `__multf3'
          vfprintf.c:(.text.fmt_fp+0x828): undefined reference to `__addtf3'
          /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-d42e80cee81b06ce.rlib(frexpl.o): In function `frexpl':
          frexpl.c:(.text.frexpl+0x38): undefined reference to `__multf3'
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: Could not compile `hello-world`.

Caused by:
  process didn't exit successfully: `rustc --crate-name hello_world src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=df195ab9189e537e -C extra-filename=-df195ab9189e537e --out-dir /home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps --target aarch64-unknown-linux-musl -C ar=aarch64-linux-gnu-ar -C linker=aarch64-linux-gnu-musl-gcc -L dependency=/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps -L dependency=/home/asonix/hello-world-rust/target/debug/deps` (exit code: 101)

[parched]

Yeah this is a known issue which needs to be fixed in the compiler_builtins crate. See https://github.com/rust-lang-nursery/compiler-builtins/issues/201

[asonix]

I guess my followup is why is aarch64-unknown-linux-musl an allowed target in Rust if f128 doesn't exist in the language and linking anything doesn't work.

[parched]

Good question, it's not entirely clear to me if it's actually compiler_builtins responsibility to provide these. For now you should be fine if you also link libgcc.

[ciofeca]

Is there any workaround?

Say, some combination of _#[cfg(link...)]_ and extern fn __addtf3(_a: i128, _b: i128) -> i128 { 1 }

[ghost]

It would be greatly appreciated if anyone could provide any ideas for any kind of temporary workaround for this until it is properly fixed. I don't care if certain functionality (such as 128-bit integers or whatever) doesn't work. Being able to build static musl executables for aarch64 would be very useful to me.

[parched]

Does linking libgcc not work?

[ciofeca]

There are some use-cases for 64-bit statically linking with _musl:_ for example, SailfishX sports a 64-bit Linux kernel and 32-bit only apps/libs (no 64-bit _ld-linux*.so_ installed). A 64-bit statically linked executable would work out of the box.

[Amanieu]

In case anyone is blocked on this, here's a workaround:

cargo rustc --target aarch64-unknown-linux-musl -- -C link-arg=-lgcc

[Amanieu]

This should be reopened due to https://github.com/rust-lang/libc/pull/1045

[killercup]

@bbqsrc just mentioned this as only-working-with-workaround (cargo rustc --target aarch64-unknown-linux-musl --bin $cratename -- -C link-arg=-lgcc).

Reopening as requested by @Amanieu because of reverted fix PR

[ciofeca]

I was anyway able to cross-compile to _aarch64 musl_ target using this _.cargo/config_ (thanks for the workaround!):

[target.aarch64-unknown-linux-musl]
linker = "aarch64-linux-gnu-gcc"
rustflags = [ "-C", "target-feature=+crt-static", "-C", "link-arg=-lgcc" ]

[qm3ster]

@ciofeca's solution worked for me on Windows
I still had to mklink ar.exe aarch64-linux-musl-ar.exe for the moment, not sure how to get gcc to ask for the right one.

[clarfonthey]

So, what exactly is needed to fix this without the -lgcc?

[12101111]

So, what exactly is needed to fix this without the -lgcc?

__addtf3 is quadruple precision add, __multf3 is quadruple precision multiplication and __subtf3is quadruple precision substruction.

the symbol __addtf3 is equal to this C function:

long double __addtf3 (long double a, long double b){
  return a+b;
}

libgcc.a from gcc and libclang_rt.builtins-aarch64.a from llvm compiler-rt provide
the implement of __addtf3, __multf3 and __subtf3.

In rust we use compiler-builtins (writing in rust) to replace llvm compiler-rt (writing in c and llvm asm)

quadruple precision, and long double in C, means 128bit float number, but
rust don't have f128 type, so compiler-builtins don't support those *tf3

A temporary fix is linking to libgcc.a or libclang_rt.builtins-aarch64.a, but permanent fix won't exist until rust have f128