Rust: Cross compilation x86_64-unknown-linux-gnu (Debian 9) -> arm-unknown-linux-gnueabihf (RPi Zero W) produces crashing binary

[Kixunil]

I tried to cross-compile a basic hello world program (just cargo new) on x86_64 Debian 9 (installed using Rustup) with arm GCC linker for RPi Zero W. The compilation finished "successfully", but the produced binary crashes right after start.

fn main() {
    println!("Hello, world!");
}

Command used for compiling: RUSTFLAGS="-v -g -C linker=arm-linux-gnueabihf-gcc" cargo build --target arm-unknown-linux-gnueabihf, also tried with --release, same issue

Same thing happens when linker is set using .cargo/config:

[target.arm-unknown-linux-gnueabihf]
linker = "arm-linux-gnueabihf-gcc"

I expected the code to produce Hello, world! when run or RPi

Instead, I got Segmentation fault with the following backtrace (gdb) when running the produced binary:

#0  0x004516c6 in ?? ()
#1  0x00402512 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Meta

rustc --version --verbose:

rustc 1.46.0-nightly (346aec9b0 2020-07-11)
binary: rustc
commit-hash: 346aec9b02f3c74f3fce97fd6bda24709d220e49
commit-date: 2020-07-11
host: x86_64-unknown-linux-gnu
release: 1.46.0-nightly
LLVM version: 10.0

The same issue is present on 1.43.0 stable.

Since it's Segfault, RUST_BACKTRACE has no effect.

CC @jooray

[Nicholas-Baron]

I have reproduced this bug with rustc 1.44.1. It only occurs with cross compilation (running cargo build on a Pi does not cause the error).
For me, the crash comes from one of the first instructions in _start trying to read an address that the program has no access to. In my gdb sessions, the pc register was sent to that address, but I could not see an instruction that assigned to the register.
I will look at rustc 1.45 and see if the error changes.

[Nicholas-Baron]

This seems to be an issue that affects all compilers from at least 1.43 onward. The binary from rustc 1.45 is also bad.
This may be a case for cargo-bisect-rustc.

(See next comment from me)

[Kixunil]

Yes, cargo build on RPi itself works for me as well.

[Nicholas-Baron]

Reading into cargo-bisect-rustc, I have realized that my current workflow will probably not work well with bisect, as I am using a VM to run Debian and using ssh from the host machine to load the binaries onto a Pi.
To manually do this, I would need to download the older rustc binaries, which I do not know where to do so.
I will not be able to diagnose this issue any further.

@Kixunil I recommend adding the E-needs-bisection tag.

[Kixunil]

Heh, I also run VMs (Qubes), but I might get it to work if I fiddle with it enough. However, my time is not very abundant these days, so I'd appreciate if someone can do it instead.

Is there some bot command to add such a tag? I don't have the rights to do it using GitHub interface and didn't find any documentation on it.

[Nicholas-Baron]

Sorry, I could not add tags either, so I assumed that you had the ability to. Shows my experience.

Also, rustup may be able to help us here. I thought that the binaries were hidden away, but here they are.

[mati865]

@rustbot modify labels: +E-needs-bisection

[Nicholas-Baron]

I have started backtracking to see what version broke. 1.35 still has the bug and 1.30 as well.

Interestingly, the debug version of 1.30 does not give a segmentation fault. Instead, it is an illegal instruction error.

Time to go to 1.25 and 1.20.

[Nicholas-Baron]

1.20.0 has the same bugs in both release and debug (segfault). This seems to be a rather old bug.

[Kixunil]

Maybe it never worked and nobody noticed? Might be worth trying on other Pis as well, unfortunately, I don't have one laying around right now.

BTW, found the docs now: https://github.com/rust-lang/triagebot/wiki/Labeling

[Nicholas-Baron]

Yeah, that hypothesis sounds more likely.
1.10.0 has the same error.

Later models of Pi have different architectures.
To sum up Wikipedia, the Model 1s and 0s have the same processor and architecture: ARM1176JZF-S (ARMv6/arm).
The largest difference between the 0s and 1s is that 0s have a higher clock than the 1s.
The Model 2s transitioned to quad-core Cortex-A7s (ARMv7) in the first revision, with all later models and board revisions use Cortex-A53s (ARMv8/aarch64).

So in terms of Rust toolchains, that would be the armv7 for the early Model 2s and the aarch64 for everything else. I also only have a Pi Zero and thus cannot test any of the other cross compilation toolchains.

Final note: Did rust support the arm-unknown-linux-gnueabihf target before 1.10.0? I tried to set up 1.0.0 and 1.5.0, but both return an error that the toolchain "does not support components". If this means that these versions will not do cross compilation, I will assume that the error has existed from when rust allowed cross compilation for the first time.

[mati865]

You are probably using armv7 linker assuming Debian and Ubuntu packages are the same:

$ arm-linux-gnueabihf-gcc -v
Using built-in specs.
COLLECT_GCC=arm-linux-gnueabihf-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/arm-linux-gnueabihf/9/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-10ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --without-target-system-zlib --enable-libpth-m2 --enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-werror --enable-multilib --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-linux-gnueabihf --program-prefix=arm-linux-gnueabihf- --includedir=/usr/arm-linux-gnueabihf/include
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu1)

Notice --with-arch=armv7-a here.
For Pi Zero I guess you need armv6 linker.

@rustbot modify labels: -E-needs-bisection

[Kixunil]

@mati865 yes, I have armv7-a as well. Do you happen to know if I have to recompile GCC or is there a better way to get it?

[mati865]

@Kixunil I don't know since I don't use Debian/Ubuntu as my daily driver.