Zig: riscv64-linux-musl fails to build because clang ignores "relax" CPU feature being disabled

[andrewrk]

Build Dependencies...c [4] Link...lld: error: /home/andy/tmp/zig/zig-cache/o/IpDYEfWhYXptyo1wUMBOHGigvD1gIwswwGsn9q8PMx_nZWbodvgUyWK82s8FMuVF/test.o: cannot link object files with different floating-point ABI
lld: error: /home/andy/.cache/zig/stage1/o/VEx_936BTIqzzeoCwN8Y-mf4ID7zKTOTVnjO97Lck-YxbGVgANh3e-2BvnjNTl9I/libcompiler_rt.a(/home/andy/.cache/zig/stage1/o/VEx_936BTIqzzeoCwN8Y-mf4ID7zKTOTVnjO97Lck-YxbGVgANh3e-2BvnjNTl9I/compiler_rt.o): cannot link object files with different floating-point ABI
lld: error: crt1.c:(.text+0x0): relocation R_RISCV_ALIGN requires unimplemented linker relaxation; recompile with -mno-relax
lld: error: /home/andy/.cache/zig/stage1/o/0DQElJcX1SKnFVMT4eVjKHnwIlJxEkzUcZKZCC4IvAzHfLoW1Qhmft4sIBdlpOCG/libc.a(/home/andy/.cache/zig/stage1/o/RQJU2zWpMxCLAgnzDiigjm77gHOJ4PSso5xecaIEan4FujS6rKR4ofNvnez377ZH/__set_thread_area.o):(.text+0x0): relocation R_RISCV_ALIGN requires unimplemented linker relaxation; recompile with -mno-relax
lld: error: /home/andy/.cache/zig/stage1/o/0DQElJcX1SKnFVMT4eVjKHnwIlJxEkzUcZKZCC4IvAzHfLoW1Qhmft4sIBdlpOCG/libc.a(/home/andy/.cache/zig/stage1/o/RQJU2zWpMxCLAgnzDiigjm77gHOJ4PSso5xecaIEan4FujS6rKR4ofNvnez377ZH/__set_thread_area.o):(.text+0x2): relocation R_RISCV_ALIGN requires unimplemented linker relaxation; recompile with -mno-relax
lld: error: /home/andy/.cache/zig/stage1/o/0DQElJcX1SKnFVMT4eVjKHnwIlJxEkzUcZKZCC4IvAzHfLoW1Qhmft4sIBdlpOCG/libc.a(/home/andy/.cache/zig/stage1/o/y23enU50XsJY1XTmX8wN98Qif0RzUX49lgc9GC54QyTuBHwdrAEEQa5Cjk5g2wTY/restore.o):(.text+0x0): relocation R_RISCV_ALIGN requires unimplemented linker relaxation; recompile with -mno-relax
lld: error: /home/andy/.cache/zig/stage1/o/0DQElJcX1SKnFVMT4eVjKHnwIlJxEkzUcZKZCC4IvAzHfLoW1Qhmft4sIBdlpOCG/libc.a(/home/andy/.cache/zig/stage1/o/y23enU50XsJY1XTmX8wN98Qif0RzUX49lgc9GC54QyTuBHwdrAEEQa5Cjk5g2wTY/restore.o):(.text+0x2): relocation R_RISCV_ALIGN requires unimplemented linker relaxation; recompile with -mno-relax

The following command exited with error code 1:
/home/andy/tmp/zig/build-llvm10-release/zig test /home/andy/tmp/zig/test/stage1/behavior.zig --library c --test-name-prefix behavior-riscv64-linux-musl-Debug-c-multi  --cache-dir /home/andy/tmp/zig/zig-cache --name test -target riscv64-linux-musl --test-cmd qemu-riscv64 --test-cmd-bin --override-lib-dir /home/andy/tmp/zig/lib 

Related: #1535

cc @luismarques

[luismarques]

See my other comment about this topic: https://github.com/ziglang/zig/pull/4409#issuecomment-587071679

[andrewrk]

I get the impression you're saying you added "relax" to solve this issue but it might be the other way around. Isn't it the case that you compile musl with code relaxation enabled, thus generating R_RISCV_ALIGNs relocations? You then need a linker which supports relaxations to handle that.

Can't you compile musl without generating those align relocations? I don't remember 100% of the top of my head if disabling code relaxations is sufficient to ensure that, but that was my impression. If you don't have the align relocations you can then use LLD, for a pure LLVM toolchain.

I believe this is a bug in clang which should be reproducible simply by building musl 1.1.24 with clang 10 and linking with lld 10, although I have not confirmed that yet. Here is an example c compiler line for _set_thread_area.o:

/home/andy/tmp/zig/build-llvm10-release/zig cc -MD -MV -MF /home/andy/.cache/zig/stage1/tmp/mLIKW-FaLpvq-__set_thread_area.o.d -nostdinc -fno-spell-checking -isystem /home/andy/tmp/zig/lib/include -target riscv64-unknown-linux-musl -Xclang -target-feature -Xclang +64bit,+a,+c,+d,-e,+f,+m,-relax,-reserve-x1,-reserve-x10,-reserve-x11,-reserve-x12,-reserve-x13,-reserve-x14,-reserve-x15,-reserve-x16,-reserve-x17,-reserve-x18,-reserve-x19,-reserve-x2,-reserve-x20,-reserve-x21,-reserve-x22,-reserve-x23,-reserve-x24,-reserve-x25,-reserve-x26,-reserve-x27,-reserve-x28,-reserve-x29,-reserve-x3,-reserve-x30,-reserve-x31,-reserve-x4,-reserve-x5,-reserve-x6,-reserve-x7,-reserve-x8,-reserve-x9,-rvc-hints -g -fno-omit-frame-pointer -D_DEBUG -fno-stack-protector -o /home/andy/.cache/zig/stage1/tmp/mLIKW-FaLpvq-__set_thread_area.o -c /home/andy/tmp/zig/lib/libc/musl/src/thread/riscv64/__set_thread_area.s -std=c99 -ffreestanding -Wa,--noexecstack -D_XOPEN_SOURCE=700 -I /home/andy/tmp/zig/lib/libc/musl/arch/riscv64 -I /home/andy/tmp/zig/lib/libc/musl/arch/generic -I /home/andy/tmp/zig/lib/libc/musl/src/include -I /home/andy/tmp/zig/lib/libc/musl/src/internal -I /home/andy/tmp/zig/lib/libc/musl/include -I /home/andy/tmp/zig/lib/libc/include/riscv64-linux-musl -I /home/andy/tmp/zig/lib/libc/include/generic-musl -Os -fomit-frame-pointer -fno-unwind-tables -fno-asynchronous-unwind-tables -ffunction-sections -fdata-sections -Qunused-arguments -w

If you copy paste that and replace spaces with newlines it becomes easier to read. But the important part is: -Xclang -target-feature -Xclang -relax which is equivalent to -mno-relax.

So it looks like a bug in clang (or lld?) because we build _set_thread_area.o with -mno-relax and then get the message when linking __set_thread_area.o):(.text+0x0): relocation R_RISCV_ALIGN requires unimplemented linker relaxation; recompile with -mno-relax.

[luismarques]

Isn't that because -Xclang -target-feature -Xclang -relax is ignored when assembling .s files?

For instance, if you try this with a clang invocation:

clang --target=riscv64-unknown-linux-musl -Xclang -target-feature -Xclang -relax -c foo.s 
clang-10: warning: argument unused during compilation: '-Xclang -target-feature' [-Wunused-command-line-argument]
clang-10: warning: argument unused during compilation: '-Xclang -relax' [-Wunused-command-line-argument]

I'm not sure at what level your zig cc is wrapping/creating the toolchain invocation, but perhaps that explains why you don't see simiar warnings, and therefore the problem doesn't become obvious.

[andrewrk]

Confirmed, that's the issue. Thanks!

[MaskRay]

.text has an initial align(4). When the target feature +c is enabled, it adds an R_RISCV_ALIGN of r_addend=2. lld errors (starting from lld 10.0.0) on R_RISCV_ALIGN because it does not support RISC-V linker relaxation. Clang cc1as does not respect -Xclang, only cc1 does. clang -target riscv64 -mno-relax a.s correctly disables +c.

[andrewrk]

Related: #3508

[luismarques]

.text has an initial align(4). When the target feature +c is enabled, it adds an R_RISCV_ALIGN of r_addend=2. lld errors (starting from lld 10.0.0) on R_RISCV_ALIGN because it does not support RISC-V linker relaxation. Clang cc1as does not respect -Xclang, only cc1 does. clang -target riscv64 -mno-relax a.s correctly disables +c.

To be clear, in clang by default only the base RISC-V integer ISA is enabled, unless you are targeting Linux. So the C extension being disabled is caused by neither specifying an march nor targeting *-linux-*. If you see R_RISCV_ALIGN relocations without code relaxations being enabled then that's a bug, so please report it, but my understanding is that in this case the issue was only due to the incorrect passing of flags, so LLVM is working as intended.

[andrewrk]

Hmm now I'm confused again because the file in question is crt1.c, not an assembly file. The clang command line is:

clang-10 -c -MD -MV -MF /home/andy/.cache/zig/stage1/tmp/iPFFmUu58z0t-crt1.o.d -nostdinc -fno-spell-checking -isystem /home/andy/tmp/zig/lib/include -target riscv64-unknown-linux-musl -Xclang -target-feature -Xclang +64bit,+a,+c,+d,-e,+f,+m,-relax,-reserve-x1,-reserve-x10,-reserve-x11,-reserve-x12,-reserve-x13,-reserve-x14,-reserve-x15,-reserve-x16,-reserve-x17,-reserve-x18,-reserve-x19,-reserve-x2,-reserve-x20,-reserve-x21,-reserve-x22,-reserve-x23,-reserve-x24,-reserve-x25,-reserve-x26,-reserve-x27,-reserve-x28,-reserve-x29,-reserve-x3,-reserve-x30,-reserve-x31,-reserve-x4,-reserve-x5,-reserve-x6,-reserve-x7,-reserve-x8,-reserve-x9,-rvc-hints -g -fno-omit-frame-pointer -D_DEBUG -Og -fno-stack-protector -o /home/andy/.cache/zig/stage1/tmp/iPFFmUu58z0t-crt1.o /home/andy/tmp/zig/lib/libc/musl/crt/crt1.c -std=c99 -ffreestanding -Wa,--noexecstack -D_XOPEN_SOURCE=700 -I /home/andy/tmp/zig/lib/libc/musl/arch/riscv64 -I /home/andy/tmp/zig/lib/libc/musl/arch/generic -I /home/andy/tmp/zig/lib/libc/musl/src/include -I /home/andy/tmp/zig/lib/libc/musl/src/internal -I /home/andy/tmp/zig/lib/libc/musl/include -I /home/andy/tmp/zig/lib/libc/include/riscv64-linux-musl -I /home/andy/tmp/zig/lib/libc/include/generic-musl -Os -fomit-frame-pointer -fno-unwind-tables -fno-asynchronous-unwind-tables -ffunction-sections -fdata-sections -fno-stack-protector -DCRT

You can see that -relax is there. Let me see if explicitly adding -mno-relax makes a difference.

[andrewrk]

Yeah, I put logic in so that a redundant -mrelax or -mno-relax parameter is added which simply matches the CPU features, and that made musl link successfully. So I do think this is an upstream issue.

The next (and last?) problem is:

lld: error: ./hello.o: cannot link object files with different floating-point ABI

The linker line that produced this error is:

lld -error-limit=0 -z stack-size=16777216 --gc-sections -m elf64lriscv -static -o ./hello /home/andy/.cache/zig/stage1/o/vVTA0dsyVsv3vh-m9jcmtJzluyuAxT-yFazbD-7coIjcMWStoqvQQZZ83Vp4TzCV/crt1.o ./hello.o /home/andy/.cache/zig/stage1/o/6_-3RpshiSeaWea6L8kn5LuE96yYu8fzHfP1phb0w3u50BvXO3q8bhmzPBxPBo_w/libcompiler_rt.a /home/andy/.cache/zig/stage1/o/e-azki2mEAcjFaG8Z8EVd29T6CJD04sgxcXN88KbcRH7MfBZ3pFZn9PMUxOGQ0V8/libc.a --allow-shlib-undefined

[andrewrk]

The relax thing is solved. New issue for the floating point ABI thing: #4863