Linux: remove cc-ldoption

[nickdesaulniers]

Forked from: https://github.com/ClangBuiltLinux/continuous-integration/issues/122

Some parts of the kernel invoke the linker via $(CC) using -Wl,... flags. This is problematic because setting LD=ld.lld is not enough for this to work correctly.

If compiling with Clang AND linking with LLD, -fuse-ld=lld needs to be added to KBUILD_CFLAGS.

The pixel kernels have a ld-name macro to distinguish between bfd and gold. I've extended them to support lld, but seems the base patch is not upstreamed.

cc @samitolvanen

[nathanchance]

Should be a fairly innocuous change to upstream as it isn't tied to upstreaming LTO support.

At the same time though, do we actually need an ld-name variable right now? Wouldn't this change work?

diff --git a/Makefile b/Makefile
index 141653226f3c..5e9781490ea6 100644
--- a/Makefile
+++ b/Makefile
@@ -500,6 +500,9 @@ ifneq ($(GCC_TOOLCHAIN),)
 CLANG_FLAGS    += --gcc-toolchain=$(GCC_TOOLCHAIN)
 endif
 CLANG_FLAGS    += -no-integrated-as
+ifneq ($(shell $(LD) --version 2>&1 | head -n 1 | grep LLD),)
+CLANG_FLAGS    += -fuse-ld=lld
+endif
 KBUILD_CFLAGS  += $(CLANG_FLAGS)
 KBUILD_AFLAGS  += $(CLANG_FLAGS)
 export CLANG_FLAGS

If we do want to introduce something like ld-name, won't we want to mirror the CC_IS_ and _VERSION kconfig variables, since cc-name was been killed off in favor of those?

[nathanchance]

Just FYI, the following diff builds and boots on arm64 in QEMU (I built with V=1 to make sure everything shows up as it should), which deals with #340, #341, this issue, and #343.

diff --git a/Makefile b/Makefile
index 18a02e4c611e..aa1d73084fe6 100644
--- a/Makefile
+++ b/Makefile
@@ -492,7 +492,7 @@ endif
 ifneq ($(shell $(CC) --version 2>&1 | head -n 1 | grep clang),)
 ifneq ($(CROSS_COMPILE),)
 CLANG_FLAGS    := --target=$(notdir $(CROSS_COMPILE:%-=%))
-GCC_TOOLCHAIN_DIR := $(dir $(shell which $(LD)))
+GCC_TOOLCHAIN_DIR := $(dir $(shell which $(AS)))
 CLANG_FLAGS    += --prefix=$(GCC_TOOLCHAIN_DIR)
 GCC_TOOLCHAIN  := $(realpath $(GCC_TOOLCHAIN_DIR)/..)
 endif
@@ -500,6 +500,10 @@ ifneq ($(GCC_TOOLCHAIN),)
 CLANG_FLAGS    += --gcc-toolchain=$(GCC_TOOLCHAIN)
 endif
 CLANG_FLAGS    += -no-integrated-as
+ifneq ($(shell $(LD) --version 2>&1 | head -n 1 | grep LLD),)
+CLANG_FLAGS    += -fuse-ld=lld
+KBUILD_LDFLAGS += -O2
+endif
 KBUILD_CFLAGS  += $(CLANG_FLAGS)
 KBUILD_AFLAGS  += $(CLANG_FLAGS)
 export CLANG_FLAGS
diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
index b215c712d897..41a611256476 100644
--- a/arch/arm64/kernel/vdso/Makefile
+++ b/arch/arm64/kernel/vdso/Makefile
@@ -55,7 +55,7 @@ $(obj-vdso): %.o: %.S FORCE

 # Actual build commands
 quiet_cmd_vdsold = VDSOL   $@
-      cmd_vdsold = $(CC) $(c_flags) -Wl,-n -Wl,-T $^ -o $@
+      cmd_vdsold = $(CC) $(c_flags) -Wl,-N -Wl,-T $^ -o $@
 quiet_cmd_vdsoas = VDSOA   $@
       cmd_vdsoas = $(CC) $(a_flags) -c -o $@ $<

[nickdesaulniers]

@nathanchance , thanks for testing. If you stripped out the arm64 vdso stuff, I'd sign off on the Makefile change. I think there's more to do for arm64 vdso (both on the lld side, and maybe in that vdso's Makefile. If you could add those links to these separate issues, then give Rui (LLD maintainer) and I suggested by tags, that would be appreciated. Otherwise I will do so myself once I get back from FOSDEM/Brussels.

[nathanchance]

I went ahead and tested arm32 as well, which now fails to build with LD=ld.lld: https://gist.github.com/9fffddefc50fa9d22983b59f4813145c

I think something is goofy with cc-option with -fuse-ld=lld.

Without that diff:

/usr/bin/ccache clang -Wp,-MD,arch/arm/vdso/.vgettimeofday.o.d  -nostdinc -isystem /home/nathan/cbl/usr/lib/clang/9.0.0/include -I/home/nathan/cbl/ln/arch/arm/include -I./arch/arm/include/generated  -I/home/nathan/cbl/ln/include -I./include -I/home/nathan/cbl/ln/arch/arm/include/uapi -I./arch/arm/include/generated/uapi -I/home/nathan/cbl/ln/include/uapi -I./include/generated/uapi -include /home/nathan/cbl/ln/include/linux/kconfig.h -include /home/nathan/cbl/ln/include/linux/compiler_types.h  -I/home/nathan/cbl/ln/arch/arm/vdso -Iarch/arm/vdso -D__KERNEL__ -mlittle-endian -Qunused-arguments -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror-implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 --target=arm-linux-gnueabi --prefix=/home/nathan/cbl/usr/bin/ --gcc-toolchain=/home/nathan/cbl/usr -no-integrated-as -fno-dwarf2-cfi-asm -mabi=aapcs-linux -mfpu=vfp -funwind-tables -marm -Wa,-mno-warn-deprecated -D__LINUX_ARM_ARCH__=7 -march=armv7-a -msoft-float -Uarm -fno-delete-null-pointer-checks -O2 -fstack-protector-strong -Wno-format-invalid-specifier -Wno-gnu -Wno-address-of-packed-member -Wno-tautological-compare -mno-global-merge -Wno-unused-const-variable -fomit-frame-pointer -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -fno-strict-overflow -fno-merge-all-constants -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-unused-value -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-uninitialized -fPIC -fno-common -fno-builtin -fno-stack-protector -DDISABLE_BRANCH_PROFILING -O2    -DKBUILD_BASENAME='"vgettimeofday"' -DKBUILD_MODNAME='"vgettimeofday"' -c -o arch/arm/vdso/vgettimeofday.o /home/nathan/cbl/ln/arch/arm/vdso/vgettimeofday.c

With that diff:

/usr/bin/ccache clang -Wp,-MD,arch/arm/vdso/.vgettimeofday.o.d  -nostdinc -isystem /home/nathan/cbl/usr/lib/clang/9.0.0/include -I/home/nathan/cbl/ln/arch/arm/include -I./arch/arm/include/generated  -I/home/nathan/cbl/ln/include -I./include -I/home/nathan/cbl/ln/arch/arm/include/uapi -I./arch/arm/include/generated/uapi -I/home/nathan/cbl/ln/include/uapi -I./include/generated/uapi -include /home/nathan/cbl/ln/include/linux/kconfig.h -include /home/nathan/cbl/ln/include/linux/compiler_types.h  -I/home/nathan/cbl/ln/arch/arm/vdso -Iarch/arm/vdso -D__KERNEL__ -mlittle-endian -Qunused-arguments -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror-implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 --target=arm-linux-gnueabi --prefix=/home/nathan/cbl/usr/bin/ --gcc-toolchain=/home/nathan/cbl/usr -no-integrated-as -fuse-ld=lld -mabi=aapcs-linux -mfpu=vfp -funwind-tables -Wa,-mno-warn-deprecated -D__LINUX_ARM_ARCH__=7 -march=armv5t -Wa,-march=armv7-a -msoft-float -Uarm -O2 -fstack-protector-strong -Wno-format-invalid-specifier -Wno-gnu -Wno-address-of-packed-member -Wno-tautological-compare -mno-global-merge -Wno-unused-const-variable -fomit-frame-pointer -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -fno-strict-overflow -fno-merge-all-constants -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-unused-value -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-uninitialized -fPIC -fno-common -fno-builtin -fno-stack-protector -DDISABLE_BRANCH_PROFILING -O2    -DKBUILD_BASENAME='"vgettimeofday"' -DKBUILD_MODNAME='"vgettimeofday"' -c -o arch/arm/vdso/vgettimeofday.o /home/nathan/cbl/ln/arch/arm/vdso/vgettimeofday.c

This particular error is coming from the lack of -march=armv7-a (instead, -march=armv5t -Wa,-march=armv7-a is present), which is cc-option'd: https://github.com/ClangBuiltLinux/linux/blob/b23b0ea3708c3dec599966fc856836aca48835b9/arch/arm/Makefile#L67

[nathanchance]

There's also this: https://github.com/ClangBuiltLinux/linux/blob/b23b0ea3708c3dec599966fc856836aca48835b9/arch/arm/vdso/Makefile#L18

Introduced by https://github.com/ClangBuiltLinux/linux/commit/d2b30cd4b7223a96e606dfc8120626f66d81e091

[dileks]

@nathanchance

Ah great. You had similiar ideas and a use-case/test-case. Makes sense to distinguish on $LD_FOR_BUILD aka ld.bfd, ld.gold and ld.lld.

[nickdesaulniers]

https://lkml.org/lkml/2019/2/7/892

Nathan, please file a separate bug for arm(32) vdso.

[nathanchance]

Done: #356 and #357.

Also, handy trick from the LLD documentation for verifying if something was linked with LLD:

$ readelf --string-dump .comment /mnt/build/kernel/vmlinux

String dump of section '.comment':
  [     0]  Linker: LLD 9.0.0
  [    12]  clang version 9.0.0 (https://github.com/llvm/llvm-project 5c086c310615873a4e6a83e0981c9e680d695819)

[nickdesaulniers]

v3: https://lkml.org/lkml/2019/4/2/128

[nathanchance]

@nickdesaulniers is this still relevant after Masahiro's commits?

https://git.kernel.org/next/linux-next/c/fe00e50b2db8c60e4ec90befad1f5bab8ca2c800

https://git.kernel.org/next/linux-next/c/691efbedc60d2a7364a90e38882fc762f06f52c4

[nickdesaulniers]

repurposing for cc-ldoption removal, since there's numerous landed and outstanding patches, and sending them to stable will likely be a fair amount of work. (TODO(@nickdesaulniers ): list all patches).

[nickdesaulniers]

final patch just hit mainline:
055efab3120bae7ab1ed841317774f3c953f6e1b

(I should go back and find all of these).

[nickdesaulniers]

So this series never landed upstream due to us removing cc-ldoption, but I should resend because we need -fuse-ld=lld. See also #479.

[nathanchance]

I think that if we need to use -fuse-ld, we should probably separate the CC and LD steps because -fuse-ld can be fragile.