Linux: objtool fixdep host build does not inherit HOSTCC/HOSTLD

[adelva1984]

While trying to enable some experimental LTO patches on android 5.10 kernels, which depend on objtool, I noticed that objtool's fixdep submake does not inherit from HOSTCC/HOSTLD correctly. Basically, they are set up in tools/objtool/Makefile but ignored in some circumstances:

make -f /home/adelva/android/kernel/common/tools/build/Makefile.build dir=. obj=fixdep
make -f /home/adelva/android/kernel/common/scripts/Makefile.build obj=scripts/mod
   ld -r -o /home/adelva/android/kernel/out/android12-5.10/common/tools/objtool/fixdep-in.o  /home/adelva/android/kernel/out/android12-5.10/common/tools/objtool/fixdep.o
/bin/sh: 1: ld: not found
make[5]: *** [/home/adelva/android/kernel/common/tools/build/Makefile.build:145: /home/adelva/android/kernel/out/android12-5.10/common/tools/objtool/fixdep-in.o] Error 127
make[4]: *** [Makefile:45: /home/adelva/android/kernel/out/android12-5.10/common/tools/objtool/fixdep-in.o] Error 2
make[3]: *** [/home/adelva/android/kernel/common/tools/build/Makefile.include:5: fixdep] Error 2

It looks like the HOSTCC and HOSTLD variables should be exported, so the submake uses them via tools/build/Makefile.build

[adelva1984]

diff --git a/tools/objtool/Makefile b/tools/objtool/Makefile
index 4ea9a833dde7..fc5cd4bbc0c9 100644
--- a/tools/objtool/Makefile
+++ b/tools/objtool/Makefile
@@ -60,7 +60,7 @@ ifeq ($(SUBCMD_ORC),y)
 endif

 export SUBCMD_CHECK SUBCMD_ORC
-export srctree OUTPUT CFLAGS SRCARCH AWK
+export srctree OUTPUT CFLAGS SRCARCH AWK HOSTLD HOSTCC
 include $(srctree)/tools/build/Makefile.include

 $(OBJTOOL_IN): fixdep FORCE

Fixes it. If nobody objects, I will send this upstream.

[nathanchance]

@adelva1984 This is resolved in mainline with https://git.kernel.org/linus/c8a950d0d3b926a02c7b2e713850d38217cec3d1. Cherry-picking that to linux-5.10.y resolves the issue you are seeing:

$ make -skj"$(nproc)" LLVM=1 defconfig init/main.o |& rg fixdep
...
   ld.lld -r -o /home/nathan/cbl/src/linux-stable/tools/objtool/fixdep-in.o  /home/nathan/cbl/src/linux-stable/tools/objtool/fixdep.o
...

...

[nickdesaulniers]

Sounds like this needs a backport then?

[dileks]

When dealing with bpf/btf and dwarf-v5patchset I saw serveral (hardcoded) assignments to LLVM utils in scripts and tools directory.

Examples (here: llvm-strip and llvm-objcopy):

$ git grep -E 'LLVM_STRIP|LLVM_OBJCOPY' scripts/
scripts/Makefile.modfinal:              LLVM_OBJCOPY=$(OBJCOPY) $(PAHOLE) -J --btf_base vmlinux $@; \
scripts/link-vmlinux.sh:        LLVM_OBJCOPY=${OBJCOPY} ${PAHOLE} -J ${1}

$ git grep -E 'LLVM_STRIP|LLVM_OBJCOPY' tools/
tools/bpf/bpftool/Makefile:LLVM_STRIP ?= llvm-strip
tools/bpf/bpftool/Makefile:             -g -O2 -target bpf -c $< -o $@ && $(LLVM_STRIP) -g $@
tools/bpf/runqslower/Makefile:LLVM_STRIP ?= llvm-strip
tools/bpf/runqslower/Makefile:  $(LLVM_STRIP) -g $@
tools/testing/selftests/bpf/Makefile:LLVM_OBJCOPY       ?= llvm-objcopy

There are no prereq or error checks.

Just my €0,02 when I see this issue here.

[1] gives some more ideas on how to enhance and fix things up.
OK, this is for HOSTXX we should do that for XX (binaries from binutils and/or LLVM utils).
I mean we can put that into tools/scripts/Makefile.include.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c8a950d0d3b926a02c7b2e713850d38217cec3d1

[dileks]

Something like this:

diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
index f897cb5fb12d..6e858ef57ce9 100644
--- a/tools/bpf/bpftool/Makefile
+++ b/tools/bpf/bpftool/Makefile
@@ -76,7 +76,6 @@ endif
 INSTALL ?= install
 RM ?= rm -f
 CLANG ?= clang
-LLVM_STRIP ?= llvm-strip

 FEATURE_USER = .bpftool
 FEATURE_TESTS = libbfd disassembler-four-args reallocarray zlib libcap \
diff --git a/tools/bpf/runqslower/Makefile b/tools/bpf/runqslower/Makefile
index 4d5ca54fcd4c..5cbd988f5deb 100644
--- a/tools/bpf/runqslower/Makefile
+++ b/tools/bpf/runqslower/Makefile
@@ -5,7 +5,6 @@ OUTPUT ?= $(abspath .output)/

 CLANG ?= clang
 LLC ?= llc
-LLVM_STRIP ?= llvm-strip
 BPFTOOL_OUTPUT := $(OUTPUT)bpftool/
 DEFAULT_BPFTOOL := $(BPFTOOL_OUTPUT)bpftool
 BPFTOOL ?= $(DEFAULT_BPFTOOL)
diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
index 1358e89cdf7d..f17235c56198 100644
--- a/tools/scripts/Makefile.include
+++ b/tools/scripts/Makefile.include
@@ -63,6 +63,8 @@ ifneq ($(LLVM),)
 HOSTAR  ?= llvm-ar
 HOSTCC  ?= clang
 HOSTLD  ?= ld.lld
+LLVM_OBJCOPY   ?= llvm-objcopy
+LLVM_STRIP     ?= llvm-strip
 else
 HOSTAR  ?= ar
 HOSTCC  ?= gcc
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index c51df6b91bef..797500012c2d 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -19,7 +19,6 @@ ifneq ($(wildcard $(GENHDR)),)
 endif

 CLANG          ?= clang
-LLVM_OBJCOPY   ?= llvm-objcopy
 BPF_GCC                ?= $(shell command -v bpf-gcc;)
 SAN_CFLAGS     ?=
 CFLAGS += -g -rdynamic -Wall -O2 $(GENFLAGS) $(SAN_CFLAGS)             \

[dileks]

The same to CLANG LLC and more assignments.

[nathanchance]

If you want to hoist those into tools/scripts/Makefile.include to have less duplication then that is probably fine but putting them under LLVM=1 changes the meaning of the code. I assume llvm-strip and llvm-objcopy were chosen because BPF was supported in LLVM first. They are still overridable to the GNU alternatives through make if you care to do that but I don't think that implicitly changing those is something that should be done without consulting with the BPF folks first. Incidentally, it seems like the LLVM_OBJCOPY assignment in tools/testing/selftests/bpf/Makefile can just be outright removed, it is not used or exported in that Makefile.

The LLVM_OBJCOPY=$(OBJCOPY) in scripts/ are not hardcoded assignments, they pass along OBJCOPY's value to pahole so that the user's OBJCOPY choice is actually respected all the way through the build system.

[dileks]

You want me drop it from ifneq ($(LLVM),)block?

Can I see your approach?

[dileks]

With the attached diff:

echo $PATH
/opt/llvm-toolchain/bin:/opt/proxychains-ng/bin:/home/dileks/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

echo $MAKE $MAKE_OPTS
make V=1 LLVM=1 LLVM_IAS=1 PAHOLE=/opt/pahole/bin/pahole

$MAKE $MAKE_OPTS -C tools/bpf/bpftool/ clean

$MAKE $MAKE_OPTS -C tools/bpf/bpftool/ 2>&1 | tee /tmp/build-log_bpftool.txt

See build-log.

tools_diff.txt
build-log_bpftool.txt

[dileks]

Update diff v2: Add missing includes scripts/Makefile.include.

tools-v2_diff.txt