Linux: Building with function tracer fails

[agners]

Building ARM32 with function tracing enabled currently fails at linking stage:

  LD      vmlinux.o                                                     

  MODPOST vmlinux.o
kernel/softirq.o: In function `_local_bh_enable':
/home/ags/projects/toradex/imx/linux-vanilla-llvm/kernel/softirq.c:152:
undefined reference to `mcount'
kernel/softirq.o: In function `__local_bh_enable_ip':
/home/ags/projects/toradex/imx/linux-vanilla-llvm/kernel/softirq.c:159:
undefined reference to `mcount'
kernel/softirq.o: In function `do_softirq':
/home/ags/projects/toradex/imx/linux-vanilla-llvm/kernel/softirq.c:316:
undefined reference to `mcount'
kernel/softirq.o: In function `irq_enter':
/home/ags/projects/toradex/imx/linux-vanilla-llvm/kernel/softirq.c:337:
undefined reference to `mcount'
kernel/softirq.o: In function `irq_exit':
/home/ags/projects/toradex/imx/linux-vanilla-llvm/kernel/softirq.c:395:
undefined reference to `mcount'

[agners]

Adding -meabi gnu allows to compile the kernel, however, it will crash silently on boot, see also:

https://www.spinics.net/lists/arm-kernel/msg671262.html

[m-gupta]

Use of "-pg" with "-meabi gnu" on ARM32 is currently known to be broken in clang :-(
https://bugs.llvm.org/show_bug.cgi?id=33845

[agners]

@m-gupta probably depends what the exact definition is of GNU mcount... I guess GCC is the ABI, and Clang/LLVM should follow. And yes, GCC pushes the lr to the stack, see also:

https://www.spinics.net/lists/arm-kernel/msg672631.html

Clang creates the mcount calls in a Function pass in lib/Transforms/Utils/EntryExitInstrumenter.cpp. Afaik, this is at LLVM IR level, it is probably not easy to just insert a push {lr} there.

[shenki]

ppc64 be fails with the same error message:

  AR      built-in.a
  LD      vmlinux.o
  MODPOST vmlinux.o
powerpc64-linux-gnu-ld: init/do_mounts_initrd.o:(__mcount_loc+0x0): undefined reference to `.init_linuxrc'

https://travis-ci.org/shenki/continuous-integration/jobs/474371134#L2813

It can be worked around by disabling ftrace.

[nathanchance]

FWIW, this, #325 (when using ld.bfd), and #17 (which has patches posted upstream) are the only three issues that prevent successfully building arm allyesconfig on linux-next (and I assume allmodconfig but I haven't tried). I have been working around it with the following patch, I wonder if it is worth trying to upstream... (given that it doesn't seem like this is going to be fixed quickly):

https://github.com/nathanchance/patches/blob/db2abfc9f51d/linux/build-hax/0003-DO-NOT-UPSTREAM-ARM-Don-t-select-HAVE_FUNCTION_TRACE.patch

We currently disable CONFIG_FTRACE for CI, the above change would give us a little extra build coverage.

[nickdesaulniers]

CONFIG_FTRACE selects a broken unwinder on arm, @vo4 is looking into this.

[agners]

@nickdesaulniers not sure what you mean, on ARM framepointer unwinder doesn't work with Clang, but the ARM EAPI unwinder should work fine. See also: https://patchwork.kernel.org/patch/10614845/

As state above, this is a LLVM issue. LLVM issue https://bugs.llvm.org/show_bug.cgi?id=33845 needs to be resolved, then CONFIG_FTRACE should magically work. Maybe @smithp35 can have a look at it?

We could work around the LLVM issue since Linux uses a custom __gnu_mcount_nc implementation, I actually managed to hack up a version which works when compiling with LLVM/Clang. But not sure if we want such a hack in the kernel.

Another issue is that CONFIG_FUNCTION_GRAPH_TRACER only works with the framepointer unwinder (which does not work on Clang). I think technically a CONFIG_FUNCTION_GRAPH_TRACER which works with the ARM EABI unwinder should be doable, its just that nobody tackled it (yet). Russel would welcome such a change too: https://www.spinics.net/lists/arm-kernel/msg699665.html

[agners]

FWIW, this is the patch mentioned above: 0001-HACK-ARM-hack-ftrace-to-work-with-LLVM.patch.gz

[smithp35]

I can add that pr to the list of things to look at although I've not got a lot of spare time right now so it may be some time before I can work on it.

[vo4]

We could also resolve this problem by supporting CONFIG_FRAME_POINTER for Clang layout of the frame. In most cases it would only require adding clang-specific offsets from frame pointer to find stuff on the stack. The problem is that backtraces might not be as meaningful, because Clang implementation of AAPCS doesn't push pc register onto the stack.

[vo4]

Building CONFIG_FRAME_POINTER for Clang frame layout will end up being essentially a revert of https://lkml.org/lkml/2018/8/26/197, since we'd need a custom mcount function. So getting clang to support "-pg -meabi gnu" is probably the more "correct" approach here.

[nickdesaulniers]

@vo4 , @nathanchance pointed out https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=503621628b32782a07b2318e4112bd4372aa3401 in the latest meeting notes. How does this affect arm32?

[nathanchance]

Unfortunately, I was a little preemptive in suggesting that this issue would be avoided with that patch. I still need to disable CONFIG_HAVE_FUNCTION_TRACER to avoid these linking errors :/

[agners]

Yeah there is another if !CC_IS_CLANG missing, I mentioned it, but it seems just slightly too late, a couple of hours after the patch got merged:
https://patchwork.kernel.org/patch/10899797/

[nickdesaulniers]

CrOS bug: https://bugs.chromium.org/p/chromium/issues/detail?id=976088

[Nathan-Huckleberry]

https://reviews.llvm.org/D65019 and https://lkml.org/lkml/2019/8/1/1090

[nathanchance]

Just for the record, @shenki's PowerPC 64-bit big endian issue is resolved as of v5.3-rc1 with the following commit:

f079bb3c5f2978b2c1a13098ab2a8c32e5d1ee3d is the first fixed commit
commit f079bb3c5f2978b2c1a13098ab2a8c32e5d1ee3d
Author: Christophe Leroy <[email protected]>
Date:   Tue May 7 13:31:38 2019 +0000

    powerpc/ftrace: Enable C Version of recordmcount

    Selects HAVE_C_RECORDMCOUNT to use the C version of the recordmcount
    intead of the old Perl Version of recordmcount.

    This should improve build time. It also seems like the old Perl Version
    misses some calls to _mcount that the C version finds.

    Signed-off-by: Christophe Leroy <[email protected]>
    Signed-off-by: Michael Ellerman <[email protected]>

 arch/powerpc/Kconfig | 1 +
 1 file changed, 1 insertion(+)

[nickdesaulniers]

https://reviews.llvm.org/rL369147 fixes __gnu_mcount_nc calling convention related errors. (CONFIG_FTRACE)

[nathanchance]

I didn't realize this at first but we need something like this

diff --git a/arch/arm/Makefile b/arch/arm/Makefile
index c3624ca6c0bc..027678b0d4a6 100644
--- a/arch/arm/Makefile
+++ b/arch/arm/Makefile
@@ -112,6 +112,10 @@ ifeq ($(CONFIG_ARM_UNWIND),y)
 CFLAGS_ABI     +=-funwind-tables
 endif

+ifeq ($(CONFIG_CC_IS_CLANG),y)
+CFLAGS_ABI     +=-meabi gnu
+endif
+
 # Accept old syntax despite ".syntax unified"
 AFLAGS_NOWARN  :=$(call as-option,-Wa$(comma)-mno-warn-deprecated,-Wa$(comma)-W)

for there to be no more build errors on arm32 all{yes,mod}config.

[nickdesaulniers]

@Nathan-Huckleberry 's unwinder patch got accepted withing 24hrs of submission, nice! :rocket:
https://www.armlinux.org.uk/developer/patches/viewpatch.php?id=8900/1

[nathanchance]

Patch sent to finally fix the build error: https://lore.kernel.org/lkml/[email protected]/

[nathanchance]

Submitted to RMK's patch system: https://www.armlinux.org.uk/developer/patches/viewpatch.php?id=8905/1

[nathanchance]

Final patch accepted for 5.4!

http://git.armlinux.org.uk/cgit/linux-arm.git/commit/?id=7b3948597372e5a6b314208ac320362c204b7f0f

[nathanchance]

Merged into mainline: https://git.kernel.org/torvalds/c/b0fe66cf095016e0b238374c10ae366e1f087d11

[nickdesaulniers]

backports sent for UNWINDER_FRAME_POINTER to 5.3, 4.19, and 4.14: https://lore.kernel.org/stable/CAKwvOd=[e-mail]/T/#u