Linux: CROSS_COMPILE_COMPAT is clang, the compat vDSO will not be built

[nickdesaulniers]

arch/arm64/Makefile:56: CROSS_COMPILE_COMPAT is clang, the compat vDSO will not be built
https://travis-ci.com/ClangBuiltLinux/continuous-integration/jobs/214492915

I'm seeing the above warnings locally when building arm64 mainline today.

Looks related to
bfe801ebe84f42b4666d3f0adde90f504d56e35b
https://lore.kernel.org/linux-arm-kernel/[email protected]/

[nathanchance]

I went ahead and hacked up a quick enabling of the compat vDSO based on what is shipping in the Pixel kernels (so the following issues may be due to this alone):

From 8103cdd7672958d1d0cc790851e78e98a7f375a3 Mon Sep 17 00:00:00 2001
From: Nathan Chancellor <[email protected]>
Date: Tue, 9 Jul 2019 18:30:38 -0700
Subject: [PATCH] vdso32 clang

Signed-off-by: Nathan Chancellor <[email protected]>
---
 arch/arm64/Makefile               |  8 +++-----
 arch/arm64/configs/defconfig      |  1 +
 arch/arm64/kernel/vdso32/Makefile | 11 +++++++++++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index e3d3fd0a4268..a790515c8ccc 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -52,12 +52,10 @@ endif
 ifeq ($(CONFIG_GENERIC_COMPAT_VDSO), y)
   CROSS_COMPILE_COMPAT ?= $(CONFIG_CROSS_COMPILE_COMPAT_VDSO:"%"=%)

-  ifeq ($(CONFIG_CC_IS_CLANG), y)
-    $(warning CROSS_COMPILE_COMPAT is clang, the compat vDSO will not be built)
-  else ifeq ($(CROSS_COMPILE_COMPAT),)
+  ifeq ($(CROSS_COMPILE_COMPAT),)
     $(warning CROSS_COMPILE_COMPAT not defined or empty, the compat vDSO will not be built)
-  else ifeq ($(shell which $(CROSS_COMPILE_COMPAT)gcc 2> /dev/null),)
-    $(error $(CROSS_COMPILE_COMPAT)gcc not found, check CROSS_COMPILE_COMPAT)
+  else ifeq ($(shell which $(CROSS_COMPILE_COMPAT)as 2> /dev/null),)
+    $(error $(CROSS_COMPILE_COMPAT)as not found, check CROSS_COMPILE_COMPAT)
   else
     export CROSS_COMPILE_COMPAT
     export CONFIG_COMPAT_VDSO := y
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index dd827e64e5fe..a67990d47199 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -794,3 +794,4 @@ CONFIG_DEBUG_KERNEL=y
 # CONFIG_DEBUG_PREEMPT is not set
 # CONFIG_FTRACE is not set
 CONFIG_MEMTEST=y
+CONFIG_CROSS_COMPILE_COMPAT_VDSO="arm-linux-gnueabi-"
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
index 288c14d30b45..5af6940899af 100644
--- a/arch/arm64/kernel/vdso32/Makefile
+++ b/arch/arm64/kernel/vdso32/Makefile
@@ -8,7 +8,18 @@
 ARCH_REL_TYPE_ABS := R_ARM_JUMP_SLOT|R_ARM_GLOB_DAT|R_ARM_ABS32
 include $(srctree)/lib/vdso/Makefile

+ifeq ($(CONFIG_CC_IS_CLANG),y)
+COMPATCC := $(CC) --target=$(notdir $(CROSS_COMPILE_COMPAT:%-=%))
+COMPAT_GCC_TOOLCHAIN_DIR := $(dir $(shell which $(CROSS_COMPILE_COMPAT)elfedit))
+COMPATCC += --prefix=$(COMPAT_GCC_TOOLCHAIN_DIR)
+COMPAT_GCC_TOOLCHAIN := $(realpath $(COMPAT_GCC_TOOLCHAIN_DIR)/..)
+ifneq ($(COMPAT_GCC_TOOLCHAIN),)
+COMPATCC += --gcc-toolchain=$(COMPAT_GCC_TOOLCHAIN)
+endif
+COMPATCC += -no-integrated-as
+else
 COMPATCC := $(CROSS_COMPILE_COMPAT)gcc
+endif

 # Same as cc-*option, but using COMPATCC instead of CC
 cc32-option = $(call try-run,\
-- 
2.22.0

However, it errors out like so (several repeated ones):

In file included from arch/arm64/kernel/vdso32/note.c:11:
In file included from include/linux/elfnote.h:62:
In file included from include/linux/elf.h:5:
In file included from arch/arm64/include/asm/elf.h:118:
In file included from arch/arm64/include/asm/processor.h:36:
arch/arm64/include/asm/pointer_auth.h:56:3: error: invalid input constraint 'rZ' in asm
                __ptrauth_key_install(APIA, keys->apia);
                ^
arch/arm64/include/asm/pointer_auth.h:49:2: note: expanded from macro '__ptrauth_key_install'
        write_sysreg_s(__pki_v.lo, SYS_ ## k ## KEYLO_EL1);     \
        ^
arch/arm64/include/asm/sysreg.h:822:37: note: expanded from macro 'write_sysreg_s'
        asm volatile(__msr_s(r, "%x0") : : "rZ" (__val));               \
                                           ^

I assume this is something related to #3: https://godbolt.org/z/8ArZoK

[nickdesaulniers]

error: invalid input constraint 'rZ' in asm

Yeah, the rZ constraints don't exist for arm32, so clang is "helpfully" telling you they don't exist. What does note.c use from elfnote.h? Maybe it's possible to break that dependency (maybe with some forward declarations of whatever note.c is using from elfnote.h in note.c?

Rather than reset a bunch of the same flags from the top level Makefile, I wonder what clang does if you set COMPATCC equal to a copy of KBUILD_CFLAGS plus -target arm-linux-gnueabi? As in:

$ clang -target aarch64-gnu-linux -targert arm-linux-gnueabi ...

would clang accept the latter of the two targets, or error? (seems like it works, but not sure what happens when wired into the above).

[nickdesaulniers]

error: invalid input constraint 'rZ' in asm

Yeah, the rZ constraints don't exist for arm32, so clang is "helpfully" telling you they don't exist.

Wait what!? Your godbolt link above proves otherwise, but they don't show up in the manual (haunted compiler 👻 👽 👾 )

oh wait little r is generic: https://gcc.gnu.org/onlinedocs/gcc/Simple-Constraints.html#Simple-Constraints. Capital Z though...

[nathanchance]

What does note.c use from elfnote.h? Maybe it's possible to break that dependency (maybe with some forward declarations of whatever note.c is using from elfnote.h in note.c?

note.c is

// SPDX-License-Identifier: GPL-2.0
/*
 * Copyright (C) 2012-2018 ARM Limited
 *
 * This supplies .note.* sections to go into the PT_NOTE inside the vDSO text.
 * Here we can supply some information useful to userland.
 */

#include <linux/uts.h>
#include <linux/version.h>
#include <linux/elfnote.h>
#include <linux/build-salt.h>

ELFNOTE32("Linux", 0, LINUX_VERSION_CODE);
BUILD_SALT;

ELFNOTE32 comes from include/linux/elfnote.h, which I don't really think we can break away from.

Rather than reset a bunch of the same flags from the top level Makefile, I wonder what clang does if you set COMPATCC equal to a copy of KBUILD_CFLAGS plus -target arm-linux-gnueabi?

It would probably work but the comment within the Makefile is pretty clear that the duplication is intentional (even if it hurts to look at/work with):

# We cannot use the global flags to compile the vDSO files, the main reason
# being that the 32-bit compiler may be older than the main (64-bit) compiler
# and therefore may not understand flags set using $(cc-option ...). Besides,
# arch-specific options should be taken from the arm Makefile instead of the
# arm64 one.
# As a result we set our own flags here.

Wait what!? Your godbolt link above proves otherwise, but they don't show up in the manual

Could it be that GCC is honoring the Z constraint from arm64? Time to go diving into GCC's source code (shudders)

[nickdesaulniers]

but the comment within the Makefile is pretty clear that the duplication is intentional

Then this Makefile should duplicate the efforts of the top level Makefile.

Probably what we can do is guard something with a preprocesser guard for "is this 32b or 64b" with a commit message like "this code is included in 32b compat vdso so only compile it if building in 64b mode".

+#ifdef __aarch64__
// arm64 only code
+endif

[nathanchance]

Yeah that's a good idea. It appears that will not work for the rZ errors above otherwise there will be -Wimplicit-function-declaration errors with regards to write_sysreg.

[nickdesaulniers]

Wow, so there's a subtle difference in how GCC and Clang validate input constraints:

void foo(void) {
    asm("@ %0":: "rZ"(0)); // gcc ok, clang error
    asm("@ %0":: "Z"(0));  // gcc error, clang error
}

"Z" is not valid in arm-linux-gnueabi. Clang must be validating all_of inline asm constraints, while GCC is validating any_of. I think this is either being relied upon, or another case of #3 .

[nickdesaulniers]

I think also another issue is that it seems from make V=1 ... that LINUXINCLUDE in arch/arm64/kernel/vdso32/Makefile is including arm64 headers in the arm part of the vdso32 build, which is why we're seeing the "rZ" issue. I wonder if the same happens for a GCC build (if so, that seems bad).

[nathanchance]

It does look like that is the case:

arm-linux-gnueabi-gcc -Wp,-MD,arch/arm64/kernel/vdso32/.vgettimeofday.o.d -nostdinc -isystem /home/nathan/toolchains/gcc/9.1.0/lib/gcc/arm-linux-gnueabi/9.1.0/include -I../arch/arm64/include -I./arch/arm64/include/generated -I../include -I./include -I../arch/arm64/include/uapi -I./arch/arm64/include/generated/uapi -I../include/uapi -I./include/generated/uapi -include ../include/linux/kconfig.h -D__KERNEL__ -mlittle-endian -DKASAN_SHADOW_SCALE_SHIFT=3   -fno-PIE -g -fno-dwarf2-cfi-asm -mabi=aapcs-linux -mfloat-abi=soft -mlittle-endian -fPIC -fno-builtin -fno-stack-protector -DDISABLE_BRANCH_PROFILING -march=armv8-a -D__LINUX_ARM_ARCH__=8 -DENABLE_COMPAT_VDSO=1 -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -O2 -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -D__uint128_t='void*' -Wno-shift-count-overflow -Wno-int-to-pointer-cast -include /home/nathan/cbl/linux/lib/vdso/gettimeofday.c -c -o arch/arm64/kernel/vdso32/vgettimeofday.o ../arch/arm64/kernel/vdso32/vgettimeofday.c

[nathanchance]

Will Deacon reported some warnings from including arm64 headers in the compat vDSO; if that gets resolved, the rest of this bug should probably be pretty easy to fix:

https://lore.kernel.org/lkml/20190920142738.qlsjwguc6bpnez63@willie-the-truck/

[nickdesaulniers]

cc @fvincenzo

[fvincenzo]

@nickdesaulniers @nathanchance Thanks for adding me to the ticket.
I answered on the list and will start following this ticket as well. I did not notice the issue in the project otherwise we could have synchronized a bit better :smiley: Can I assign the issue to myself?

[fvincenzo]

@nickdesaulniers @nathanchance I posted today the patch series that covers this issue: https://lore.kernel.org/lkml/[email protected]/

[fvincenzo]

@nickdesaulniers @nathanchance I posted the second version of the patch series: https://lore.kernel.org/lkml/[email protected]/

Please have a look :smiley:

[fvincenzo]

@nickdesaulniers @nathanchance I posted the third version of the patch series:
https://lore.kernel.org/lkml/[email protected]/

If you want to retest it is ready :+1:

[nickdesaulniers]

Closing out this issue. Thanks for all of the work that went into the compat vdso @fvincenzo . I've seen the series backported to CrOS and Android devices.