Linux: Mainline kernel build fails early on ThunderX2

[ikitayama]

Hi folks,

1. what kernel version (mainline, LTS) - Mainline
2. any config in particular is related - defconfig is selected
3. the precise error message

[r1050@apollo linux]$ make CC=clang
scripts/kconfig/conf  --syncconfig Kconfig
arch/arm64/Makefile:54: CROSS_COMPILE_COMPAT is clang, the compat vDSO will not be built
  UPD     include/config/kernel.release
  UPD     include/generated/utsrelease.h
  CC      scripts/mod/empty.o
  MKELF   scripts/mod/elfconfig.h
  HOSTCC  scripts/mod/modpost.o
  CC      scripts/mod/devicetable-offsets.s
In file included from scripts/mod/devicetable-offsets.c:3:
In file included from ./include/linux/mod_devicetable.h:13:
In file included from ./include/linux/uuid.h:12:
./include/linux/string.h:9:10: fatal error: 'stdarg.h' file not found
#include <stdarg.h>
         ^~~~~~~~~~
1 error generated.
make[2]: *** [scripts/mod/devicetable-offsets.s] Error 1
make[1]: *** [prepare0] Error 2
make: *** [sub-make] Error 2

1. what version of LLD you're using (as we may have already observed the error and fixed it in a newer version). - I am going to use the latest LLD from Trunk of LLVM LLD project repo.

[ikitayama]

I'm reporting here as Clang on a ppc64le machine (POWER8) at JSC is not affected with this header issue at all.

[ihalip]

You can run make V=1 or look at scripts/mod/.devicetable-offsets.s.cmd to see how the compiler is invoked. You can then copy/paste the entire line and add -v to get more diagnostics, such as these:

ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /usr/lib/llvm-8/lib/clang/8.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

On my Ubuntu workstation, that header seems to be provided by the libclang-common-VER-dev package:

$ apt-file search /usr/lib/llvm-8/lib/clang/8.0.0/include/stdarg.h
libclang-common-8-dev: /usr/lib/llvm-8/lib/clang/8.0.0/include/stdarg.h

[ikitayama]

I have Clang installed non-standard location as I build Trunk as a normal
user daily, in that case is there an environmental variable to set prior to
invoking a kernel build job?

On Mon, Sep 16, 2019 at 15:04 Ilie Halip [e-mail] wrote:

You can run make V=1 or look at scripts/mod/.devicetable-offsets.s.cmd to
see how the compiler is invoked. You can then copy/paste the entire line
and add -v to get more diagnostics, such as these:

ignoring nonexistent directory "/include"

include "..." search starts here:

include <...> search starts here:

/usr/local/include
/usr/lib/llvm-8/lib/clang/8.0.0/include
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.

On my Ubuntu workstation, that header seems to be provided by the
libclang-VER-common-dev package:

$ apt-file search /usr/lib/llvm-8/lib/clang/8.0.0/include/stdarg.h
libclang-common-8-dev: /usr/lib/llvm-8/lib/clang/8.0.0/include/stdarg.h

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ClangBuiltLinux/linux/issues/656?email_source=notifications&email_token=AAHLBZZH74PJZFXQJK65SILQJ4OWHA5CNFSM4IWYVAP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6YFUBA#issuecomment-531651076,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHLBZ2LP65GYBTBW3KZDKLQJ4OWHANCNFSM4IWYVAPQ
.

[nickdesaulniers]

I have Clang installed non-standard location

I think 785f11aa595bc3d4e74096cbd598ada54ecc0d81 and ef8c4ed9db802 were made to support this use case.

I am going to use the latest LLD from Trunk of LLVM LLD project repo.
The invocation [r1050@apollo linux]$ make CC=clang alludes to LLD not being used, though Clang can be configured to default to LLD. LLD at this point seems to be a red herring; not related.

@ikitayama what version of clang are you using? Can it build hello world? scripts/mod/devicetable-offsets.s is a host utility, so stdarg.h should be the host's stdarg.h IIUC. stdarg.h is used for declaring va_start and friends for variadic function support.

I recommend first trying to compile the example from man va_start on your system with that version of clang. It seems like there's something going wrong or missing on your host that would prevent the building of such host side utilities.

[ikitayama]

@nickdesaulniers first of all, as working on ThunderX2, I am not doing cross-builds.
Here's the Clang version:

$ clang --version
clang version 10.0.0 (/fefs/home/r1050/projects/llvm-project/clang 10151f661854e3ee4922662f1d0f62b327cbfa8c)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /fefs/home/r1050/opt/clang/current/bin

... and yes I can compile a hello world program without an issue.

[nathanchance]

packet.com has Thunder X systems available for spot usage, I will try to test/investigate this tonight or tomorrow.

EDIT: Thunder X 2x, not Thunder X2. Shouldn't matter, this is most likely a native arm64 issue, not related to a certain processor.

[ikitayama]

packet.com has Thunder X systems available for spot usage, I will try to test/investigate this tonight or tomorrow.

EDIT: Thunder X 2x, not Thunder X2. Shouldn't matter, this is most likely a native arm64 issue, not related to a certain processor.

Correct.

[ikitayama]

Using Arm Clang (part of HPC Compiler Suite) does not solve the issue either.

[nathanchance]

I cannot reproduce this on Packet's Ampere system on Ubuntu 18.04.

My environment setup script (for this experiment, I did not install clang): https://github.com/nathanchance/scripts/blob/cb7b1b2c2bae6940b11d9db1f399802bb213cc3b/env/generic

$ git clone git://github.com/ClangBuiltLinux/tc-build && cd tc-build && ./build-llvm.py --build-stage1-only

========================
== Checking CC and LD ==
========================

CC: /usr/bin/aarch64-linux-gnu-gcc-7
CXX: /usr/bin/g++
LD: /usr/bin/ld.gold

===========================
== Checking dependencies ==
===========================

/usr/bin/cmake
/usr/bin/curl
/home/nathan/usr/bin/git
/usr/bin/ninja
...

$ cd .. && git clone git://github.com/torvalds/linux && cd linux

$ PATH=/home/nathan/tc-build/build/llvm/stage1/bin:${PATH} make -j$(nproc) CC=clang KCFLAGS=-Wno-implicit-fallthrough LD=ld.lld defconfig vmlinux
...
  AR      drivers/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  LD      vmlinux.o
  MODPOST vmlinux.o
  MODINFO modules.builtin.modinfo
  KSYM    .tmp_kallsyms1.o
  KSYM    .tmp_kallsyms2.o
  LD      vmlinux
  SORTEX  vmlinux
  SYSMAP  System.map

V=1 shows that lib/clang/10.0.0/include is added to the system include search path via -isystem which is where stdarg.h is available:

clang -Wp,-MD,scripts/mod/.devicetable-offsets.s.d -nostdinc -isystem /home/nathan/tc-build/build/llvm/stage1/lib/clang/10.0.0/include -I../arch/arm64/include -I./arch/arm64/include/generated -I../include -I./include -I../arch/arm64/include/uapi -I./arch/arm64/include/generated/uapi -I../include/uapi -I./include/generated/uapi -include ../include/linux/kconfig.h -include ../include/linux/compiler_types.h -D__KERNEL__ -mlittle-endian -DKASAN_SHADOW_SCALE_SHIFT=3 -Qunused-arguments -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -no-integrated-as -Werror=unknown-warning-option -mgeneral-regs-only -DCONFIG_AS_LSE=1 -DCONFIG_CC_HAS_K_CONSTRAINT=1 -fno-asynchronous-unwind-tables -DKASAN_SHADOW_SCALE_SHIFT=3 -fno-delete-null-pointer-checks -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-format-invalid-specifier -Wno-gnu -Wno-tautological-compare -mno-global-merge -Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -Wdeclaration-after-statement -Wimplicit-fallthrough -Wvla -Wno-pointer-sign -fno-strict-overflow -fno-merge-all-constants -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-implicit-fallthrough -I ../scripts/mod -I ./scripts/mod -DKBUILD_BASENAME='"devicetable_offsets"' -DKBUILD_MODNAME='"devicetable_offsets"'  -fverbose-asm -S -o scripts/mod/devicetable-offsets.s ../scripts/mod/devicetable-offsets.c

[ikitayama]

Nathan,
Can you do the same if possible, on CentOS?

On Thu, Sep 19, 2019 at 3:54 Nathan Chancellor [e-mail]
wrote:

I cannot reproduce this on Packet's Ampere system
https://www.packet.com/cloud/servers/c2-large-arm/ on Ubuntu 18.04.

My environment setup script (for this experiment, I did not install clang):

https://github.com/nathanchance/scripts/blob/cb7b1b2c2bae6940b11d9db1f399802bb213cc3b/env/generic

$ git clone git://github.com/ClangBuiltLinux/tc-build && cd tc-build && ./build-llvm.py --build-stage1-only

========================

== Checking CC and LD ==

CC: /usr/bin/aarch64-linux-gnu-gcc-7
CXX: /usr/bin/g++
LD: /usr/bin/ld.gold

===========================

== Checking dependencies ==

/usr/bin/cmake
/usr/bin/curl
/home/nathan/usr/bin/git
/usr/bin/ninja
...

$ cd .. && git clone git://github.com/torvalds/linux && cd linux

$ PATH=/home/nathan/tc-build/build/llvm/stage1/bin:${PATH} make -j$(nproc) CC=clang KCFLAGS=-Wno-implicit-fallthrough LD=ld.lld defconfig vmlinux
...
AR drivers/built-in.a
GEN .version
CHK include/generated/compile.h
LD vmlinux.o
MODPOST vmlinux.o
MODINFO modules.builtin.modinfo
KSYM .tmp_kallsyms1.o
KSYM .tmp_kallsyms2.o
LD vmlinux
SORTEX vmlinux
SYSMAP System.map

V=1 shows that lib/clang/10.0.0/include is added to the system include
search path via -isystem which is where stdarg.h is available:

clang -Wp,-MD,scripts/mod/.devicetable-offsets.s.d -nostdinc -isystem /home/nathan/tc-build/build/llvm/stage1/lib/clang/10.0.0/include -I../arch/arm64/include -I./arch/arm64/include/generated -I../include -I./include -I../arch/arm64/include/uapi -I./arch/arm64/include/generated/uapi -I../include/uapi -I./include/generated/uapi -include ../include/linux/kconfig.h -include ../include/linux/compiler_types.h -D__KERNEL__ -mlittle-endian -DKASAN_SHADOW_SCALE_SHIFT=3 -Qunused-arguments -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -no-integrated-as -Werror=unknown-warning-option -mgeneral-regs-only -DCONFIG_AS_LSE=1 -DCONFIG_CC_HAS_K_CONSTRAINT=1 -fno-asynchronous-unwind-tables -DKASAN_SHADOW_SCALE_SHIFT=3 -fno-delete-null-pointer-checks -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-format-invalid-specifier -Wno-gnu -Wno-tautological-compare -mno-global-merge -Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -Wdeclaration-after-statement -Wimplicit-fallthrough -Wvla -Wno-pointer-sign -fno-strict-overflow -fno-merge-all-constants -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-implicit-fallthrough -I ../scripts/mod -I ./scripts/mod -DKBUILD_BASENAME='"devicetable_offsets"' -DKBUILD_MODNAME='"devicetable_offsets"' -fverbose-asm -S -o scripts/mod/devicetable-offsets.s ../scripts/mod/devicetable-offsets.c

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ClangBuiltLinux/linux/issues/656?email_source=notifications&email_token=AAHLBZ4A6FVPIPZ7T2PZ5BLQKJ2NHA5CNFSM4IWYVAP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7BCZDA#issuecomment-532819084,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHLBZ6IULBFLBKE3CS4CR3QKJ2NHANCNFSM4IWYVAPQ
.

[nathanchance]

Sure, I will try to get to it tonight.

[nathanchance]

@ikitayama everything is fine on CentOS as well, although I had to do a lot more environment hacking because of how outdated CentOS is... I'd recommend giving the tc-build script a shot to see if it is something up with the way that you have configured LLVM. Alternatively, try make CC="clang -v" V=1 like @ihalip suggested to see if lib/clang/10.0.0/include is getting added to the include paths properly.

[ikitayama]

@nathanchance thanks! Our machine, HPE Apollo is backed by Red Hat Enterprise Linux Server release 7.5 (Maipo), and it doesn't seem to contain Python 3, so I can not try your script at this moment.

[ikitayama]

$ clang -Wp,-MD,scripts/mod/.devicetable-offsets.s.d -nostdinc -isystem /opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/include -I./arch/arm64/include -I./arch/arm64/include/generated -I./include -I./arch/arm64/include/uapi -I./arch/arm64/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -mlittle-endian -DKASAN_SHADOW_SCALE_SHIFT=3 -Qunused-arguments -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -no-integrated-as -Werror=unknown-warning-option -mgeneral-regs-only -DCONFIG_AS_LSE=1 -DCONFIG_CC_HAS_K_CONSTRAINT=1 -fno-asynchronous-unwind-tables -DKASAN_SHADOW_SCALE_SHIFT=3 -fno-delete-null-pointer-checks -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-format-invalid-specifier -Wno-gnu -Wno-tautological-compare -mno-global-merge -Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -pg -Wdeclaration-after-statement -Wimplicit-fallthrough -Wvla -Wno-pointer-sign -fno-strict-overflow -fno-merge-all-constants -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -fsanitize=kernel-address -mllvm -asan-mapping-offset=0xdffffe8000000000 -mllvm -asan-globals=1 -mllvm -asan-instrumentation-with-call-threshold=10000 -mllvm -asan-stack=0 --param asan-instrument-allocas=1 -fsanitize=shift -fsanitize=integer-divide-by-zero -fsanitize=unreachable -fsanitize=signed-integer-overflow -fsanitize=bounds -fsanitize=object-size -fsanitize=bool -fsanitize=enum -fsanitize-coverage=trace-pc -fsanitize-coverage=trace-cmp -DKBUILD_BASENAME='"devicetable_offsets"' -DKBUILD_MODNAME='"devicetable_offsets"'  -fverbose-asm -S -o scripts/mod/devicetable-offsets.s scripts/mod/devicetable-offsets.c
In file included from scripts/mod/devicetable-offsets.c:3:
In file included from ./include/linux/mod_devicetable.h:13:
In file included from ./include/linux/uuid.h:12:
./include/linux/string.h:9:10: fatal error: 'stdarg.h' file not found
#include <stdarg.h>
         ^~~~~~~~~~
1 error generated.

[ikitayama]

GCC reports the include directory where stdarg.h lives

$ gcc -print-file-name=include
/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/bin/../lib/gcc/aarch64-linux/8.2.0/include

, but locally built trunk Clang does not:

$ clang -print-file-name=include
/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/include

[ikitayama]

Now I am getting lots of these:

./include/linux/mm.h:162:2: warning: unannotated fall-through between switch labels [-Wimplicit-fallthrough]
        case 72:
        ^

Have you guys seen these before on arm64?

[nathanchance]

This is being tracked in #236. It is disabled in -next and will be disabled in the next kbuild pull request (so 5.4-rc1 will have that resolved).

[nathanchance]

GCC reports the include directory where stdarg.h lives

$ gcc -print-file-name=include
/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/bin/../lib/gcc/aarch64-linux/8.2.0/include

, but locally built trunk Clang does not:

$ clang -print-file-name=include
/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/include

This seems like a configuration issue, can you post your clang configure command?

[ikitayama]

This is how I configure clang and other sub-probjects atop of llvm.

cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=On -DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${now} -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_FLAGS="" -DLLVM_TARGETS_TO_BUILD="AArch64;NVPTX" -DLLVM_ENABLE_PROJECTS="clang;openmp;libcxx;libcxxabi;libunwind;lld" ../llvm

[nathanchance]

Wonder if it could somehow be related to libcxx;libcxxabi, since I don't enable those projects for the kernel...

[ikitayama]

Did not change at all.

On Sat, Sep 21, 2019 at 2:45 AM Nathan Chancellor [e-mail]
wrote:

Wonder if it could somehow be related to libcxx;libcxxabi, since I don't
enable those projects for the kernel...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ClangBuiltLinux/linux/issues/656?email_source=notifications&email_token=AAHLBZZ7AQ5U7VHX7ZYUZS3QKUD2JA5CNFSM4IWYVAP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HNIEQ#issuecomment-533648402,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHLBZ4PWAXUYVCHFX3DZCDQKUD2JANCNFSM4IWYVAPQ
.

[ikitayama]

I’ll check out your Clang config/build script and
Let you know how it goes on ThindrtX2.

On Sat, Sep 21, 2019 at 7:17 Itaru Kitayama [e-mail]
wrote:

Did not change at all.

On Sat, Sep 21, 2019 at 2:45 AM Nathan Chancellor <
[email protected]> wrote:

Wonder if it could somehow be related to libcxx;libcxxabi, since I don't
enable those projects for the kernel...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ClangBuiltLinux/linux/issues/656?email_source=notifications&email_token=AAHLBZZ7AQ5U7VHX7ZYUZS3QKUD2JA5CNFSM4IWYVAP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HNIEQ#issuecomment-533648402,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHLBZ4PWAXUYVCHFX3DZCDQKUD2JANCNFSM4IWYVAPQ
.

[nickdesaulniers]

or -DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${now} (not sure how that would affect the resulting build).

@ikitayama my CMake invocation usually looks like:

cmake .. -DCMAKE_BUILD_TYPE=Release -G Ninja -DCMAKE_C_COMPILER=/android1/android-master/prebuilts/clang/host/linux-x86/clang-r353983c/bin/clang -DCMAKE_CXX_COMPILER=/android1/android-master/prebuilts/clang/host/linux-x86/clang-r353983c/bin/clang++ -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_TARGETS_TO_BUILD="AArch64;ARM;X86"

[ikitayama]

Our Apollo server will be offline from this weekend for about a month (expected). I will try to re-visit this Issue when it comes back on-line. So, please keep this Issue open.

[ikitayama]

By the way, I heard Travis now can offer Arm, would you consider deploying it on that arch?

[nathanchance]

https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support

I would like to eventually do this. The main issue is apt.llvm.org only does x86_64 right now, meaning we would need to build tip of tree LLVM ourselves but I am not sure any of us have consistent access to arm64 hardware and cross compiling LLVM for arm64 does not exactly sound fun :(

[ikitayama]

Our HPE Apollo is back after some downtime. I am still seeing the same error when trying to build kernel with my trunk Clang.

[ikitayama]

Hi,
Turned out my local build returns some irrelevant directory using a full path returned by

clang -print-file-name=include

. So, I am in the meantime setting it manually in the Makefile.
Do you guys know how to set it properly?

[ikitayama]

Now I am able to build the kernel with trunk Clang 10ish.

[ikitayama]

$ clang -print-search-dirs
programs: =/fefs/home/r1050/opt/clang/current/bin:/fefs/home/r1050/opt/clang/201912120719/bin:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0/../../../../aarch64-linux-gnu/bin
libraries: =/fefs/home/r1050/opt/clang/201912120719/lib/clang/10.0.0:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0/../../../../lib64:/lib/../lib64:/usr/lib/../lib64:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0/../../../../aarch64-linux-gnu/lib:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0/../../..:/fefs/home/r1050/opt/clang/201912120719/bin/../lib:/lib:/usr/lib
$ clang -print-file-name=include
/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/include

I am puzzled, but can someone see what's wrong with my build configuration?

[nickdesaulniers]

➜  ~ clang -print-search-dirs
programs: =/android0/llvm-project/llvm/build/bin:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/bin
libraries: =/android0/llvm-project/llvm/build/lib/clang/10.0.0:/usr/lib/gcc/x86_64-linux-gnu/8:/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../lib64:/lib/x86_64-linux-gnu:/lib/../lib64:/usr/lib/x86_64-linux-gnu:/usr/lib/../lib64:/usr/lib/x86_64-linux-gnu/../../lib64:/usr/lib/gcc/x86_64-linux-gnu/8/../../..:/android0/llvm-project/llvm/build/bin/../lib:/lib:/usr/lib
➜  ~ clang -print-file-name=include
/android0/llvm-project/llvm/build/lib/clang/10.0.0/include

/android0/llvm-project/ is my llvm tree.

[ikitayama]

Nick, is it also complicated Clang’s library finding scheme?

[ikitayama]

The directory, /fefs/home/r1050/opt/clang/201912120719/lib/clang/10.0.0, does exist, but Clang
reports with the option -print-file-name=include that picked up the second one from the "libraries" list of directoeis.

$ clang -print-search-dirs
programs: =/fefs/home/r1050/opt/clang/current/bin:/fefs/home/r1050/opt/clang/201912120719/bin:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0/../../../../aarch64-linux-gnu/bin
libraries: =/fefs/home/r1050/opt/clang/201912120719/lib/clang/10.0.0:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0/../../../../lib64:/lib/../lib64:/usr/lib/../lib64:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0/../../../../aarch64-linux-gnu/lib:/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/lib/gcc/aarch64-linux-gnu/8.2.0/../../..:/fefs/home/r1050/opt/clang/201912120719/bin/../lib:/lib:/usr/lib
$ clang -print-file-name=include
/opt/arm/gcc-8.2.0_Generic-AArch64_RHEL-7_aarch64-linux/include