Linux: Building VirtualBox kernel modules with Clang and DKMS

[torvic9]

Hi,

I don't know where to report this, but...let me try :) feel free to close if it's the wrong place.

So I finally managed to build current 5.7.10 kernel with LLVM=1 LLVM_IAS=1 with clang11 from the release/11.x branch, and the IAS related patches I found (among others) here: https://github.com/ClangBuiltLinux/linux/issues/1049 ,
and which I assembled together here: https://github.com/torvic9/linux57-vd/blob/master/0006-clang-llvm-ias-support.patch

The problem is that I cannot build the VBox host modules using CC=clang LD=ld.lld. I get the following errors:

./arch/x86/include/asm/page_64.h:49:2: error: expected '(' after 'asm'
        alternative_call_2(clear_page_orig,
        ^
/arch/x86/include/asm/segment.h:266:2: error: expected '(' after 'asm'
        alternative_io ("lsl %[seg],%[p]",
        ^

./arch/x86/include/asm/special_insns.h:212:2: error: expected '(' after 'asm'
        alternative_io(".byte " __stringify(NOP_DS_PREFIX) "; clflush %P0",
        ^

and many more of the same type.

Compiling the kernel with clang10 and without LLVM_IAS results in working VBox modules.

Is this an issue with the fact that I used LLVM_IAS for the kernel?
Or is it rather related to using clang11 instead of clang10?

Thanks!

[dileks]

LLVM_IAS=1 is still categorized EXPERIMENTAL.

Unfortunately, you forgot about some important informations which might help.
What (snapshot) versions of clang-10 and clang-11?
What version of virtualbox?

As said in #1049:
Linux v5.8-rcX together with LLVM_IAS=1 and clang-11 is a good choice.
Doing some -no-integrated-as workarounds helped but not in any case with Linux v5.7.y.
As you have seen by yourself it depends also on the clang version.

Compiling/assembling LLVM_IAS=1 plus LLVM=1 is another story.

Currently, I use this kernel-release:

$ cat /proc/version 
Linux version 5.8.0-rc6-4-amd64-llvm11-ias ([email protected]@iniza) (clang version 11.0.0 (https://github.com/llvm/llvm-project cebd637c88624dfd44520848cb1f43dc8a02ba80), LLD 11.0.0 (https://github.com/llvm/llvm-project cebd637c88624dfd44520848cb1f43dc8a02ba80)) #4~bullseye+dileks1 SMP 2020-07-22

...with some additional patches - the one to support LLVM_IAS=1 are in my series: 5 patches.

Unsure - when I will update #1049 - but Linux v5.7.y is no more on my radar or interest.
I have Debian's 5.7.6 plus my Linux 5.8-rcX (LLVM_IAS=1 + LLVM=1) kernel installed.

Parts like "elfcore: build with -no-integrated-as when LLVM_IAS=1" I published in #1049.
Some parts (diffs) I do not know where you have it from and what it is fixing.

Without further informations you are at your own...

Update: Add clang-11 to good choice.

[nickdesaulniers]

The problem is that I cannot build the VBox host modules

What config is that?

[dileks]

Updated #1049.

[1] https://github.com/ClangBuiltLinux/linux/issues/1049#issuecomment-662609378

[torvic9]

Here is the version info:
clang version 11.0.0 (git://github.com/llvm/llvm-project 8f8ec9927623eb54f003993b3d8411fc0130ca90)

I'm not sure it's the IAS. The modules cannot be built either way with the clang version mentioned above, i.e. with the kernel built with/without IAS. The kernel itself builds, installs and runs just fine.

Is it possible thatt additional flags have to be passed to the VBox make command?

What works is clang10.0.1(rc4) without IAS, on the same kernel+config.

[ihalip]

What config is that?

VirtualBox modules are out-of-tree.

[dileks]

We have parts of "VBOX" in-tree - depending on your Linux-kernel version :-) - not a full support like VirtualBox upstream.

$ grep VBOX /boot/config-5.8.0-rc6-4-amd64-llvm11-ias 
CONFIG_DRM_VBOXVIDEO=m
CONFIG_VBOXGUEST=m
CONFIG_VBOXSF_FS=m

Here on Debian we have virtualbox-dkms to build out-of-tree with DKMS.

[torvic9]

Yes, I mean the out-of-tree host modules (vboxdrv etc.).
I also use dkms to build them, with the following make command added to dkms.conf:

MAKE[0]='make CC=clang LD=ld.lld KERNELRELEASE=${_kernver}'
(_kernver is of course the kernel version)

[dileks]

I am building a new kernel and will try with virtualbox-dkms version 6.1.12-dfsg-6 from Debian/unstable after it has finished.

[torvic9]

Thanks!
You're right, I forgot some info.
VBox version is 6.1.12.
clang10 is: clang version 10.0.1 (git://github.com/llvm/llvm-project 77d76b71d7df39b573dfa1e391096a040e9b7bd3)
clang11 is: clang version 11.0.0 (git://github.com/llvm/llvm-project 8f8ec9927623eb54f003993b3d8411fc0130ca90)

Some parts (diffs) I do not know where you have it from and what it is fixing.

The only part that I didn't get from here is the last diff, which eliminates __force_order errors I encountered.
Here's the link to LKML: https://lkml.org/lkml/2020/5/27/766

I'm aware that IAS is still experimental, I'm just trying out a few things (and, as you can see, I don't have the required knowledge...). I have not yet moved to 5.8.

I'm typing this from 5.7.10 built with LLVM=1 LLVM_IAS=1:
Linux version 5.7.10-1-vd (linux57-vd@manjaro) (clang-vd clang version 11.0.0 (git://github.com/llvm/llvm-project 8f8ec9927623eb54f003993b3d8411fc0130ca90)

I can paste the logfile from the failed module building process if that helps.

[nickdesaulniers]

Ah! out of tree/dkms. Yeah, that's not tested. IIRC, @ColinIanKing mentioned being able to work around this. @ColinIanKing do you remember the particular issue you saw with DKMS?

@torvic9 how do we fetch this out of tree code, and invoke a build of it?

[dileks]

You need linux-headers packages installed:

# dpkg -l | grep linux-headers | awk '/^ii/ {print $1 " " $2 " " $3}' | column -t
ii  linux-headers-5.7.0-1-amd64   5.7.6-1
ii  linux-headers-5.7.0-1-common  5.7.6-1
ii  linux-headers-amd64           5.7.6-1

Install VBox DKMS package, build+install and check DKMS status:

# apt update && apt install virtualbox-dkms

# dpkg -l | grep dkms | awk '/^ii/ {print $1 " " $2 " " $3}' | column -t
ii  acpi-call-dkms   1.1.0-6
ii  dkms             2.8.2-2
ii  virtualbox-dkms  6.1.12-dfsg-6

# dkms install -m virtualbox -v 6.1.12 -k 5.7.0-1-amd64

# dkms status
acpi-call, 1.1.0, 5.7.0-1-amd64, x86_64: installed
virtualbox, 6.1.12, 5.7.0-1-amd64, x86_64: installed

[dileks]

$ sudo dpkg -i linux-headers-5.8.0-rc6-6-amd64-llvm11-ias_5.8.0~rc6-6~bullseye+dileks1_amd64.deb

$ sudo dkms build -m virtualbox -v 6.1.12 -k 5.8.0-rc6-6-amd64-llvm11-ias

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
make -j4 KERNELRELEASE=5.8.0-rc6-6-amd64-llvm11-ias -C /lib/modules/5.8.0-rc6-6-amd64-llvm11-ias/build M=/var/lib/dkms/virtualbox/6.1.12/build....(bad exit status: 2)
Error! Bad return status for module build on kernel: 5.8.0-rc6-6-amd64-llvm11-ias (x86_64)
Consult /var/lib/dkms/virtualbox/6.1.12/build/make.log for more information.

make-log_vbox-6-1-12_llvm11-ias.txt

[dileks]

My hack was to add a temporary symlink /path/to/clang-$VER -> /usr/bin/gcc.

[dileks]

With that hack:

# cd /usr/bin/
# mv gcc gcc.orig
# ln -sf clang-11 gcc

# dkms build -m virtualbox -v 6.1.12 -k 5.8.0-rc6-6-amd64-llvm11-ias            

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
make -j4 KERNELRELEASE=5.8.0-rc6-6-amd64-llvm11-ias -C /lib/modules/5.8.0-rc6-6-amd64-llvm11-ias/build M=/var/lib/dkms/virtualbox/6.1.12/build......(bad exit status: 2)
Error! Bad return status for module build on kernel: 5.8.0-rc6-6-amd64-llvm11-ias (x86_64)
Consult /var/lib/dkms/virtualbox/6.1.12/build/make.log for more information.

# cat /var/lib/dkms/virtualbox/6.1.12/build/make.log
DKMS make.log for virtualbox-6.1.12 for kernel 5.8.0-rc6-6-amd64-llvm11-ias (x86_64)
Thu Jul 23 19:36:14 CEST 2020
make: Entering directory '/usr/src/linux-headers-5.8.0-rc6-6-amd64-llvm11-ias'
  CC [M]  /var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/SUPDrv.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/SUPDrvGip.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/SUPDrvSem.o
  CC [M]  /var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/SUPDrvTracer.o
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
    RTCCUINTREG uOld = this_cpu_read(cpu_tlbstate.cr4);
                                     ^
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:38: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:760:17: error: initializing 'RTCCUINTREG' (aka 'unsigned long long') with an expression of incompatible type 'void'
    RTCCUINTREG uOld = this_cpu_read(cpu_tlbstate.cr4);
                ^      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:764:24: error: use of undeclared identifier 'cpu_tlbstate'
        this_cpu_write(cpu_tlbstate.cr4, uNew);
                       ^
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:764:24: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:764:24: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:764:24: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:764:24: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:764:24: error: use of undeclared identifier 'cpu_tlbstate'
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.c:764:24: error: use of undeclared identifier 'cpu_tlbstate'
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[2]: *** [scripts/Makefile.build:281: /var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/linux/SUPDrv-linux.o] Fehler 1
make[2]: *** Es wird auf noch nicht beendete Prozesse gewartet....
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/SUPDrvTracer.o: warning: objtool: .text+0x7: indirect jump found in RETPOLINE build
/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/SUPDrvTracer.o: warning: objtool: supdrvTracerProbeFireStub() is missing an ELF size annotation
make[1]: *** [scripts/Makefile.build:497: /var/lib/dkms/virtualbox/6.1.12/build/vboxdrv] Fehler 2
make: *** [Makefile:1762: /var/lib/dkms/virtualbox/6.1.12/build] Error 2
make: Leaving directory '/usr/src/linux-headers-5.8.0-rc6-6-amd64-llvm11-ias'

[dileks]

The problem is in DKMS code:

3614: # Unset environment variables that may interfere with the build
3615: unset CC CXX CFLAGS CXXFLAGS LDFLAGS

[1] https://github.com/dell/dkms/blob/master/dkms#L3614

[torvic9]

My hack was to add a temporary symlink /path/to/clang-$VER -> /usr/bin/gcc.

I worked around it by adding the make command with CC=clang to dkms.conf (see above).

Edit: however your error messages are completely different to the asm errors?

[dileks]

@torvic9 ah cool.

Hmm, you need to add all $MAKE_OPTS from your make line of the kernel-build including LLVM_IAS=1 and LLVM=1?

[dileks]

Edit: however your error messages are completely different to the asm errors?

I have some additional x86/asm patches in my series and Linux v5.8-rc6.
Cannot say which VBox version supports it.

[torvic9]

I only added CC.
I'm now rebuilding clang11 and will try again, however still on 5.7. Will switch to 5.8 soon though.

Where can I find your current series?

[dileks]

That's the minimal patchset:

https://github.com/ClangBuiltLinux/linux/issues/1049#issuecomment-662609378

[torvic9]

(I deleted my previous comment as it was just a typo)

[dileks]

There are two different version of @arndb crc32c patch - the original [1] is for Linux 5.8 - I published an adapted version for Linux v5.7 in #1043.

[1] https://git.kernel.org/herbert/cryptodev-2.6/c/44623b2818f4a442726639572f44fd9b6d0ef68c

[dileks]

Depending on your llvm-toolchain (here: release/11.x Git) - the patches you mentioned are no more necessary.
In my last posting in #1043 I displayed the version I used.

[torvic9]

Depending on your llvm-toolchain (here: release/11.x Git) - the patches you mentioned are no more necessary.

Here: git-release/11.x from July 21st. Still necessary.
I haven't updated the toolchain yet due to a problem with the build-llvm.py script (I'll open an issue there).

To get back on topic: I still encounter the same ASM related errors on 5.8-rc6 when compiling VBox host modules 6.1.12.

[dileks]

Nice to see you use our tools like tc-build and infrastructure :-).

As said https://github.com/llvm/llvm-project/commit/cebd637c88624dfd44520848cb1f43dc8a02ba80 fixed for me all recent Linux/x86 problems reported in the CBL issue-tracker. So, I will keep this version for a while.
Maybe till there is an official llvmorg-11.0.0-rc1 release tag.

[1] https://github.com/ClangBuiltLinux/tc-build/issues/104
[2] https://github.com/llvm/llvm-project/tags

[dileks]

Depending on your llvm-toolchain (here: release/11.x Git) - the patches you mentioned are no more necessary.

Here: git-release/11.x from July 21st. Still necessary.

Be precise as possible via clang -v.
The output displays a unique commit-id when build from Git.

[dileks]

You looked at https://github.com/llvm/llvm-project/commit/833f8c958601bb640ba6a25d627c1dc58dad14d2?

[torvic9]

Be precise as possible via clang -v.

I did, maybe you missed it: clang version 11.0.0 (git://github.com/llvm/llvm-project 8f8ec9927623eb54f003993b3d8411fc0130ca90)

I'll build a new toolchain later and report back; this laptop doesn't have enough processing power for a PGO build...

Nice to see you use our tools like tc-build and infrastructure :-).

Sure, it's fun experimenting with clang :) I hope I can get a better knowledge about it. Technically I don't have any real use for it.

[dileks]

Hehe, it's CPU power plus enough RAM (and/or SWAP) for PGO and/or Thin-LTO.

BTW, I am doing a stage1-only build.

[nickdesaulniers]

ok, let's report then to the DKMS developers. Let's find their bug tracker.

[dileks]

@nickdesaulniers

This is a known issue which I reported as "dkms: Build with an alternative like compiler rather than gcc" to debian-kernel ML in July 2018.

Mario Limonciello pointed me to the above lines in the dkms source.

It does not hurt to report to dkms upstream but as @torvic9 pointed out you can override/pass individual make options via/to dkms.conf.
( Untested by /me. )

[1] https://lists.debian.org/debian-kernel/2018/07/subject.html

[torvic9]

The issue is not related to LLVM_IAS=1.

With 5.8-rc6 and clang-11 3c1fca803bc14617b67ba2125e1b4b77190e9f86, I get the same ASM errors with and without IAS.

However, using instead clang-10 77d76b71d7df39b573dfa1e391096a040e9b7bd3, I do not get these ASM errors, instead I have similar error messages as in @dileks comment above.

It seems that the host modules are not yet fully supported, a similar bug report can be found here:
https://bugs.launchpad.net/ubuntu/+source/virtualbox/+bug/1884652 (see log file)

[torvic9]

Here are some patches to make VBox 6.1.12 host modules compatible with kernel 5.8.
I have only tried with gcc so far, but they seem to build correctly.
There is one patch for the kernel itself, and 3 patches for the host modules.

https://www.virtualbox.org/ticket/19644

[dileks]

That means be patient - wait for VBox version 6.1.14 which will have support for Linux-kernel version 5.8.

UPDATE: s/6.1.13/6.1.14 (stable versions have even numbers)

[dileks]

[1] says:

virtualbox (6.1.12-dfsg-8) unstable; urgency=medium

  * debian/patches/kernel-5.8-1.patch:
  * debian/patches/kernel-5.8-2.patch:
  * debian/patches/kernel-5.8-3.patch:
    - apply upstream proposed patches to fix a build failure with kernel 5.8
      (note, this requires also a kernel patch)
      https://www.virtualbox.org/ticket/19644

 -- Gianfranco Costamagna <[email protected]>  Thu, 30 Jul 2020 18:58:32 +0200

[1] https://metadata.ftp-master.debian.org/changelogs//contrib/v/virtualbox/virtualbox_6.1.12-dfsg-8_changelog

[dileks]

Missing patch against Linux v5.8-rc7:

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -315,6 +315,7 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size,

        return 0;
 }
+EXPORT_SYMBOL(map_kernel_range);

 int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot,
                struct page **pages)
@@ -2129,6 +2130,7 @@ struct vm_struct *__get_vm_area_caller(unsigned long size, unsigned long flags,
        return __get_vm_area_node(size, 1, flags, start, end, NUMA_NO_NODE,
                                  GFP_KERNEL, caller);
 }
+EXPORT_SYMBOL(__get_vm_area_caller);

 /**
  * get_vm_area - reserve a contiguous kernel virtual area

UPDATE: Attached a simple RFC patch without any Links or Credits.

0001-mm-vmalloc-Export-map_kernel_range-and-__get_vm_area_patch.txt

0001-mm-vmalloc-vbox-Export-map_kernel_range-and-__get_vm_patch.txt

[dileks]

VBox patches to build Linux v5.8-rcX:

1. https://sources.debian.org/src/virtualbox/6.1.12-dfsg-8/debian/patches/
2. https://sources.debian.org/data/contrib/v/virtualbox/6.1.12-dfsg-8/debian/patches/kernel-5.8-1.patch
3. https://sources.debian.org/data/contrib/v/virtualbox/6.1.12-dfsg-8/debian/patches/kernel-5.8-2.patch
4. https://sources.debian.org/data/contrib/v/virtualbox/6.1.12-dfsg-8/debian/patches/kernel-5.8-3.patch

[torvic9]

Thanks. Already applied all 4 directly from vbox site, i.e. put them in PKGBUILDs (vbox and kernel). No missing patch there. Builds fine with rc7 and gcc. Clang not tested yet.

(Rather wait for .14, I think odd numbers are dev versions)

[dileks]

@torvic9

Good.
I will try with the kernel-patch only as Debian/unstable has the 3 other ones.
Here it's 36 degrees Celsius @18:02 so I will do build this night.

[torvic9]

35° here :) good luck. I'll try with clang as well and report back.

[dileks]

I was able to build a Vbox kernel-modules with my gcc-symlink trick.
Need some more playing with dkms.conf.

[dileks]

Passing LLVM_IAS=1 results in:

  AR      /var/lib/dkms/virtualbox/6.1.12/build/built-in.a
  MODPOST /var/lib/dkms/virtualbox/6.1.12/build/Module.symvers
ERROR: modpost: "__force_order" [/var/lib/dkms/virtualbox/6.1.12/build/vboxdrv/vboxdrv.ko] undefined!

Patch from @arndb https://lkml.org/lkml/2020/5/27/766

[1] https://lore.kernel.org/patchwork/patch/1248382/

[dileks]

Applied @arndb patch on top of Linux v5.8-rc7.

With this diff:

--- /usr/src/virtualbox-6.1.12/dkms.conf.orig   2020-07-30 18:58:32.000000000 +0200
+++ /usr/src/virtualbox-6.1.12/dkms.conf        2020-08-01 11:57:36.420490455 +0200
@@ -1,5 +1,6 @@
 PACKAGE_NAME="virtualbox"
 PACKAGE_VERSION="6.1.12"
+MAKE="make CC=clang-11 LD=ld.lld-11 AR=llvm-ar-11 NM=llvm-nm-11 OBJCOPY=llvm-objcopy-11 OBJDUMP=llvm-objdump-11 OBJSIZE=llvm-size-11 READELF=llvm-readelf-11 STRIP=llvm-strip-11 LLVM_IAS=1 -C /lib/modules/5.8.0-rc7-6-amd64-llvm11-ias/build M=/var/lib/dkms/virtualbox/6.1.12/build"
 CLEAN="rm -f *.*o"
 BUILT_MODULE_NAME[0]="vboxdrv"
 BUILT_MODULE_LOCATION[0]="vboxdrv"

Build VBox kernel-modules:

root@iniza:~# LC_ALL=C dkms build -m virtualbox -v 6.1.12 -k 5.8.0-rc7-6-amd64-llvm11-ias              

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
make -j4 KERNELRELEASE=5.8.0-rc7-6-amd64-llvm11-ias CC=clang-11 LD=ld.lld-11 AR=llvm-ar-11 NM=llvm-nm-11 OBJCOPY=llvm-objcopy-11 OBJDUMP=llvm-objdump-11 OBJSIZE=llvm-size-11 READELF=llvm-readelf-11 STRIP=llvm-strip-11 LLVM_IAS=1 -C /lib/modules/5.8.0-rc7-6-amd64-llvm11-ias/build M=/var/lib/dkms/virtualbox/6.1.12/build.........................
cleaning build area...

DKMS: build completed.

Install VBox kernel-modules:

root@iniza:~# LC_ALL=C dkms install -m virtualbox -v 6.1.12 -k 5.8.0-rc7-6-amd64-llvm11-ias

Check DKMS status:

root@iniza:~# dkms status -m virtualbox
virtualbox, 6.1.12, 5.8.0-rc7-6-amd64-llvm11-ias, x86_64: installed

Check status of virtualbox.service and start it:

root@iniza:~# systemctl status virtualbox.service 
● virtualbox.service - LSB: VirtualBox Linux kernel module
     Loaded: loaded (/etc/init.d/virtualbox; generated)
     Active: inactive (dead)
       Docs: man:systemd-sysv-generator(8)

root@iniza:~# systemctl start virtualbox.service 

root@iniza:~# systemctl status virtualbox.service 
● virtualbox.service - LSB: VirtualBox Linux kernel module
     Loaded: loaded (/etc/init.d/virtualbox; generated)
     Active: active (exited) since Sat 2020-08-01 12:00:41 CEST; 8min ago
       Docs: man:systemd-sysv-generator(8)
    Process: 8247 ExecStart=/etc/init.d/virtualbox start (code=exited, status=0/SUCCESS)

Aug 01 12:00:41 iniza systemd[1]: Starting LSB: VirtualBox Linux kernel module...
Aug 01 12:00:41 iniza virtualbox[8247]: Loading VirtualBox kernel modules... vboxdrv vboxnetflt vboxnetadp.
Aug 01 12:00:41 iniza systemd[1]: Started LSB: VirtualBox Linux kernel module.

Check loaded VBox kernel-modules:

root@iniza:~# lsmod | grep vbox
vboxnetadp             28672  0
vboxnetflt             32768  0
vboxdrv               520192  2 vboxnetadp,vboxnetflt

Check dmesg-outputs:

root@iniza:~# LC_ALL=C dmesg -T | tail
[Sat Aug  1 11:47:13 2020] wlp1s0: associate with 50:d4:f7:2e:17:da (try 1/3)
[Sat Aug  1 11:47:13 2020] wlp1s0: RX AssocResp from 50:d4:f7:2e:17:da (capab=0x411 status=0 aid=1)
[Sat Aug  1 11:47:13 2020] wlp1s0: associated
[Sat Aug  1 11:47:13 2020] IPv6: ADDRCONF(NETDEV_CHANGE): wlp1s0: link becomes ready
[Sat Aug  1 12:00:40 2020] vboxdrv: loading out-of-tree module taints kernel.
[Sat Aug  1 12:00:40 2020] vboxdrv: Found 4 processor cores
[Sat Aug  1 12:00:40 2020] vboxdrv: TSC mode is Invariant, tentative frequency 1597812133 Hz
[Sat Aug  1 12:00:40 2020] vboxdrv: Successfully loaded version 6.1.12_Debian (interface 0x002d0001)
[Sat Aug  1 12:00:40 2020] VBoxNetFlt: Successfully started.
[Sat Aug  1 12:00:40 2020] VBoxNetAdp: Successfully started.

[dileks]

New VBox dkms.conf diff:

--- /usr/src/virtualbox-6.1.12/dkms.conf.orig   2020-07-30 18:58:32.000000000 +0200
+++ /usr/src/virtualbox-6.1.12/dkms.conf        2020-08-01 13:06:54.522773507 +0200
@@ -1,6 +1,11 @@
 PACKAGE_NAME="virtualbox"
 PACKAGE_VERSION="6.1.12"
-CLEAN="rm -f *.*o"
+CC_FOR_BUILD="CC=clang-11"
+LD_FOR_BUILD="LD=ld.lld-11"
+LLVM_UTILS="AR=llvm-ar-11 NM=llvm-nm-11 OBJCOPY=llvm-objcopy-11 OBJDUMP=llvm-objdump-11 OBJSIZE=llvm-size-11 READELF=llvm-readelf-11 STRIP=llvm-strip-11"
+MAKE_OPTS="${CC_FOR_BUILD} ${LD_FOR_BUILD} ${LLVM_UTILS} LLVM_IAS=1"
+MAKE[0]="make V=1 ${MAKE_OPTS} -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build"
+CLEAN="make -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build clean"
 BUILT_MODULE_NAME[0]="vboxdrv"
 BUILT_MODULE_LOCATION[0]="vboxdrv"
 DEST_MODULE_LOCATION[0]="/updates"

[dileks]

Full VBox dkms.conf:

[ /usr/src/virtualbox-6.1.12/dkms.conf ]

PACKAGE_NAME="virtualbox"
PACKAGE_VERSION="6.1.12"
CC_FOR_BUILD="CC=clang-11"
LD_FOR_BUILD="LD=ld.lld-11"
LLVM_UTILS="AR=llvm-ar-11 NM=llvm-nm-11 OBJCOPY=llvm-objcopy-11 OBJDUMP=llvm-objdump-11 OBJSIZE=llvm-size-11 READELF=llvm-readelf-11 STRIP=llvm-strip-11"
MAKE_OPTS="${CC_FOR_BUILD} ${LD_FOR_BUILD} ${LLVM_UTILS} LLVM_IAS=1"
MAKE[0]="make V=1 ${MAKE_OPTS} -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build"
CLEAN="make -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build clean"
BUILT_MODULE_NAME[0]="vboxdrv"
BUILT_MODULE_LOCATION[0]="vboxdrv"
DEST_MODULE_LOCATION[0]="/updates"
BUILT_MODULE_NAME[1]="vboxnetadp"
BUILT_MODULE_LOCATION[1]="vboxnetadp"
DEST_MODULE_LOCATION[1]="/updates"
BUILT_MODULE_NAME[2]="vboxnetflt"
BUILT_MODULE_LOCATION[2]="vboxnetflt"
DEST_MODULE_LOCATION[2]="/updates"
AUTOINSTALL="yes"

[dileks]

I was able to import my stretch-buildd.vmdk, boot VM and shutdown.

[torvic9]

No luck here.
Still the same error messages as in OP:

In file included from ./arch/x86/include/asm/page.h:12:
./arch/x86/include/asm/page_64.h:49:2: error: expected '(' after 'asm'
        alternative_call_2(clear_page_orig,
        ^

Any ideas?

Edit: happens both with and without IAS-built kernel.

[dileks]

@torvic9
I do not see this.
But I have some extra patches applied like from .

[1] https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/asm

[dileks]

I have sent an email to the Debian virtualbox maintainers and asked for an alternative handling of dkms.conf within virtualbox-dkms package in Debian/unstable.

[torvic9]

I'm giving up on this one, it's not a big issue for me as I can just build with either clang-10 or gcc.
I retried with binutils 2.35, x86/asm from tip as suggested, and a freshly built toolchain, but no... still the same ASM error messages, happens with every version of clang-11 I've tried.
Could be a config issue, but I'm not going to investigate further anymore, already spent too much time with this ;)

[nickdesaulniers]

So this issue is only observed with clang-11, but not clang-10? If so, that's curious. Are you building with both explicitly suffixed? Ie. make CC=clang-11 and make CC=clang-10 as opposed to make CC=clang?

[torvic9]

I'm using LLVM=1 (and additionally LLVM_IAS=1 with clang-11).

[nathanchance]

I have re-read this thread a couple times and I cannot follow it.

Is there any way to get a TL;DR with some commands to reproduce this issue locally? If this is a LLVM regression from 10.0.x to 11.0.x, we might have some time to get this fixed in ToT then backport to LLVM 11.0.0 before it ships. If not, we should ideally shoot for LLVM 11.0.1. We are committing to supporting the latest stable release from LLVM (patch) so it would be nice not to have a regression in LLVM 11 right off the bat :)

[torvic9]

Hi Nathan,

I have not tested it for some time now, but let me try to give you a TL;DR.
Please note that I'm not very knowledgeable about this topic, I cannot exclude the possibility that the error comes from my system.

1. build clang (here: with ThinLTO and PGO) with the build-llvm.py script using release-11.x branch
2. build the 5.8 kernel with that toolchain (make LLVM=1)
3. use DKMS to build the VBox modules by passing CC=clang LD=ld.lld to DKMS' make command

Result: a lot of ASM related error messages as mentioned in OP. I can post the whole build log file later this week.

Instead, when using clang-10 from the Arch repos to build the kernel (version 10.0.1 at the time of writing), the VBox modules build just fine without errors.
Also builds fine with clang-10 built with the build-llvm.py script, although it's been more than a month since I last built clang-10.

Not related to IAS btw, as the errors pop up with and without using LLVM_IAS=1 with clang-11.

Hope this helps.

[nathanchance]

I am starting to think this might be something with DKMS because it builds fine without it? If I am holding it wrong, let me know. I would like to avoid installing virtualbox-dkms on my server.

$ cd "$(mktemp -d)"

$ git clone https://github.com/ClangBuiltLinux/tc-build
...

$ tc-build/build-llvm.py --assertions --branch "release/11.x" --build-stage1-only --projects "clang;lld" --targets X86
...

$ curl -LSs https://download.virtualbox.org/virtualbox/6.1.12/VirtualBox-6.1.12a.tar.bz2 | tar -xjf -

$ curl -LSs https://github.com/archlinux/svntogit-community/raw/20aca7aca83f45bb1505c44955a3614d461f171f/trunk/021-linux-5-8.patch | patch -d VirtualBox-6.1.12 -p1
...

$ VirtualBox-6.1.12/src/VBox/HostDrivers/linux/export_modules.sh --folder vboxmod
VirtualBox-6.1.12/src/VBox/HostDrivers/linux/export_modules.sh: 120: [: Illegal number:
VirtualBox-6.1.12/src/VBox/HostDrivers/linux/export_modules.sh: 204: [: Illegal number:

$ curl -LSs https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.8.3.tar.xz | tar -xJf -

$ PATH=${PWD}/tc-build/build/llvm/stage1/bin:${PATH} make -C linux-5.8.3 -skj"$(nproc)" LLVM=1 defconfig bzImage

$ PATH=${PWD}/tc-build/build/llvm/stage1/bin:${PATH} make -C linux-5.8.3 -skj"$(nproc)" LLVM=1 M=${PWD}/vboxmod
/tmp/tmp.l1J0MXwOsz/vboxmod/vboxdrv/SUPDrvTracer.o: warning: objtool: .text+0x7: indirect jump found in RETPOLINE build
/tmp/tmp.l1J0MXwOsz/vboxmod/vboxdrv/SUPDrvTracer.o: warning: objtool: supdrvTracerProbeFireStub() is missing an ELF size annotation
/tmp/tmp.l1J0MXwOsz/vboxmod/vboxdrv/r0drv/linux/memuserkernel-r0drv-linux.o: warning: objtool: VBoxHost_RTR0MemKernelCopyFrom()+0xe: redundant CLD
/tmp/tmp.l1J0MXwOsz/vboxmod/vboxdrv/r0drv/linux/memuserkernel-r0drv-linux.o: warning: objtool: VBoxHost_RTR0MemKernelCopyTo()+0xe: redundant CLD
WARNING: Symbol version dump "Module.symvers" is missing.
         Modules may not have dependencies or modversions.
WARNING: modpost: Symbol info of vmlinux is missing. Unresolved symbol check will be entirely skipped.

$ fd -e ko . vboxmod
vboxmod/vboxdrv/vboxdrv.ko

[torvic9]

I repeated your procedure, except for the toolchain build, and effectively I don't get any errors either.
So it could be DKMS, or maybe even the kernel config?

[nathanchance]

It certainly could be configuration related. If you want to upload that, I can try it locally.

[torvic9]

I'm now trying with the default Arch config and I'll report back. I guess it's enough to build bzImage only, without the modules?

EDIT: Arch config works, too.
EDIT 2: Mine works as well. I think you're right - this could be an issue with DKMS.

[nickdesaulniers]

@ColinIanKing I think mentioned encountering issues with DKMS when building with Clang. I don't know the details, @ColinIanKing was that ever reported/resolved?

[dileks]

@torvic9

Which version of DKMS?

[torvic9]

pacman -Qi dkms | grep -i vers                     
Version                  : 2.8.3-1

@nickdesaulniers , you probably mean this: https://github.com/dell/dkms/issues/124