Gentoolto: Building the Linux kernel using LTO

Created on 27 Jan 2018  Â·  108Comments  Â·  Source: InBetweenNames/gentooLTO

I find it interesting that there hasn't been more push to build the kernel using LTO. I've found a couple of mailing list threads about it, including a patchset to let it happen, but there wasn't a lot of interest upstream. I've created this issue as a way to track what the current LTO progress in the kernel is, and possibly even add some patchsets to let it happen. I know I'd for sure use it on my router if I could with OpenWRT.

picture InBetweenNames

Most helpful comment

For those interested, it still works on 5.9

https://gist.github.com/jiblime/ee9f22f4073fbd8ee8add4ffd78abe70

Edit 2: I use this janky patch right after applying the lto patch so 5.9.x patches apply cleanly: https://gist.github.com/jiblime/ee9f22f4073fbd8ee8add4ffd78abe70#file-revert-cec_c-changes-diff

Using make clean instead of make mrproper should still be fine, I had an errant patch causing issues.

Here's a fun config introduced in 5.9: DEBUG_FORCE_FUNCTION_ALIGN_32B

There are cases that a commit from one domain changes the function                                                                                                                                                                                                                                                  │  
  │ address alignment of other domains, and cause magic performance                                                                                                                                                                                                                                                     │  
  │ bump (regression or improvement). Enable this option will help to                                                                                                                                                                                                                                                   │  
  │ verify if the bump is caused by function alignment changes, while                                                                                                                                                                                                                                                   │  
  │ it will slightly increase the kernel size and affect icache usage.                                                                                                                                                                                                                                                  │  
  │                                                                                                                                                                                                                                                                                                                     │  
  │ It is mainly for debug and performance tuning use.

https://github.com/torvalds/linux/blob/v5.9/Makefile#L891

ifdef CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_32B
KBUILD_CFLAGS += -falign-functions=32
endif

Definitely a plus for Intel, unsure if it would be beneficial for newer AMD Zen 1/+/2/3/4/etc.

Unless I am misinterpreting Agner's analysis, it appears Zen models would still have best cache hits with -falign-functions=16 (or lower?)
Source: https://www.agner.org/optimize/microarchitecture.pdf

20.2 Instruction fetch
The instruction fetcher can fetch 32 aligned bytes of code per clock cycle from the level-1 code cache, _according to AMD documents_, but the maximum measured throughput is only 16 bytes per clockfor a single thread. The bottleneck is probably an "instruction byte queue" between the fetch unit and the decoder.

The maximum measured fetch rate for two threads per core is 12 bytes per clock per thread. This maximum is obtained when all instructions are 8 bytes long. The fetch rate is lower for other instruction sizes.

picture jiblime on 1 Nov 2020
👍2 🎉1

All 108 comments

@InBetweenNames I would use it on my router too, I think at the time the gcc LTO toolchain wasn't very mature and few were able too make much use of it, particularly embedded* Linux where there would be most interest. Without that buy-in the kernel devs weren't going to let the patches in.

Perhaps resurrecting the patch set and getting it working again could be successful now that lto support is pretty ubiquitous in distros and most embedded devs must be using it by now for their user space.

  • embedded toolchains tend to be quite conservative and stick around for a while
sjnewbury picture sjnewbury on 22 Oct 2018

Seems some remnants of those patches are still in the kernel (notably DISABLE_LTO so it doesn't use it for vdso), so I tried with 4.19.1. Formerly used scripts/gcc-ld but didn't work for me so I used gold. I doubt it's accomplishing anything built this way (size barely changed with other defaults). Despite using gcc-ar, was also complaining about the lto plugin unless -ffat-lto. Patchset used to use -fwhole-program too but that didn't work. Nonetheless, thought I'd do the crazy thing and build the kernel with:

make -j8 AR=gcc-ar NM=gcc-nm LD=ld.gold KCFLAGS="-march=native -O3 -falign-functions=32 -fipa-pta -fno-semantic-interposition -fgraphite-identity -floop-nest-optimize -flto=8 -ffat-lto-objects" DISABLE_LTO=-fno-lto

Which.. worked.. and booted fine. I am now the proud owner of a kernel that 30% bigger than before, probably not faster, and set out to kill my dog, but thankfully running in QEMU away from my dog.
Edit: well, removing LTO with the same options does make it like 10% even bigger.

ionenwks picture ionenwks on 11 Nov 2018

It might be interesting to compare the speed of some syscall- / kernel-bound workloads when successfully built with LTO. Anyone with an idea on how to start benchmarking our gains or losses?

gcs-github picture gcs-github on 11 Nov 2018

Not sure, but if you check the kernel mailing list plenty of those benchmarks have been done in the past. I remember seeing pretty big gains with LTO, but not sure if those reflected into any gain for daily usage.
Some more info about how to benchmark the kernel: https://github.com/graysky2/kernel_gcc_patch

darkbasic picture darkbasic on 11 Nov 2018
👍1

One thing about LTO is you have to build as many of your models into the kernel as possible... so it knows what it can eliminate when linking... so you get the biggest gains on a completely static kernel (this of course breaks somethings that load firmware etc... some of that you can work around by building in the blobs though).

cb88 picture cb88 on 14 Nov 2018

Andi Kleen rebased his LTO patches for the Kernel on 4.20 recently. I've tried it out but had no luck and several module errors along the way. Nevertheless, you can find these patches here: https://github.com/andikleen/linux-misc/tree/lto-420-1

ms178 picture ms178 on 8 Jan 2019
👍1

^ Didn't experiment much but gave it a quick try and it built fine for me with my configuration and CONFIG_LTO=y which auto-adds -flto -fno-fat-lto-objects. Didn't try a generic one and I use almost no modules which, as stated in the other above post, is better suited for a LTO kernel anyway.

Looks like it's using the gcc-ld script and working properly. I do have gold as my default linker (been using it even for kernel).

I imagine it may make more of a difference on a less-lean kernel, but my resulting 4.20 kernel is about 1% smaller than my old, didn't try to boot and also no idea for any performance gains.

ionenwks picture ionenwks on 9 Jan 2019
👍1

@ionenwks I'm trying to replicate the steps on a gentoo system to build an LTO'd kernel. However, I always error out on the linking portion: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: error: arch/x86/kernel/head_64.o: requires unsupported dynamic reloc 11; recompile with -fPIC
I have added this flag to the base KBUILD_CFLAGS but to no effect. I also have ld.gold enabled by default.
What version of GCC and binutils are you using? Did you make any configurations to the Makefile from Andi Kleen's repo?

Promaethius picture Promaethius on 11 Aug 2019

Hmm... I tried again both with the lto-420-1 branch from back then along with same configuration and the newer lto-5.1-3, and I'm getting the same errors as you now (using gcc 8.3.0 and ld.bfd 2.32).

Not sure what I was using back then but looking at the date I assume I was on gcc 8.2 and binutils 2.30 I think? It's only something I tried real quick, I had no intention to stick with that for now (or boot it).

Edit: Retried with gold as default (switched back to bfd a while ago), doesn't work either, not with current toolchain anyway.
Edit2: And no, I hadn't made any changes, used as-is.

ionenwks picture ionenwks on 11 Aug 2019

@ionenwks thank you for taking the time to check through the issue! I was afraid it was a toolchain version issue, so I wonder if this is a reportable bug?
I'm going to take some time today and check if its a gcc or binutils issue.
Edit: I'm throwing some more configuration testing into this mess. Found this article over on the patch list: https://patchwork.kernel.org/patch/10000627/

Promaethius picture Promaethius on 11 Aug 2019

I was able to build 5.0-1 successfully, however I did not test it and the system it was on it now gone.

-fPIC would cause reloc .text errors if it was built with visibility=hidden or ssp(but the Makefile already filters that). Maybe -flinker-output=rel would make sense here, but I couldn't get the syntax correct. ~because parts of the kernel build are still static, and static objects aren't able to find PIC references~. If anyone knows his full patchset without a kernel tree that'd be really helpful.

jiblime picture jiblime on 12 Aug 2019

@jiblime You can find his patchset on the kernel mailing list but it won't really help: https://lkml.org/lkml/2017/11/27/1052
THIN_ARCHIVES was a config option that was removed in 4.19+. It went around the supposed issue of ld -r. But, I've narrowed it down to a ld issue of some sort. There are kernel patches that let you fPIC the code but they aren't working for me yet.

Promaethius picture Promaethius on 12 Aug 2019

@Promaethius Thanks for the link. I'm currently trying to edit arch/x86/entry/vdso/Makefile to work. At the very bottom you can try appending flags after ${LD} but nothing has worked for me, even the options to specifically suppress the error.

I went and checked a regular kernel and I noticed that it's normal(?) for a hidden symbol to be there.

Both comands ran were readelf vclock_gettime.o -s

5.1-3 LTO:

Symbol table '.symtab' contains 25 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS vclock_gettime.c
     2: 0000000000000000     0 SECTION LOCAL  DEFAULT    1 
     3: 0000000000000000     0 SECTION LOCAL  DEFAULT    3 
     4: 0000000000000000     0 SECTION LOCAL  DEFAULT    4 
     5: 0000000000000000   174 FUNC    LOCAL  DEFAULT    1 do_hres
     6: 0000000000000000     0 SECTION LOCAL  DEFAULT    5 
     7: 0000000000000000     0 SECTION LOCAL  DEFAULT    7 
     8: 0000000000000000     0 SECTION LOCAL  DEFAULT    8 
     9: 0000000000000000     0 SECTION LOCAL  DEFAULT   10 
    10: 0000000000000000     0 SECTION LOCAL  DEFAULT   11 
    11: 0000000000000000     0 SECTION LOCAL  DEFAULT   12 
    12: 0000000000000000     0 SECTION LOCAL  DEFAULT   14 
    13: 0000000000000000     0 SECTION LOCAL  DEFAULT   15 
    14: 0000000000000000     0 SECTION LOCAL  DEFAULT   17 
    15: 0000000000000000     0 SECTION LOCAL  DEFAULT   19 
    16: 0000000000000000     0 SECTION LOCAL  DEFAULT   20 
    17: 0000000000000000     0 SECTION LOCAL  DEFAULT   18 
    18: 0000000000000000     0 NOTYPE  GLOBAL HIDDEN   UND vvar_vsyscall_gtod_data
    19: 00000000000000b0   111 FUNC    GLOBAL DEFAULT    1 __vdso_clock_gettime
    20: 00000000000000b0   111 FUNC    WEAK   DEFAULT    1 clock_gettime
    21: 0000000000000120    98 FUNC    GLOBAL DEFAULT    1 __vdso_gettimeofday
    22: 0000000000000120    98 FUNC    WEAK   DEFAULT    1 gettimeofday
    23: 0000000000000190    16 FUNC    GLOBAL DEFAULT    1 __vdso_time
    24: 0000000000000190    16 FUNC    WEAK   DEFAULT    1 time
readelf: Warning: compressed section '.debug_str' is corrupted

5.2.8 kernel:

Symbol table '.symtab' contains 27 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS vclock_gettime.c
     2: 0000000000000000     0 SECTION LOCAL  DEFAULT    1 
     3: 0000000000000000     0 SECTION LOCAL  DEFAULT    3 
     4: 0000000000000000     0 SECTION LOCAL  DEFAULT    4 
     5: 0000000000000000   392 FUNC    LOCAL  DEFAULT    1 do_hres
     6: 0000000000000000     0 SECTION LOCAL  DEFAULT    5 
     7: 0000000000000000     0 SECTION LOCAL  DEFAULT    7 
     8: 0000000000000000     0 SECTION LOCAL  DEFAULT    8 
     9: 0000000000000000     0 SECTION LOCAL  DEFAULT   10 
    10: 0000000000000000     0 SECTION LOCAL  DEFAULT   11 
    11: 0000000000000000     0 SECTION LOCAL  DEFAULT   12 
    12: 0000000000000000     0 SECTION LOCAL  DEFAULT   14 
    13: 0000000000000000     0 SECTION LOCAL  DEFAULT   15 
    14: 0000000000000000     0 SECTION LOCAL  DEFAULT   17 
    15: 0000000000000000     0 SECTION LOCAL  DEFAULT   19 
    16: 0000000000000000     0 SECTION LOCAL  DEFAULT   20 
    17: 0000000000000000     0 SECTION LOCAL  DEFAULT   18 
    18: 0000000000000000     0 NOTYPE  GLOBAL HIDDEN   UND vvar_vsyscall_gtod_data
    19: 0000000000000000     0 NOTYPE  GLOBAL HIDDEN   UND hvclock_page
    20: 0000000000000000     0 NOTYPE  GLOBAL HIDDEN   UND pvclock_page
    21: 0000000000000190   102 FUNC    GLOBAL DEFAULT    1 __vdso_clock_gettime
    22: 0000000000000190   102 FUNC    WEAK   DEFAULT    1 clock_gettime
    23: 0000000000000200    98 FUNC    GLOBAL DEFAULT    1 __vdso_gettimeofday
    24: 0000000000000200    98 FUNC    WEAK   DEFAULT    1 gettimeofday
    25: 0000000000000270    16 FUNC    GLOBAL DEFAULT    1 __vdso_time
    26: 0000000000000270    16 FUNC    WEAK   DEFAULT    1 time

readelf: Warning: compressed section '.debug_str' is corrupted looks to be of interest. Does this mean there needs to be more debug information built in?

jiblime picture jiblime on 12 Aug 2019

@jiblime I found this on the gcc site today: https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#index-fuse-linker-plugin-916

When a file is compiled with -flto without -fuse-linker-plugin, the generated object file is larger than a regular object file because it contains GIMPLE bytecodes and the usual final code (see -ffat-lto-objects. This means that object files with LTO information can be linked as normal object files; if -fno-lto is passed to the linker, no interprocedural optimizations are applied. Note that when -fno-fat-lto-objects is enabled the compile stage is faster but you cannot perform a regular, non-LTO link on them.

I've witnessed Andi Kleen's patchset passing -fno-fat-lto-objects without -fuse-linker-plugin. Will test this theory later today. This could explain why readelf is returning corruption, but pardon my ignorance if that's not the case.

Promaethius picture Promaethius on 12 Aug 2019

@Promaethius

I've witnessed Andi Kleen's patchset passing -fno-fat-lto-objects without -fuse-linker-plugin

That explains why he uses -fwhole-program and and -fipa-cp-clone, since collect2 would be used instead of a linker. I'm assuming he's doing that for compatibility, as GCC documentation claims it's likely to increase code size vs. bfd/gold. I wonder if GentooLTO would be able to do something better...

I believe it's a glibc issue. I've upgraded to sys-libs/glibc-2.30::gentoo and have been able to get past it. Currently recompiling since paravirtualization options, not sure which, causes it to error.

https://sourceware.org/ml/libc-alpha/2019-08/msg00029.html

* The dynamic linker no longer refuses to load objects which reference
  versioned symbols whose implementation has moved to a different soname
  since the object has been linked.  The old error message, symbol
  FUNCTION-NAME, version SYMBOL-VERSION not defined in file DSO-NAME with
  link time reference, is gone.

It emits a warning, I'm still not sure why since Andi Kleen filters LTO out of it from what I can tell.


Warnings emitted with V=2

CC arch/x86/entry/vdso/vdso32-setup.o - due to target missing
LDS arch/x86/entry/vdso/vdso.lds - due to target missing
AS arch/x86/entry/vdso/vdso-note.o - due to target missing
CC arch/x86/entry/vdso/vclock_gettime.o - due to target missing
In file included from ./arch/x86/include/asm/vgtod.h:5,
from arch/x86/entry/vdso/vclock_gettime.c:15:
arch/x86/entry/vdso/vclock_gettime.c: In function ‘do_hres’:
./include/linux/compiler.h:182:26: warning: array subscript 1 is outside array bounds of ‘u8[1]’ {aka ‘unsigned char[1]’} [-Warray-bounds]
182 | case 8: *(__u64 *)res = *(volatile __u64 *)p; break; \
| ^~~~~~
./include/linux/compiler.h:193:2: note: in expansion of macro ‘__READ_ONCE_SIZE’
193 | __READ_ONCE_SIZE;
| ^~~~
arch/x86/entry/vdso/vclock_gettime.c:37:11: note: while referencing ‘hvclock_page’
37 | extern u8 hvclock_page
| ^~~~
In file included from ./arch/x86/include/asm/vgtod.h:5,
from arch/x86/entry/vdso/vclock_gettime.c:15:
./include/linux/compiler.h:182:26: warning: array subscript 2 is outside array bounds of ‘u8[1]’ {aka ‘unsigned char[1]’} [-Warray-bounds]
182 | case 8: *(__u64 *)res = *(volatile __u64 *)p; break; \
| ^~~~~~
./include/linux/compiler.h:193:2: note: in expansion of macro ‘__READ_ONCE_SIZE’
193 | __READ_ONCE_SIZE;
| ^~~~
arch/x86/entry/vdso/vclock_gettime.c:37:11: note: while referencing ‘hvclock_page’
37 | extern u8 hvclock_page
| ^~~~
CC arch/x86/entry/vdso/vgetcpu.o - due to target missing
VDSO arch/x86/entry/vdso/vdso64.so.dbg - due to target missing
OBJCOPY arch/x86/entry/vdso/vdso64.so - due to target missing
HOSTCC arch/x86/entry/vdso/vdso2c - due to target missing
VDSO2C arch/x86/entry/vdso/vdso-image-64.c - due to target missing
CC arch/x86/entry/vdso/vdso-image-64.o - due to target missing
LDS arch/x86/entry/vdso/vdso32/vdso32.lds - due to target missing
CC arch/x86/entry/vdso/vdso32/vclock_gettime.o - due to target missing
In file included from ./arch/x86/include/asm/vgtod.h:5,
from arch/x86/entry/vdso/vdso32/../vclock_gettime.c:15,
from arch/x86/entry/vdso/vdso32/vclock_gettime.c:31:
arch/x86/entry/vdso/vdso32/../vclock_gettime.c: In function ‘do_hres’:
./include/linux/compiler.h:182:26: warning: array subscript 1 is outside array bounds of ‘u8[1]’ {aka ‘unsigned char[1]’} [-Warray-bounds]
182 | case 8: *(__u64 *)res = *(volatile __u64 *)p; break; \
| ^~~~~~
./include/linux/compiler.h:193:2: note: in expansion of macro ‘__READ_ONCE_SIZE’
193 | __READ_ONCE_SIZE;
| ^~~~
In file included from arch/x86/entry/vdso/vdso32/vclock_gettime.c:31:
arch/x86/entry/vdso/vdso32/../vclock_gettime.c:37:11: note: while referencing ‘hvclock_page’
37 | extern u8 hvclock_page
| ^~~~
In file included from ./arch/x86/include/asm/vgtod.h:5,
from arch/x86/entry/vdso/vdso32/../vclock_gettime.c:15,
from arch/x86/entry/vdso/vdso32/vclock_gettime.c:31:
./include/linux/compiler.h:182:26: warning: array subscript 2 is outside array bounds of ‘u8[1]’ {aka ‘unsigned char[1]’} [-Warray-bounds]
182 | case 8: *(__u64 *)res = *(volatile __u64 *)p; break; \
| ^~~~~~
./include/linux/compiler.h:193:2: note: in expansion of macro ‘__READ_ONCE_SIZE’
193 | __READ_ONCE_SIZE;
| ^~~~
In file included from arch/x86/entry/vdso/vdso32/vclock_gettime.c:31:
arch/x86/entry/vdso/vdso32/../vclock_gettime.c:37:11: note: while referencing ‘hvclock_page’
37 | extern u8 hvclock_page
| ^~~~
AS arch/x86/entry/vdso/vdso32/note.o - due to target missing
AS arch/x86/entry/vdso/vdso32/system_call.o - due to target missing
AS arch/x86/entry/vdso/vdso32/sigreturn.o - due to target missing
VDSO arch/x86/entry/vdso/vdso32.so.dbg - due to target missing
OBJCOPY arch/x86/entry/vdso/vdso32.so - due to target missing
VDSO2C arch/x86/entry/vdso/vdso-image-32.c - due to target missing
CC arch/x86/entry/vdso/vdso-image-32.o - due to target missing

So as I understand, it would be a huge issue to have a textrel in a/the vdso because it'd be a vulnerability in a security feature. Gentoo's wiki actually has a guide on finding and fixing textrels: https://wiki.gentoo.org/wiki/Hardened/Textrels_Guide

But hopefully there's no need to recreate anything. While the vdso*.so files have a textrel flag marked on them, scanelf -T shows that there isn't anything that would point to it.

Glibc 2.29, GCC 9.1.0

 TYPE    PAX   PERM ENDIAN STK/REL/PTL TEXTREL RPATH BIND TEXTRELS FILE 
scanelf: scanelf_file_textrels(): ELF is missing relocation information
scanelf: scanelf_file_textrels(): ELF vdso32.so has TEXTREL markings but doesnt appear to have any real TEXTREL's !?
ET_DYN PeMRxS 0755 LE --- --- R-X TEXTREL   -   LAZY  vdso32.so

It did also emit this, though:

arch/x86/kernel/dumpstack.o: warning: objtool: show_regs.cold()+0x16: sibling call from callable instruction with modified stack frame
arch/x86/kernel/dumpstack.o: warning: objtool: show_regs()+0x0: stack state mismatch: cfa1=7+24 cfa2=7+8

So it looks like it can be possible, but definitely experimental and not a daily driver for myself. I'm going to be grabbing GCC 9.2 now so I won't be getting to it anytime soon (btw, I added 20G of swap with -j5 and it still failed, dammit), but if Glibc 2.30 is the fix, I think it'd be worth a shot to try using this kernel for testing.

If you were to use a linker instead of collect2 you can run replace -fwhole-program with fuse-linker-plugin in scripts/Makefile.lto as Gnu documentation states it's best not to use the former with the latter. Optimizations that would also help LTO specifically would be -fdevirtualize-at-ltrans and the -fgraphite-identity -floop-nest-optimize options. I've used these along other flags to compile and run my kernel, but if the linking stage is too much the process will overflow and it'll end.

What's interesting is that his newest version (as far as I can tell) lacks explicit linker usage but his older versions use -fuse-linker-plugin. So I could be wrong in assuming that removing -fwhole-program is the right way to go.

jiblime picture jiblime on 13 Aug 2019

Andi Kleen's lto-5.7-2 branch branch builds and I am currently running it. I've applied the 5.7.14 patch, Gentoo distro patches, and a few other misc. patches with no rejects.

Notes:

  • nouveau does not build due to the command to building it being too long for shell

  • I wasn't able to load any modules, not when booted in or through my initrd. Used make mod2yesconfig to convert modules to being built in. [Correction 1 below]

  • ./scripts/Makefile.lto contains the settings for how LTO is done upon the kernel. I appended -flto-compression-level=9 to LTO_CFLAGS

  • In the same file, TMPDIR sets the building directory in the kernel directory instead of /tmp to prevent OOM. I copied the kernel to /var/tmp/portage instead because there are massive writes on disk during linking.

  • The primary demographic of LTO'd kernels seems to be embedded systems

The size of my LTO'd kernel is 22M, modules folder is 800K. Vs. my normal kernel at 11M and modules folder at 71M

  • It feels fast, that counts

Semi-related:

GCC 10's -O2 might be slightly slower than GCC 9's -O2

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96337#c15

Inliner changes was not targetting to make compile time faster and
compiled code slower. It was intended to reflect more closely modern C++
codebases and get faster binaries (at -O2 and -O2 -flto) without
regressing in code sizes.  In fact more inlining happens and thus we
needed to optimize inliner code carefully to avoid regressions with LTO.

If you have a -march=znver1/znver2 processor and run x86_64 multilib, rebuilding the current GCC 10.2.0 would mean a nice performance boost with this patch:

patch 1, patch 2

Refer to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95435

Correction 1: I incorrectly assumed modules weren't supported with -flto. While building everything into the kernel alleviated the issues, namely framebuffer and Logitech USB support, kernel compilation time was too long and I prefer being able to reload modules. The likely culprit in module failure was TRIM_UNUSED_KSYMS and possibly dracut defaulting to --strip the generated initrd; can't say for certain yet. I didn't get around to testing it enough but now I am able to load amdgpu in my initrd as usual instead of compiling it in.

jiblime picture jiblime on 8 Aug 2020
🎉1 
* It feels fast, that counts

Can you describe in what way?

Cheers for the gcc links too

telans picture telans on 8 Aug 2020

oooh, imma test

barolo picture barolo on 8 Aug 2020

@jiblime Could you list the patches applied? All are from gentoo's ebuild?

barolo picture barolo on 8 Aug 2020

@jiblime Could you list the patches applied? All are from gentoo's ebuild?

I haven't built it yet, but this patch applies fine to gentoo-sources-5.8 (just a diff from the lto-5.8.0-1 branch)

https://gist.github.com/telans/728b63dd07c41c9ca6e2ca3d4431db8e

Doesn't build for me unfortunately, lots of:

/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: ./.tmp_vmlinux.kallsyms1.mJXteD.ltrans123.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
telans picture telans on 8 Aug 2020
👍1

also, there's no 5.7.14 patch

https://cdn.kernel.org/pub/linux/kernel/v5.x/patch-5.7.14.xz

telans picture telans on 8 Aug 2020

that patch is already applied... nvm, messed up something, ended with upstream master somehow...
this will impact my sdd most def

barolo picture barolo on 8 Aug 2020

@telans
No problem. The first thing I noticed was my dmesg timestamps were lower than usual :p ideally I'll set up a phoronix benchmark to have actual data.

relocation R_X86_64_32S

Are you using ld.gold as your default linker? The Linux kernel needs either GCC/ld.bfd or Clang/ld.lld. https://github.com/InBetweenNames/gentooLTO/issues/338

sys-devel/gcc-10.2.0::gentoo was built with the following:
USE="(cxx) fortran graphite lto (multilib) nls nptl objc openmp pch pgo sanitize ssp zstd (-ada) -d -debug -doc (-fixed-point) -go (-hardened) -jit (-libssp) -objc++ -objc-gc -pie -systemtap -test -vanilla -vtv" ABI_X86="(64)"

sys-devel/binutils-2.34-r2::gentoo was built with the following:
USE="gold multitarget nls plugins static-libs -default-gold -doc -test" ABI_X86="(64)"

@barolo
https://github.com/jiblime/linux-misc/commits/lto-5.7-prjc-r3
You can pull the patches from here or clone the single branch and build off that. The CPPC patch doesn't work for me, so I leave it off just in case it would case me to fail to boot.
It's a bit messy, I'm still not the greatest at making clean commits. I chose the 5.7-2 branch instead of 5.8 because I wanted to try the Project C scheduler (previously named BMQ, now abbreviated prjc). I'll try the 5.8 branch sometime.

I generally download a vanilla tarball from kernel.org (v5.7, v5.8, etc) and apply the Gentoo patches and incremental patches afterwards. That way I don't have to worry about rejected patches as often

jiblime picture jiblime on 8 Aug 2020

@jiblime thanks for the branch, made it much easier for me. Compiling

barolo picture barolo on 8 Aug 2020

compiled almost cleanly for me, didn't take that long too, had a bunch of "-Wstringop-overflow" warnings for Bluetooth module. Didn't boot for me with error related to scsi.
With modules builtin it is 20M , modules dir i 1M
I have nvme and amdgpu on that box, gonna try to strip it a bit more

barolo picture barolo on 8 Aug 2020

Narrowed it down, hidpp/logitech's stuff makes it crash, and it doesn't switch to amdgpu output @jiblime it seems like you\ve had similar issues, how did you solve them?
Edit. Cleaned it a bit, built amdgpu, bluetooth, and logitech hidpp as modules, the remaining issue seems to be that framebuffer isn't being switched during boot

barolo picture barolo on 8 Aug 2020

Are you using ld.gold as your default linker? The Linux kernel needs either GCC/ld.bfd or Clang/ld.lld.

Nope, using ld.bfd ( or at least I haven't changed it.)

sys-devel/gcc-10.2.0::gentoo was built with the following:
USE="(cxx) fortran graphite lto (multilib) nls nptl openmp pch pgo (pie) sanitize ssp vtv zstd (-ada) -d -debug -doc (-fixed-point) -go (-hardened) (-jit) (-libssp) -objc -objc++ -objc-gc -systemtap -test -vanilla" ABI_X86="(64)"

sys-devel/binutils-2.34-r2::gentoo was built with the following:
USE="gold nls plugins -default-gold -doc -multitarget -static-libs -test" ABI_X86="(64)"

Forcing LD=ld.bfd doesn't change anything either. I thought it might have been an issue with ripping a patch from the lto-5.8-1 branch, however, the branch too builds with the same relocation errors

Same issue with lto-5.7-2

telans picture telans on 8 Aug 2020

Update, managed to run it and reach the desktop. The issue was with building all modules in.
So I took my working config as base, used genkernel and made sure that it runs without LTO enabled first, then enabled LTO and booted into desktop successfully.
Ended with a bunch of drivers disabled, most importantly for network and sata, luckily my main is a pcie one.
Each failed module had disagrees about version of symbol module_layout in dmesg, gonna investigate it now.

Edit. It seems that all of those are modules that weren't built in, so it seems that initramfs isn't working for me
Edit2. I'm typing from it, had to recompile it cleanly, cleaned it a bit and built some stuff in, module loading doesn't seem to work as I still got two of those disagrees... warnings

Can't really compare it yet, since it seems to use diff schedulers than I had with zen kernel, and spends more time at lower frequencies, would have to bench it properly to test it seriously.

I can already tell though that building that kernel is significantly faster under it

barolo picture barolo on 9 Aug 2020

My gut tells me it has something to do with the -fPIE flag

On Sun, Aug 9, 2020, 3:27 AM Greg Shuiske notifications@github.com wrote:

Update, managed to run it and reach the desktop. The issue was with
building all modules in.
So I took my working config as base, used genkernel and made sure that it
runs without LTO enabled first, then enabled LTO.
Ended with a bunch of drivers disabled, most importantly for network and
sata, luckily my main is a pcie one.
Each module had disagrees about version of symbol module_layout in dmesg,
gonna investigate it now.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/InBetweenNames/gentooLTO/issues/90#issuecomment-671029236,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFFNN3MKRMCYOAXWIEI7BLR7ZTWVANCNFSM4EN5L3PQ
.

Promaethius picture Promaethius on 9 Aug 2020

@Promaethius I've solved that by having those with warnings changed to built-in, It's running fine so far, gonna bench it with something now.
My whole kernel with inbuilt stuff is 10 MB, with useless 4MB initramfs, for gaming desktop

barolo picture barolo on 9 Aug 2020

when linux 5.7.14 w/ the lto-5.7-2 using ld.bfd from binutils 2.35 and gcc-10.1.0 i'm getting countless errors all telling me to recompile with -fPIE similar to what @telans has mentioned:

  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  LDFINAL vmlinux.o
kernel/bpf/core.c: In function 'bpf_patch_insn_single':
kernel/bpf/core.c:442:3: warning: writing 8 bytes into a region of size 0 [-Wstringop-overflow=]
  442 |   memcpy(prog->insnsi + off, patch, sizeof(*patch));
      |   ^
./include/linux/filter.h:550:20: note: at offset 0 to object 'insnsi' with size 0 declared here
  550 |   struct bpf_insn  insnsi[0];
      |                    ^
  MODPOST vmlinux.o
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LDFINAL .tmp_vmlinux.kallsyms1
/usr/bin/ld: arch/x86/kernel/head_64.o: relocation R_X86_64_32S against symbol `early_top_pgt' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/entry/entry_64.o: relocation R_X86_64_32S against symbol `cpu_tss_rw' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/entry/vdso/vma.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/entry/vdso/vdso32-setup.o: relocation R_X86_64_32S against symbol `vdso_image_32' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/entry/entry_64_compat.o: relocation R_X86_64_32S against symbol `cpu_tss_rw' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/kvm/vmx/vmenter.o: relocation R_X86_64_32S against symbol `kvm_rebooting' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/realmode/init.o: relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/kernel/acpi/wakeup_64.o: relocation R_X86_64_32S against symbol `saved_magic' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/kernel/ftrace_64.o: relocation R_X86_64_32S against symbol `ftrace_trace_function' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/crypto/aesni-intel_asm.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/crypto/aesni-intel_avx-x86_64.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/crypto/ghash-clmulni-intel_asm.o: relocation R_X86_64_32S against `.rodata.cst16.bswap_mask' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/power/hibernate_asm_64.o: relocation R_X86_64_32S against symbol `saved_context' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans0.ltrans.o: relocation R_X86_64_32S against hidden symbol `cpu_hw_events' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans1.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans2.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans3.ltrans.o: relocation R_X86_64_32S against symbol `em_bsf' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans4.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans5.ltrans.o: relocation R_X86_64_32S against hidden symbol `cpu_bit_bitmap' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans6.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans7.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans8.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans9.ltrans.o: relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans10.ltrans.o: relocation R_X86_64_32S against `.data..read_mostly' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans11.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans12.ltrans.o: relocation R_X86_64_32S against hidden symbol `sig_sicodes.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans13.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans14.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans15.ltrans.o: relocation R_X86_64_32S against hidden symbol `rcu_data.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans16.ltrans.o: relocation R_X86_64_32S against hidden symbol `tk_fast_mono.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans17.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans18.ltrans.o: relocation R_X86_64_32S against hidden symbol `trace_types_lock' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans19.ltrans.o: relocation R_X86_64_32S against hidden symbol `ftrace_common_fields.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans20.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans21.ltrans.o: relocation R_X86_64_32S against hidden symbol `init_task' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans22.ltrans.o: relocation R_X86_64_32S against hidden symbol `contig_page_data' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans23.ltrans.o: relocation R_X86_64_32S against hidden symbol `vma_interval_tree_augment_rotate.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans24.ltrans.o: relocation R_X86_64_32S against hidden symbol `vmap_area_list' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans25.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans26.ltrans.o: relocation R_X86_64_32S against hidden symbol `__per_cpu_offset' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans27.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans28.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans29.ltrans.o: relocation R_X86_64_32S against `.data..percpu' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans30.ltrans.o: relocation R_X86_64_32S against hidden symbol `bit_wait_table.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans31.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans32.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans33.ltrans.o: relocation R_X86_64_32S against hidden symbol `tty_drivers' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans34.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans35.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans36.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans37.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans38.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans39.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans40.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans41.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans42.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans43.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans44.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans45.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans46.ltrans.o: relocation R_X86_64_32S against `.rodata..c_jump_table' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans47.ltrans.o: relocation R_X86_64_32S against hidden symbol `queue_io_timeout_entry.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans48.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans49.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans50.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans51.ltrans.o: relocation R_X86_64_32S against hidden symbol `dev_attr_boot_vga.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans52.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans53.ltrans.o: relocation R_X86_64_32S against hidden symbol `v86d_path.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans54.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans55.ltrans.o: relocation R_X86_64_32S against hidden symbol `_ctype' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans56.ltrans.o: relocation R_X86_64_32S against hidden symbol `_ctype' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans57.ltrans.o: relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans58.ltrans.o: relocation R_X86_64_32S against hidden symbol `vc_cons' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans59.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans60.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans61.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans62.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans63.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans64.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans65.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans66.ltrans.o: relocation R_X86_64_32S against hidden symbol `execlists_submit_request.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans67.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans68.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans69.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans70.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans71.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans72.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans73.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans74.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans75.ltrans.o: relocation R_X86_64_32S against hidden symbol `device_ktype.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans76.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans77.ltrans.o: relocation R_X86_64_32S against hidden symbol `dma_buf_fops.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans78.ltrans.o: relocation R_X86_64_32S against hidden symbol `scsi_host_type.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans79.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans80.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans81.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans82.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans83.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans84.ltrans.o: relocation R_X86_64_32S against hidden symbol `ant_toggle_lookup.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans85.ltrans.o: relocation R_X86_64_32S against hidden symbol `dev_attr_manufacturer.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans86.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans87.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans88.ltrans.o: relocation R_X86_64_32S against hidden symbol `scsi_host_type.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans89.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans90.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans91.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans92.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans93.ltrans.o: relocation R_X86_64_32S against hidden symbol `tpacpi_all_drivers.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans94.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans95.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans96.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans97.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans98.ltrans.o: relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans99.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans100.ltrans.o: relocation R_X86_64_32S against hidden symbol `rtnl_af_ops.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans101.ltrans.o: relocation R_X86_64_32S against hidden symbol `offload_base.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans102.ltrans.o: relocation R_X86_64_32S against hidden symbol `loggers.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans103.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans104.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans105.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans106.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans107.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans108.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans109.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans110.ltrans.o: relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans111.ltrans.o: relocation R_X86_64_32S against hidden symbol `ptype_lock.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans112.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans113.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans114.ltrans.o: relocation R_X86_64_32S against hidden symbol `init_uts_ns' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans115.ltrans.o: relocation R_X86_64_32S against hidden symbol `swevent_htable.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans116.ltrans.o: relocation R_X86_64_32S against hidden symbol `ieee802_1d_to_ac' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans117.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans118.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans119.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans120.ltrans.o: relocation R_X86_64_32S against hidden symbol `tcf_action_policy.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans121.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans122.ltrans.o: relocation R_X86_64_32S against hidden symbol `con2fb_map.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans123.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans124.ltrans.o: relocation R_X86_64_32S against symbol `_stext' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans125.ltrans.o: relocation R_X86_64_32S against hidden symbol `init_mm' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans126.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans127.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/lib/getuser.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/lib/putuser.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: drivers/base/firmware_loader/builtin/iwlwifi-4965-2.ucode.gen.o: warning: relocation in read-only section `.builtin_fw'
collect2: error: ld returned 1 exit status
make: *** [Makefile:1111: vmlinux] Error 1
mssx86 picture mssx86 on 10 Aug 2020
😕1

My gut tells me it has something to do with the -fPIE flag

I agree. It looks like PIE may be the main reason why there are build issues. I'm unsure of what implementation is best to remove auto PIE just for the kernel.

When testing out with gcc -m32 {-fpie ; -fPIE ; -Wl,-pie} file.c -o m32. 

$ gcc -m32 -Wl,-pie file.c -o m32
$ file m32

ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 3.2.0, not stripped

Only -Wl,-pie worked, meaning -Wl,-no-pie or -Wl,-no-PIE would be ideal? Except it error'd on me, and file tells me that -m64 compiles aren't PIE even with those set. Could be a problem with my toolchain and inexpertise.

I'd suggest just rebuilding GCC without pie, though you may need to rebuild multiple parts of your toolchain that have been emerged with pie, namely binutils/glibc/libtool, but some other suggestions you can try first:

make KCFLAGS="-fno-pie -fno-PIE -Wl,-no-pie" LDFLAGS_MODULE="Wl,-no-pie" -j"$(grep processor /proc/cpuinfo | wc -l)"

or

In ./scripts/Makefile.build, I believe these are inherited for all builds unless the Makefile has a specific override.

EXTRA_AFLAGS   :=
EXTRA_CFLAGS   :=
EXTRA_CPPFLAGS :=
EXTRA_LDFLAGS  :=

You can also append V=1 to make and get verbose output to see what flags are being used if that may help:

With the suggested KCFLAGS and LDFLAGS_MODULE

gcc -Wp,-MD,drivers/gpio/.gpiolib-sysfs.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include -I./arch/x86/include -I./arch/x86/include/generated -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -march=native -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mindirect-branch=thunk-extern -mindirect-branch-register -fno-jump-tables -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-address-of-packed-member -O2 -fno-allow-store-data-races -fstack-protector-strong -Wno-unused-but-set-variable -Wimplicit-fallthrough -Wno-unused-const-variable -fomit-frame-pointer -fno-var-tracking-assignments -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -Wno-zero-length-bounds -Wno-array-bounds -Wno-stringop-overflow -Wno-restrict -Wno-maybe-uninitialized -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -fcf-protection=none -Wno-packed-not-aligned -flto -flto-compression-level=9 -fno-fat-lto-objects -fno-pie -fno-PIE -Wl,-no-pie -DKBUILD_MODFILE='"drivers/gpio/gpiolib-sysfs"' -DKBUILD_BASENAME='"gpiolib_sysfs"' -DKBUILD_MODNAME='"gpiolib_sysfs"' -c -o drivers/gpio/gpiolib-sysfs.o drivers/gpio/gpiolib-sysfs.c

And without:

  gcc -Wp,-MD,kernel/power/.qos.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -march=native -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mindirect-branch=thunk-extern -mindirect-branch-register -fno-jump-tables -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-address-of-packed-member -O2 -fno-allow-store-data-races -fstack-protector-strong -Wno-unused-but-set-variable -Wimplicit-fallthrough -Wno-unused-const-variable -fomit-frame-pointer -fno-var-tracking-assignments -fno-inline-functions-called-once -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -Wno-zero-length-bounds -Wno-array-bounds -Wno-stringop-overflow -Wno-restrict -Wno-maybe-uninitialized -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -fcf-protection=none -Wno-packed-not-aligned -flto -flto-compression-level=9 -fno-fat-lto-objects -DDEBUG    -DKBUILD_MODFILE='"kernel/power/qos"' -DKBUILD_BASENAME='"qos"' -DKBUILD_MODNAME='"qos"' -c -o kernel/power/qos.o kernel/power/qos.c


Some more info on PIE:
https://wiki.gentoo.org/wiki/Hardened/Toolchain#Position_Independent_Executables_.28PIEs.29
https://stackoverflow.com/questions/34519521/why-does-gcc-create-a-shared-object-instead-of-an-executable-binary-according-to
https://stackoverflow.com/questions/43367427/32-bit-absolute-addresses-no-longer-allowed-in-x86-64-linux

jiblime picture jiblime on 10 Aug 2020

Some issues I found with the v5.8-1 linux-misc LTO branch:


config SINGLE_LINK

Problem: Final stages of kernel compile failed

Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround
make: *** [Makefile:1142: vmlinux] Error 1
KALLSYMS_SINGLE

Appending the aforementioned did not work. Disabling and rebuilding the kernel fixed that issue.


KALLSYMS_SINGLE,

Kernel failed to boot. Only tested this one a couple of times.



CRYPTO_DEV_CCP_DD

An AMD specific thing, to my knowledge it has never worked for anyone.

May cause your kernel to not boot if you have it built in. I used to have it as a module until I realized nobody has it working on Linux.

> dmesg | grep ccp

ccp 0000:09:00.1: runtime IRQ mapping not provided by arch
ccp 0000:09:00.1: enabling device (0000 -> 0002)
ccp 0000:09:00.1: enabling bus mastering
ccp 0000:09:00.1: ccp: unable to access the device: you might be running a broken BIOS.

Interesting thread on AMD's cryptographic coprocessor: https://forum.gigabyte.us/thread/9479/bug-linux-x570-aorus-initialize


Regarding module configs

What I current have:

# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_MODULE_SIG=y
CONFIG_MODULE_SIG_FORCE=y
CONFIG_MODULE_SIG_ALL=y
# CONFIG_MODULE_SIG_SHA1 is not set
# CONFIG_MODULE_SIG_SHA224 is not set
CONFIG_MODULE_SIG_SHA256=y
# CONFIG_MODULE_SIG_SHA384 is not set
# CONFIG_MODULE_SIG_SHA512 is not set
CONFIG_MODULE_SIG_HASH="sha256"
# CONFIG_MODULE_COMPRESS is not set
# CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS is not set
# CONFIG_UNUSED_SYMBOLS is not set
# CONFIG_TRIM_UNUSED_KSYMS is not set

This is for 5.8.0; what may work for me might not work for you.


Options unrelated to LTO

These are needed for my system, UEFI with systemd-boot as my manager

EFI_STUB=y
This is required when booting a kernel + initrd with systemd-boot
https://www.freedesktop.org/wiki/Software/systemd/systemd-boot/

X86_SYSFB and FB_SIMPLE
DRM and AMDGPU
Required to get simple framebuffer to load, then to switch over to amdgpu

EFIVAR_FS=y and EFI_VARS=n
The first one is modern, I don't know why the second one still exists.

EXTRA_FIRMWARE
I've always done this, but you can also try embedding driver firmware into the kernel. (the binary blobs in /lib/firmware)

Gentoo's distro patches that autoselects options that most need to boot is very helpful

# My current dracut invoke -- never used the stub before now but "shrug"
dracut --kver {kernel-version} --no-compress --fstab  --uefi-stub /usr/lib/systemd/boot/efi/linuxx64.efi.stub

# My dracut.conf

add_dracutmodules+="
        systemd systemd-initrd busybox dracut-systemd
        kernel-modules drm usrmount fs-lib
        crypt dm lvm lvmmerge securityfs "

omit_dracutmodules+="i18n shutdown ecryptfs nbd dmraid mdraid plymouth bootchart dash btrfs stratis cifs nfs biosdevname iscsi fcoe fcoe-uefi"

add_drivers+=" amdgpu vfio_pci vfio vfio_iommu_type1 vfio_virqfd "
# I comment this out when the modules are built in

compiled almost cleanly for me, didn't take that long too, had a bunch of "-Wstringop-overflow" warnings for Bluetooth module. Didn't boot for me with error related to scsi.
With modules builtin it is 20M , modules dir i 1M
I have nvme and amdgpu on that box, gonna try to strip it a bit more

I got those too, bluetooth and a couple other things. Apparently it's related to GCC aggressively inlining for no reason, and it's been a thing for a while. I believe the the fix is to directly tell GCC not to inline those modules. I think it would be a good idea to make a list of modules to tell GCC not to inline.

I'm sure there's a kernel doc on that somewhere...Linus really hates -O3 for its inlining lol. Sidenote, GCC 10 no longer enables -funroll-loops at any opt level. Side-side note: RDRAND is re-enabled, rerun cpuid2cpuflags to see if your list has changed

jiblime picture jiblime on 10 Aug 2020

I have GCC built without -fPIE, That would explain why I'm succeding. I still can't load modules though

barolo picture barolo on 10 Aug 2020

@barolo Perhaps try KALLSYMS_ALL?

jiblime picture jiblime on 10 Aug 2020

@jiblime are you suggesting that rebuilding of a toolchain w/o pie as a possible solution to my problem here? i don't have any modules.

mssx86 picture mssx86 on 10 Aug 2020

@jblime I'm yet to try KALLSYMS_ALL but I've already added -fdevirtualize-at-ltrans -fgraphite-identity -floop-nest-optimize, checked that it's actually added during compilation and it works! Running it right now, now to complete the trifecta PGO would be needed xD

barolo picture barolo on 10 Aug 2020
😄1

added -fdevirtualize-at-ltrans -fgraphite-identity -floop-nest-optimize

Benchmarks would be awesome if you could

@barolo Mirroring a phoronix kernel benchmark would probably be the easiest with benches such as: https://www.phoronix.com/scan.php?page=article&item=linux-50-sliding&num=2

telans picture telans on 10 Aug 2020

@telans I was just about to ask... xD
Could someone recommend a simple bench to test the kernel?
( possibly not Phoronix.. )

Edit. Averaged Motionmark 1.1 is 20% faster than under Zen kernel, both set to performance mode. I will try the same kernel now just without LTO/graphite, to have a valid comparison.

barolo picture barolo on 10 Aug 2020

@mssx86
Yes. Ideally your toolchain and @system. If that is too time consuming it'd probably be enough to just rebuild GCC and binutils. There was a discussion about it: https://github.com/InBetweenNames/gentooLTO/issues/261 It has a paper on the overhead that PIE introduces and how to unmask the (pie) useflag.

@barolo
I did too. I'm surprised it's running haha! Check out this patch too, I think it might make sense: https://raw.githubusercontent.com/jiblime/clear-ck-gentoo-sources/5.8-lto-exp/misc/0004-Add-option-to-use-fno-semantic-interposition-and-fde.patch

I also added -fno-inline-functions -fno-inline-functions-called-once --param=large-stack-frame-growth=100 to my KCFLAGS, no more overflow warnings.

And I agree @telans, a standard would be nice to measure. It would be a lot easier to quantify/benchmark changes in the kernel vs. something system-wide. These are from perf but I wouldn't put any weight in it

5.8 LTO

# Running sched/messaging benchmark...
# 20 sender and receiver processes per group
# 10 groups == 400 processes run

     Total time: 0.077 [sec]

# Running sched/pipe benchmark...
# Executed 1000000 pipe operations between two processes

     Total time: 1.887 [sec]

       1.887349 usecs/op
         529843 ops/sec

5.7.14 no LTO

# Running sched/messaging benchmark...
# 20 sender and receiver processes per group
# 10 groups == 400 processes run

     Total time: 0.091 [sec]

# Running sched/pipe benchmark...
# Executed 1000000 pipe operations between two processes

     Total time: 5.107 [sec]

       5.107420 usecs/op
         195793 ops/sec

If only Phoronix suites were so quick 😛

jiblime picture jiblime on 10 Aug 2020

@jiblime ~so you have fully working 5.8 now? any differences from 5.7 regarding LTO?~
Could you explain those additional flags?

It's definitely quicker, you can see, feel it even without benches. The difference is so significant in execution time that I thought it's the different scheduler lowering peak cpu usage initially
Are you doing phoronix? That would save me from some pain...

Edit. I'm on zen patched 5.8 lto kernel now, for fair comparison since Zen is my kernel of choice due to fsync. Doing some phoronix tests now

barolo picture barolo on 10 Aug 2020

@jiblime can confirm that after rebuilding gcc with --disable-default-pie, i managed to build and boot with the lto'd kernel. thanks a bunch.

mssx86 picture mssx86 on 10 Aug 2020
👍2

Same here, sys-devel/gcc without (pie) works a treat. I did rebuild @system just to be safe too.

I also encountered the same issue with config SINGLE_LINK @jiblime. However, my kernel isn't any larger than before (no external modules aside from nvidia) at 9.3M vs the old at 9.5M

Boots fine but I'm coming across nvidia: disagrees about version of symbol module_layout. I did rebuild multiple time with the kernel loaded/unloaded etc. Even when lto is disabled from the kernel config options the error appears. The 'fix' for me was to remove the lto patch I mentioned above.

Looks like this might be as far as I can go with this until I get an AMD card or something along those lines. Any other suggestions? Otherwise I'm not sure what to do.

Thanks all

telans picture telans on 11 Aug 2020

Could you explain those additional flags?

@barolo -fno-inline-functions disables automatic inlining on any possible function which was the default -O2 for GCC9. The second one probably isn't helpful to disable, the third is a reduction from a max of +1000% to +100% for stack growth caused by inlining. I chose those because overly aggressive inlining in some of the kernel's code only makes it bigger with no benefit and if I understand, large stacks in the kernel will mean some instructions won't fit in the CPU cache -> slower

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=49194
https://lwn.net/Articles/166172/

@telans I've found that rebuilding the kernel will remove some symbols from Module.symvers. What I did to fix that was backup my .config, run make mrproper and move the config back. After rebuilding the kernel a new Module.symvers will be generated. I also have MODVERSIONS and MODULE_SRCVERSION_ALL off.

I think external kernel modules are affected by what the kernel was built with. Maybe try adding -ffat-lto-objects to x11-drivers/nvidia-drivers?

jiblime picture jiblime on 11 Aug 2020

Finally managed to get things sorted out and have some sane results: it's a Zen 5.8 -O3, native vs Zen 5.8 -O3, native, graphite, LTO
@telans
[ it's a 4 core , AMD APU ]


G'MIC [ graphics processing suite ]

``` G'MIC
Test: 2D Function Plotting, 1000 Times
Seconds < Lower Is Better
gmic-zen-LTO_0 ... 120.13 |=============================================
gmic-zen_0 ....... 134.51 |==================================================

G'MIC
Test: Plotting Isosurface Of A 3D Volume, 1000 Times
Seconds < Lower Is Better
gmic-zen-LTO_0 .. 17.11 |===========================================
gmic-zen_0 ...... 20.17 |===================================================


G'MIC
Test: 3D Elevated Function In Random Colors, 100 Times
Seconds < Lower Is Better
gmic-zen-LTO_0 .. 96.70  |===========================================
gmic-zen_0 ...... 111.35 |==================================================

```


GL-vs-VK [ Vulkan versus OpenGL GPU tests ]

```
GL-vs-VK 2017-06-05
Test: Static Scene - API: OpenGL - Multi-Threaded: No
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 156.71 |==========================================
gl-vs-vk-zen_-0 ..... 171.59 |==============================================

GL-vs-VK 2017-06-05
Test: Static Scene - API: OpenGL - Multi-Threaded: No
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 6.381339 |============================================
gl-vs-vk-zen_-0 ..... 5.827955 |========================================


GL-vs-VK 2017-06-05
Test: Static Scene - API: Vulkan - Multi-Threaded: No
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 108.84 |=============================================
gl-vs-vk-zen_-0 ..... 112.11 |==============================================


GL-vs-VK 2017-06-05
Test: Static Scene - API: Vulkan - Multi-Threaded: No
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 9.187918 |============================================
gl-vs-vk-zen_-0 ..... 8.919964 |===========================================


GL-vs-VK 2017-06-05
Test: Static Scene - API: OpenGL - Multi-Threaded: Yes
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 102.76 |============================================
gl-vs-vk-zen_-0 ..... 107.34 |==============================================


GL-vs-VK 2017-06-05
Test: Static Scene - API: OpenGL - Multi-Threaded: Yes
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 9.731480 |============================================
gl-vs-vk-zen_-0 ..... 9.316619 |==========================================


GL-vs-VK 2017-06-05
Test: Static Scene - API: Vulkan - Multi-Threaded: Yes
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 40.47 |==============================================
gl-vs-vk-zen_-0 ..... 41.63 |===============================================


GL-vs-VK 2017-06-05
Test: Static Scene - API: Vulkan - Multi-Threaded: Yes
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 24.71 |===============================================
gl-vs-vk-zen_-0 ..... 24.02 |=============================================



GL-vs-VK 2017-06-05
Test: Shadow Mapping - API: OpenGL - Multi-Threaded: No
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 6.757658 |====================================
gl-vs-vk-zen_-0 ..... 8.352796 |============================================


GL-vs-VK 2017-06-05
Test: Shadow Mapping - API: OpenGL - Multi-Threaded: No
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 147.98 |==============================================
gl-vs-vk-zen_-0 ..... 119.72 |=====================================


GL-vs-VK 2017-06-05
Test: Shadow Mapping - API: Vulkan - Multi-Threaded: No
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 4.987357 |===========================================
gl-vs-vk-zen_-0 ..... 5.029075 |============================================


GL-vs-VK 2017-06-05
Test: Shadow Mapping - API: Vulkan - Multi-Threaded: No
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 200.51 |==============================================
gl-vs-vk-zen_-0 ..... 197.91 |=============================================



GL-vs-VK 2017-06-05
Test: Shadow Mapping - API: Vulkan - Multi-Threaded: Yes
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 4.797430 |============================================
gl-vs-vk-zen_-0 ..... 4.820551 |============================================



GL-vs-VK 2017-06-05
Test: Shadow Mapping - API: Vulkan - Multi-Threaded: Yes
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 208.45 |==============================================
gl-vs-vk-zen_-0 ..... 207.45 |==============================================


GL-vs-VK 2017-06-05
Test: Terrain With Dynamic LoD - API: OpenGL - Multi-Threaded: No
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 23.71 |=====================================
gl-vs-vk-zen_-0 ..... 29.72 |===============================================


GL-vs-VK 2017-06-05
Test: Terrain With Dynamic LoD - API: OpenGL - Multi-Threaded: No
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 42.17 |===============================================
gl-vs-vk-zen_-0 ..... 33.64 |=====================================


GL-vs-VK 2017-06-05
Test: Terrain With Dynamic LoD - API: Vulkan - Multi-Threaded: No
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 28.40 |============================================
gl-vs-vk-zen_-0 ..... 30.33 |===============================================


GL-vs-VK 2017-06-05
Test: Terrain With Dynamic LoD - API: Vulkan - Multi-Threaded: No
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 35.21 |===============================================
gl-vs-vk-zen_-0 ..... 32.97 |============================================


GL-vs-VK 2017-06-05
Test: Terrain With Dynamic LoD - API: Vulkan - Multi-Threaded: Yes
Frame Time - ms < Lower Is Better
gl-vs-vk-zen-LTO_-0 . 20.50 |==============================================
gl-vs-vk-zen_-0 ..... 21.04 |===============================================


GL-vs-VK 2017-06-05
Test: Terrain With Dynamic LoD - API: Vulkan - Multi-Threaded: Yes
FPS > Higher Is Better
gl-vs-vk-zen-LTO_-0 . 48.79 |===============================================
gl-vs-vk-zen_-0 ..... 47.57 |==============================================

```

I don't know what's going on with a single threaded OpenGL but I could tell the difference even without that bench... The results for a non-Zen kernel are even worse

barolo picture barolo on 11 Aug 2020
👍2

run make mrproper and move the config back. After rebuilding the kernel a new Module.symvers will be generated

Thanks! I believe that's what fixed it. Currently booted now with no issues.

telans picture telans on 11 Aug 2020
👍1

Test parameters:
GCC 10.2.0
binutils 2.35
glibc 2.30-r9
Ran in tty
CONFIG_GENERIC_CPU=y (-mtune=generic, -march not set)
LTO patch, genpatches 1000-4567 (none of which affect performance)

Notes about testing:
Phoronix Test Suite v9.8.0 (Nesodden)
pts/stress-ng requires sys-libs/libapparmor to link -lapparmor for tests
7-8% deviation on CPU Cache for both results, removed in later testing

Kernel 5.8.0, both -O2
Baseline: no-lto
Result: lto

Size comparisons, using lz4 compression
no-LTO -O2: 15939744
LTO -O2: 15464160
~3% size decrease


Results to baseline

| Test | Configuration | Relative |
| --------- | --------------- | -------- |
| hackbench | 1 - Thread | 1.032 |
| hackbench | 2 - Thread | 1.051 |
| hackbench | 1 - Process | 1.029 |
| hackbench | 16 - Thread | 1.052 |
| hackbench | 2 - Process | 1.042 |
| hackbench | 16 - Process | 1.024 |
| stress-ng | Crypto | 0.998 |
| stress-ng | Malloc | 0.997 |
| stress-ng | Forking | 1.028 |
| stress-ng | CPU Cache | 1.024 |
| stress-ng | Vector Math | 0.998 |
| stress-ng | Memory Copying | 1.002 |
| stress-ng | Socket Activity | 1.016 |



Details for -O2 vs. -O2 LTO

    Count: 1 - Type: Thread
    Seconds < Lower Is Better
    lto o2 generic .... 4.569 |=====================================================================================================================================================================================================================================================================================
    no lto o2 generic . 4.717 |==============================================================================================================================================================================================================================================================================================


    Hackbench
    Count: 2 - Type: Thread
    Seconds < Lower Is Better
    lto o2 generic .... 6.047 |================================================================================================================================================================================================================================================================================
    no lto o2 generic . 6.357 |==============================================================================================================================================================================================================================================================================================


    Hackbench
    Count: 1 - Type: Process
    Seconds < Lower Is Better
    lto o2 generic .... 4.352 |======================================================================================================================================================================================================================================================================================
    no lto o2 generic . 4.477 |==============================================================================================================================================================================================================================================================================================


    Hackbench
    Count: 16 - Type: Thread
    Seconds < Lower Is Better
    lto o2 generic .... 43.48 |================================================================================================================================================================================================================================================================================
    no lto o2 generic . 45.73 |==============================================================================================================================================================================================================================================================================================


    Hackbench
    Count: 2 - Type: Process
    Seconds < Lower Is Better
    lto o2 generic .... 5.741 |==================================================================================================================================================================================================================================================================================
    no lto o2 generic . 5.982 |==============================================================================================================================================================================================================================================================================================


    Hackbench
    Count: 16 - Type: Process
    Seconds < Lower Is Better
    lto o2 generic .... 42.57 |=======================================================================================================================================================================================================================================================================================
    no lto o2 generic . 43.59 |==============================================================================================================================================================================================================================================================================================


    Stress-NG 0.11.07
    Test: Crypto
    Bogo Ops/s > Higher Is Better
    lto o2 generic .... 1574.10 |============================================================================================================================================================================================================================================================================================
    no lto o2 generic . 1576.77 |============================================================================================================================================================================================================================================================================================


    Stress-NG 0.11.07
    Test: Malloc
    Bogo Ops/s > Higher Is Better
    lto o2 generic .... 58909034.94 |=======================================================================================================================================================================================================================================================================================
    no lto o2 generic . 59083725.81 |========================================================================================================================================================================================================================================================================================


    Stress-NG 0.11.07
    Test: Forking
    Bogo Ops/s > Higher Is Better
    lto o2 generic .... 28953.09 |===========================================================================================================================================================================================================================================================================================
    no lto o2 generic . 28167.74 |===================================================================================================================================================================================================================================================================================


    Stress-NG 0.11.07
    Test: CPU Cache
    Bogo Ops/s > Higher Is Better
    lto o2 generic .... 28.55 |==============================================================================================================================================================================================================================================================================================
    no lto o2 generic . 27.87 |=======================================================================================================================================================================================================================================================================================


    Stress-NG 0.11.07
    Test: Vector Math
    Bogo Ops/s > Higher Is Better
    lto o2 generic .... 53981.76 |==========================================================================================================================================================================================================================================================================================
    no lto o2 generic . 54099.63 |===========================================================================================================================================================================================================================================================================================


    Stress-NG 0.11.07
    Test: Memory Copying
    Bogo Ops/s > Higher Is Better
    lto o2 generic .... 9299.38 |============================================================================================================================================================================================================================================================================================
    no lto o2 generic . 9282.92 |===========================================================================================================================================================================================================================================================================================


    Stress-NG 0.11.07
    Test: Socket Activity
    Bogo Ops/s > Higher Is Better
    lto o2 generic .... 7781.26 |============================================================================================================================================================================================================================================================================================
    no lto o2 generic . 7656.45 |=======================================================================================================================================================================================================================================================================================

Summary: clear performance benefit to using an LTO kernel

Kernel 5.8.0, both LTO
Baseline: LTO -O2
Result: LTO -O3 -fno-inline-functions

Note: ~3% deviation on memory copying, unsure why


Results to baseline

| Test | Configuration | Relative |
| --------- | --------------- | -------- |
| hackbench | 1 - Thread | 0.984 |
| hackbench | 2 - Thread | 0.976 |
| hackbench | 1 - Process | 0.98 |
| hackbench | 16 - Thread | 0.969 |
| hackbench | 2 - Process | 0.967 |
| hackbench | 16 - Process | 0.976 |
| stress-ng | Crypto | 0.999 |
| stress-ng | Malloc | 1.02 |
| stress-ng | Forking | 0.97 |
| stress-ng | Vector Math | 1 |
| stress-ng | Memory Copying | 1.045 |
| stress-ng | Socket Activity | 0.981 |


Results to baseline: no-lto -O2

| Test | Configuration | Relative |
| --------- | --------------- | -------- |
| hackbench | 1 - Thread | 1.016 |
| hackbench | 2 - Thread | 1.026 |
| hackbench | 1 - Process | 1.008 |
| hackbench | 16 - Thread | 1.019 |
| hackbench | 2 - Process | 1.008 |
| hackbench | 16 - Process | 0.999 |
| stress-ng | Crypto | 0.997 |
| stress-ng | Malloc | 1.017 |
| stress-ng | Forking | 0.997 |
| stress-ng | Vector Math | 0.997 |
| stress-ng | Memory Copying | 1.047 |
| stress-ng | Socket Activity | 0.997 |


Details for LTO -O2 vs. LTO -O3 -fno-inline-functions

    Count: 1 - Type: Thread
    Seconds < Lower Is Better
    lto o2 generic ............ 4.569 |===========================================================
    O3LTO-fnoinline-functions . 4.645 |============================================================


    Hackbench
    Count: 2 - Type: Thread
    Seconds < Lower Is Better
    lto o2 generic ............ 6.047 |===========================================================
    O3LTO-fnoinline-functions . 6.197 |============================================================


    Hackbench
    Count: 1 - Type: Process
    Seconds < Lower Is Better
    lto o2 generic ............ 4.352 |===========================================================
    O3LTO-fnoinline-functions . 4.443 |============================================================


    Hackbench
    Count: 16 - Type: Thread
    Seconds < Lower Is Better
    lto o2 generic ............ 43.48 |==========================================================
    O3LTO-fnoinline-functions . 44.89 |============================================================


    Hackbench
    Count: 2 - Type: Process
    Seconds < Lower Is Better
    lto o2 generic ............ 5.741 |==========================================================
    O3LTO-fnoinline-functions . 5.934 |============================================================


    Hackbench
    Count: 16 - Type: Process
    Seconds < Lower Is Better
    lto o2 generic ............ 42.57 |===========================================================
    O3LTO-fnoinline-functions . 43.63 |============================================================


    Stress-NG 0.11.07
    Test: Crypto
    Bogo Ops/s > Higher Is Better
    lto o2 generic ............ 1574.10 |==========================================================
    O3LTO-fnoinline-functions . 1572.03 |==========================================================


    Stress-NG 0.11.07
    Test: Malloc
    Bogo Ops/s > Higher Is Better
    lto o2 generic ............ 58909034.94 |=====================================================
    O3LTO-fnoinline-functions . 60113960.06 |======================================================


    Stress-NG 0.11.07
    Test: Forking
    Bogo Ops/s > Higher Is Better
    lto o2 generic ............ 28953.09 |=========================================================
    O3LTO-fnoinline-functions . 28078.41 |=======================================================


    Stress-NG 0.11.07
    Test: Vector Math
    Bogo Ops/s > Higher Is Better
    lto o2 generic ............ 53981.76 |=========================================================
    O3LTO-fnoinline-functions . 53960.10 |=========================================================


    Stress-NG 0.11.07
    Test: Memory Copying
    Bogo Ops/s > Higher Is Better
    lto o2 generic ............ 9299.38 |========================================================
    O3LTO-fnoinline-functions . 9716.74 |==========================================================


    Stress-NG 0.11.07
    Test: Socket Activity
    Bogo Ops/s > Higher Is Better
    lto o2 generic ............ 7781.26 |==========================================================
    O3LTO-fnoinline-functions . 7632.56 |=========================================================


Details for -O2, LTO -O2, and LTO -O3 -fno-inline-functions

    Count: 1 - Type: Thread
    Seconds < Lower Is Better
    no lto o2 generic ......... 4.717 |============================================
    lto o2 generic ............ 4.569 |===========================================
    O3LTO-fnoinline-functions . 4.645 |===========================================


    Hackbench
    Count: 2 - Type: Thread
    Seconds < Lower Is Better
    no lto o2 generic ......... 6.357 |============================================
    lto o2 generic ............ 6.047 |==========================================
    O3LTO-fnoinline-functions . 6.197 |===========================================


    Hackbench
    Count: 1 - Type: Process
    Seconds < Lower Is Better
    no lto o2 generic ......... 4.477 |============================================
    lto o2 generic ............ 4.352 |===========================================
    O3LTO-fnoinline-functions . 4.443 |============================================


    Hackbench
    Count: 16 - Type: Thread
    Seconds < Lower Is Better
    no lto o2 generic ......... 45.73 |============================================
    lto o2 generic ............ 43.48 |==========================================
    O3LTO-fnoinline-functions . 44.89 |===========================================


    Hackbench
    Count: 2 - Type: Process
    Seconds < Lower Is Better
    no lto o2 generic ......... 5.982 |============================================
    lto o2 generic ............ 5.741 |==========================================
    O3LTO-fnoinline-functions . 5.934 |============================================


    Hackbench
    Count: 16 - Type: Process
    Seconds < Lower Is Better
    no lto o2 generic ......... 43.59 |============================================
    lto o2 generic ............ 42.57 |===========================================
    O3LTO-fnoinline-functions . 43.63 |============================================


    Stress-NG 0.11.07
    Test: Crypto
    Bogo Ops/s > Higher Is Better
    no lto o2 generic ......... 1576.77 |==========================================
    lto o2 generic ............ 1574.10 |==========================================
    O3LTO-fnoinline-functions . 1572.03 |==========================================


    Stress-NG 0.11.07
    Test: Malloc
    Bogo Ops/s > Higher Is Better
    no lto o2 generic ......... 59083725.81 |=====================================
    lto o2 generic ............ 58909034.94 |=====================================
    O3LTO-fnoinline-functions . 60113960.06 |======================================


    Stress-NG 0.11.07
    Test: Forking
    Bogo Ops/s > Higher Is Better
    no lto o2 generic ......... 28167.74 |========================================
    lto o2 generic ............ 28953.09 |=========================================
    O3LTO-fnoinline-functions . 28078.41 |========================================


    Stress-NG 0.11.07
    Test: Vector Math
    Bogo Ops/s > Higher Is Better
    no lto o2 generic ......... 54099.63 |=========================================
    lto o2 generic ............ 53981.76 |=========================================
    O3LTO-fnoinline-functions . 53960.10 |=========================================


    Stress-NG 0.11.07
    Test: Memory Copying
    Bogo Ops/s > Higher Is Better
    no lto o2 generic ......... 9282.92 |========================================
    lto o2 generic ............ 9299.38 |========================================
    O3LTO-fnoinline-functions . 9716.74 |==========================================


    Stress-NG 0.11.07
    Test: Socket Activity
    Bogo Ops/s > Higher Is Better
    no lto o2 generic ......... 7656.45 |=========================================
    lto o2 generic ............ 7781.26 |==========================================
    O3LTO-fnoinline-functions . 7632.56 |=========================================

Summary: LTO -O3 with -fno-inline-functions performs worse against LTO -O2, and performs better or the same as no-LTO -O2

Kernel 5.8.0, both LTO
Baseline: LTO -O2
Result: LTO -O3

Size comparisons, using lz4 compression
no-LTO -O2: 15939744
LTO -O2: 15464160
LTO -O3: 16665088
~4.4% size increase against no-LTO -O2, ~7.4% increase compared against LTO -O2


Results to baseline

| Test | Configuration | Relative |
| --------- | --------------- | -------- |
| hackbench | 1 - Thread | 0.99 |
| hackbench | 2 - Thread | 1.022 |
| hackbench | 1 - Process | 0.989 |
| hackbench | 16 - Thread | 0.988 |
| hackbench | 2 - Process | 1.01 |
| hackbench | 16 - Process | 1.003 |
| stress-ng | Crypto | 0.998 |
| stress-ng | Malloc | 1.017 |
| stress-ng | Forking | 0.98 |
| stress-ng | Vector Math | 0.996 |
| stress-ng | Memory Copying | 1.005 |
| stress-ng | Socket Activity | 1.009 |


Results to baseline: LTO -O3 -fno-inline-functions

| Test | Configuration | Relative |
| --------- | --------------- | -------- |
| hackbench | 1 - Thread | 1.007 |
| hackbench | 2 - Thread | 1.047 |
| hackbench | 1 - Process | 1.01 |
| hackbench | 16 - Thread | 1.02 |
| hackbench | 2 - Process | 1.044 |
| hackbench | 16 - Process | 1.028 |
| stress-ng | Crypto | 0.999 |
| stress-ng | Malloc | 0.996 |
| stress-ng | Forking | 1.01 |
| stress-ng | Vector Math | 0.997 |
| stress-ng | Memory Copying | 0.962 |
| stress-ng | Socket Activity | 1.028 |


Details for LTO -O2 vs. LTO -O3

    Count: 1 - Type: Thread
    Seconds < Lower Is Better
    O3lto-inline-functions . 4.614 |=============================================================
    lto o2 generic ......... 4.569 |============================================================


    Hackbench
    Count: 2 - Type: Thread
    Seconds < Lower Is Better
    O3lto-inline-functions . 5.918 |============================================================
    lto o2 generic ......... 6.047 |=============================================================


    Hackbench
    Count: 1 - Type: Process
    Seconds < Lower Is Better
    O3lto-inline-functions . 4.399 |=============================================================
    lto o2 generic ......... 4.352 |============================================================


    Hackbench
    Count: 16 - Type: Thread
    Seconds < Lower Is Better
    O3lto-inline-functions . 44.00 |=============================================================
    lto o2 generic ......... 43.48 |============================================================


    Hackbench
    Count: 2 - Type: Process
    Seconds < Lower Is Better
    O3lto-inline-functions . 5.685 |============================================================
    lto o2 generic ......... 5.741 |=============================================================


    Hackbench
    Count: 16 - Type: Process
    Seconds < Lower Is Better
    O3lto-inline-functions . 42.42 |=============================================================
    lto o2 generic ......... 42.57 |=============================================================


    Stress-NG 0.11.07
    Test: Crypto
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions . 1570.90 |===========================================================
    lto o2 generic ......... 1574.10 |===========================================================


    Stress-NG 0.11.07
    Test: Malloc
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions . 59900927.06 |=======================================================
    lto o2 generic ......... 58909034.94 |======================================================


    Stress-NG 0.11.07
    Test: Forking
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions . 28364.34 |=========================================================
    lto o2 generic ......... 28953.09 |==========================================================


    Stress-NG 0.11.07
    Test: Vector Math
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions . 53790.84 |==========================================================
    lto o2 generic ......... 53981.76 |==========================================================


    Stress-NG 0.11.07
    Test: Memory Copying
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions . 9342.95 |===========================================================
    lto o2 generic ......... 9299.38 |===========================================================


    Stress-NG 0.11.07
    Test: Socket Activity
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions . 7849.58 |===========================================================
    lto o2 generic ......... 7781.26 |==========================================================


Details for no-LTO -O2, LTO -O2, LTO -O3 -fno-inline-functions, LTO -O3

    Count: 1 - Type: Thread
    Seconds < Lower Is Better
    O3lto-inline-functions .... 4.614 |=========================================================
    lto o2 generic ............ 4.569 |========================================================
    no lto o2 generic ......... 4.717 |==========================================================
    O3LTO-fnoinline-functions . 4.645 |=========================================================


    Hackbench
    Count: 2 - Type: Thread
    Seconds < Lower Is Better
    O3lto-inline-functions .... 5.918 |======================================================
    lto o2 generic ............ 6.047 |=======================================================
    no lto o2 generic ......... 6.357 |==========================================================
    O3LTO-fnoinline-functions . 6.197 |=========================================================


    Hackbench
    Count: 1 - Type: Process
    Seconds < Lower Is Better
    O3lto-inline-functions .... 4.399 |=========================================================
    lto o2 generic ............ 4.352 |========================================================
    no lto o2 generic ......... 4.477 |==========================================================
    O3LTO-fnoinline-functions . 4.443 |==========================================================


    Hackbench
    Count: 16 - Type: Thread
    Seconds < Lower Is Better
    O3lto-inline-functions .... 44.00 |========================================================
    lto o2 generic ............ 43.48 |=======================================================
    no lto o2 generic ......... 45.73 |==========================================================
    O3LTO-fnoinline-functions . 44.89 |=========================================================


    Hackbench
    Count: 2 - Type: Process
    Seconds < Lower Is Better
    O3lto-inline-functions .... 5.685 |=======================================================
    lto o2 generic ............ 5.741 |========================================================
    no lto o2 generic ......... 5.982 |==========================================================
    O3LTO-fnoinline-functions . 5.934 |==========================================================


    Hackbench
    Count: 16 - Type: Process
    Seconds < Lower Is Better
    O3lto-inline-functions .... 42.42 |========================================================
    lto o2 generic ............ 42.57 |=========================================================
    no lto o2 generic ......... 43.59 |==========================================================
    O3LTO-fnoinline-functions . 43.63 |==========================================================


    Stress-NG 0.11.07
    Test: Crypto
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions .... 1570.90 |========================================================
    lto o2 generic ............ 1574.10 |========================================================
    no lto o2 generic ......... 1576.77 |========================================================
    O3LTO-fnoinline-functions . 1572.03 |========================================================


    Stress-NG 0.11.07
    Test: Malloc
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions .... 59900927.06 |====================================================
    lto o2 generic ............ 58909034.94 |===================================================
    no lto o2 generic ......... 59083725.81 |===================================================
    O3LTO-fnoinline-functions . 60113960.06 |====================================================


    Stress-NG 0.11.07
    Test: Forking
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions .... 28364.34 |======================================================
    lto o2 generic ............ 28953.09 |=======================================================
    no lto o2 generic ......... 28167.74 |======================================================
    O3LTO-fnoinline-functions . 28078.41 |=====================================================


    Stress-NG 0.11.07
    Test: Vector Math
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions .... 53790.84 |=======================================================
    lto o2 generic ............ 53981.76 |=======================================================
    no lto o2 generic ......... 54099.63 |=======================================================
    O3LTO-fnoinline-functions . 53960.10 |=======================================================


    Stress-NG 0.11.07
    Test: Memory Copying
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions .... 9342.95 |======================================================
    lto o2 generic ............ 9299.38 |======================================================
    no lto o2 generic ......... 9282.92 |=====================================================
    O3LTO-fnoinline-functions . 9716.74 |========================================================


    Stress-NG 0.11.07
    Test: Socket Activity
    Bogo Ops/s > Higher Is Better
    O3lto-inline-functions .... 7849.58 |========================================================
    lto o2 generic ............ 7781.26 |========================================================
    no lto o2 generic ......... 7656.45 |=======================================================
    O3LTO-fnoinline-functions . 7632.56 |======================================================

Summary: fno-inline-functions has an observable impact on performance. LTO -O2 performs marginally better than LTO -O3 overall, and is smaller too.

It'd be better if I had more tests to reduce variation, also tested the effect of setting -march. Would GCC have been able to take advantage of my processor cache better? I found out recently that if you don't set -march=native, these defaults will be set: --param=l1-cache-line-size=32 & --param=l1-cache-size=64 unless user specified.

jiblime picture jiblime on 12 Aug 2020

I'll run those same benchmarks later, however, currently perf is giving me approximately 20% lower results on an lto kernel. Using KCFLAGS=-fno-inline-functions -fno-inline-functions-called-once --param=large-stack-frame-growth=100 -fdevirtualize-at-ltrans -fgraphite-identity -floop-nest-optimize but I'll try with default flags.

telans picture telans on 12 Aug 2020

After some rebuilding, I've found -fno-inline-functions is enough to stop the warnings when using -O3. Still need to test if -O3 is faster, with the inlining flags I mentioned LTO -O3 is actually slower than LTO -O2 😞

jiblime picture jiblime on 12 Aug 2020

Successful building/booting summary:

  • you need a GCC without -fPIE
  • MODVERSIONS and MODULE_SRCVERSION_ALL set to NO in kernel config if you want module loading
  • plus KALLSYMS_ALL set to YES and also do make mrproper to have a fresh Module.symvers
  • CRYPTO_DEV_CCP_DD [ an amd thing ] WILL crash your kernel if built in
  • SINGLE_LINK and KALLSYMS_SINGLE set to OFFor build will fail

[ that's what's working for me ]

@jiblime got everything?

barolo picture barolo on 12 Aug 2020
👍3

@barolo great idea to have guidelines on how to have a successful build. For all of the options, IMO more testing is needed. A lot of the suggestions I made were on the fly so to speak; I could be wrong and I'd feel bad if I led someone astray.

you need a GCC without -fPIE

I can agree with this. I'm interested in knowing if a simple -fno-PIE would suffice. I'll rebuild GCC:10.2.0 and will update.

MODVERSIONS and MODULE_SRCVERSION_ALL set to NO in kernel config if you want module loading

This seems to be important because some modules do not support MODVERSIONS. I figured some modules wouldn't load if the modversion couldn't be identified, but that issue may have arised from a broken Module.symvers. Although...does this work for anyone? I've never been able to load two different kernels and use the same /lib/modules directory for modules.

plus KALLSYMS_ALL set to YES and also do make mrproper to have a fresh Module.symvers

CONFIG_KALLSYMS_ALL=y only if your modules are lacking symbols required on installation/loading. make mrproper has been working for me, but it's possbile make clean && rm Module.symvers would be enough. It would definitely save the trouble of moving the .config in and out of the build directory.

CRYPTO_DEV_CCP_DD [ an amd thing ] WILL crash your kernel if built in

So, tying back to the Module.symvers missing symbols again...that might've been the cause. I am once again able to load module ccp. However, loaded or not it is useless since the current implementation in Linux is purposefully broken (and by extension module ccp_crypto), and maybe even for good reason?...This part of the summary can be excluded.

SINGLE_LINK and KALLSYMS_SINGLE set to OFF or build will fail

Ideally let users know the config options may fail but recommend they try it so we have a wider group of testers before declaring it unusable. More info about those options can be found in ./scripts/link-vmlinux.sh.

For SINGLE_LINK, try building the kernel with it on, and if it fails, use the the proposed workaround, KALLSYMS_EXTRA_PASS=1, and if that fails then disable it. The same goes for KALLSYMS_SINGLE.

Also recommended:
GCC version slot >=10
binutils version slot >=2.34

Using -fno-inline-functions with -O3, or stick with -O2

Besides that, your summary is a good way to get things going with no issues. With more time, I'd like to figure out why a particular config is needed, and if the solution proposed was actually correct. Did you encounter any particular quirks regarding external modules or generating an initrd? I understand that not many people use dracut on Gentoo.

jiblime picture jiblime on 12 Aug 2020
👍1

I'm interested in knowing if a simple -fno-PIE would suffice. I'll rebuild GCC:10.2.0 and will update.

@jiblime i tried that on KCFLAGS as @telans stated here and it does not work w/ my gcc 10.2.0 built with pie support. had to rebuild w/o pie.

mssx86 picture mssx86 on 12 Aug 2020
👍1

@jiblime is -fno-inline-functions really necessary? The only one seemingly unstable thing on my sys seems to be bluetooth segfaulting from time to time

barolo picture barolo on 12 Aug 2020

@barolo Undefined behavior in a unique kernel build could lead to system instability and it'd be hard to diagnose. Still I think understand what you mean; it is your system after all. Maybe it could be considered a safe recommendation instead?

It would be good to benchmark -fno-inline-functions either way

Edit:
Unconfirmed bug affecting GCC 10.2.0/11: False positive -Wstringop-overflow with -O3

jiblime picture jiblime on 12 Aug 2020

5.8.1-gentoo + lto-5.8-1.patch compiles fine for me (my regular config + minimum patching).

I have default pie enabled in GCC, so I disabled it like this: gcc -fno-pie -no-pie. Also I have CONFIG_TRIM_UNUSED_KSYMS=y, so I have to trick lto to leave special symbols alone.
Everything else the same as with my previous kernel config, no monolithic kernel, no extra optimizations this time.

--- a/Makefile
+++ b/Makefile
@@ -2,7 +2,7 @@
 VERSION = 5
 PATCHLEVEL = 8
 SUBLEVEL = 1
-EXTRAVERSION = -gentoo
+EXTRAVERSION = -gentoo-lto
 NAME = Kleptomaniac Octopus

 # *DOCUMENTATION*
@@ -437,7 +437,7 @@
 OBJSIZE                = llvm-size
 STRIP          = llvm-strip
 else
-CC             = $(CROSS_COMPILE)gcc
+CC             = $(CROSS_COMPILE)gcc -fno-pie -no-pie
 LD             = $(CROSS_COMPILE)ld
 LDFINAL                = $(LD)
 AR             = $(CROSS_COMPILE)ar
--- a/include/linux/export.h
+++ a/include/linux/export.h
@@ -111,7 +111,7 @@
  * discarded in the final link stage.
  */
 #define __ksym_marker(sym)     \
-       static int __ksym_marker_##sym[0] __section(".discard.ksym") __used
+       int __ksym_marker_##sym[0] __section(".discard.ksym") __used

 #define __EXPORT_SYMBOL(sym, sec, ns)                                  \
        __ksym_marker(sym);                                             \

Note:

  • didn't checked with pie, if default kernel compiles fine with pie then lto-ed should too, but I will trust you;
  • SINGLE_LINK and KALLSYMS_SINGLE both enabled or both disabled works for me, but being enabled they break TRIM_UNUSED_KSYMS and resulting files even bigger than w/o LTO;

Currently under successfully booted lto-ed kernel, no noticeable changes or breakages so far. Resulting kernel file size ~10% smaller and modules ~15% smaller.

pchome picture pchome on 13 Aug 2020
👍3

I'm going wild with my kernel, currently tking it [ which is zen combined with pjrc/bmq ] then sprinkling lto on top.
I have couple of devices which are on the edge of comfortable graphical use and any, even tiniest speedups help.
Gonna bench/publish
Prjc/bmq seem to be really bad for graphical performance I see differences of ~15fps in OpenGL games, which matter a lot for my particular use case

barolo picture barolo on 13 Aug 2020

currently tking it

Give the PDS patch a go. I believe that while TkG is currently maintaining it the prjc maintainer is in the process of updating it. It's what I use as default (for the past year or so). I've never had any issues with desktop performance, and it usually trumps cfs' graphics performance

telans picture telans on 13 Aug 2020

@telans I'm not sure what is pure Zen doing, or using by default, I'm in the process of checking that but tkg/prjcs straight brings FPS down by 15+ in OpenGL. I'll check pds later

barolo picture barolo on 13 Aug 2020

@telans Yup, PDS gives me the highest max fps overall, gonna stay on that. Thanks!
Ended with ten patches on top of LTO kernel
Edit. I take that back, pure Zen is getting me 3-4 fps more, and lower frame time, which is exactly what I need to hit 60fps in some stuff on my puny APU, gonna investigate what's the difference

barolo picture barolo on 14 Aug 2020

Regarding GCC USE=pie automatically making all compiles PIE, what does the static-pie USE flag in sys-libs/glibc-2.32 do? Would it fix that issue?

Version 2.27

Major new features:

* The GNU C Library can now be compiled with support for building static
  PIE executables (See --enable-static-pie in INSTALL).  These static PIE
  executables are like static executables but can be loaded at any address
  and provide additional security hardening benefits at the cost of some
  memory and performance.  When the library is built with --enable-static-pie
  the resulting libc.a is usable with GCC 8 and above to create static PIE
  executables using the GCC option '-static-pie'.  This feature is currently
  supported on i386, x86_64 and x32 with binutils 2.29 or later, and on
  aarch64 with binutils 2.30 or later.

Two things I've noted:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81498

Andi Kleen uses -flinker-output=nolto-rel while there are other options listed here: https://gcc.gnu.org/onlinedocs/gcc-9.2.0/gcc/Link-Options.html#Link-Options

Finally ‘nolto-rel’ configure compiler to for incremental linking where code generation is forced, final binary is produced and the intermediate code for later link-time optimization is stripped. When multiple object files are linked together the resulting code will be optimized better than with link time optimizations disabled (for example, the cross-module inlining will happen), most of benefits of whole program optimizations are however lost.

What effect would the new glibc USE flag have in tandem with PIE compilation?

Also, @pchome how were you able to figure removing static int from __ksym_marker(sym) would fix it?

jiblime picture jiblime on 15 Aug 2020

@jiblime

Also, @pchome how were you able to figure removing static int from __ksym_marker(sym) would fix it?

First of all this is TRIM_UNUSED_KSYMS only part, so only those with CONFIG_TRIM_UNUSED_KSYMS=y may want this fix.

Also, I was only interested in quick and dirty lto-ed build, so I didn't pay much attention to correctness, didn't expected a working kernel either. Be warned, don't treat this hack as correct solution.

I got tons of errors about missing symbols, compared random module files with the previous kernel build, found the macros and tried several solutions to keep those symbols from being optimized out. It looks like those symbols are temporary and should be removed during final link step, so I didn't bothered to make them correct but to make them present. Also I ignored license symbols, which are present in *_sym.o files in the regular build, but they seem not involved in trim process, at least modinfo shows correct licenses for module in lto-ed kernel.

pchome picture pchome on 15 Aug 2020
👍2

@jiblime I've tried -flinker-output=rel instead [ which is LTO's default isn't it? ] and it works just fine, it took longer to compile.
I have all modules built in though since I couldn't really make it work otherwise, even with pchome patch [ also genkernel fails with bunch of missing symbols at modules install ], they would error out after waking from suspend for example.

From the bunch of kernels that I've tested lto-ed the fastest is xanmod right after pure zen, both 5.8, they're all using default albeit tweaked CFS scheduler, 5.7 and all other schedulers are slower atm, at least when latency and raw gpu power is concerned.
It's really really smooth and snappy

Edit. Its truly, amazingly, snappy. I'm done with experimenting for now, when it's too good you better stop :)

barolo picture barolo on 17 Aug 2020
👍1

For those interested, it still works on 5.9

https://gist.github.com/jiblime/ee9f22f4073fbd8ee8add4ffd78abe70

Edit 2: I use this janky patch right after applying the lto patch so 5.9.x patches apply cleanly: https://gist.github.com/jiblime/ee9f22f4073fbd8ee8add4ffd78abe70#file-revert-cec_c-changes-diff

Using make clean instead of make mrproper should still be fine, I had an errant patch causing issues.

Here's a fun config introduced in 5.9: DEBUG_FORCE_FUNCTION_ALIGN_32B

There are cases that a commit from one domain changes the function                                                                                                                                                                                                                                                  │  
  │ address alignment of other domains, and cause magic performance                                                                                                                                                                                                                                                     │  
  │ bump (regression or improvement). Enable this option will help to                                                                                                                                                                                                                                                   │  
  │ verify if the bump is caused by function alignment changes, while                                                                                                                                                                                                                                                   │  
  │ it will slightly increase the kernel size and affect icache usage.                                                                                                                                                                                                                                                  │  
  │                                                                                                                                                                                                                                                                                                                     │  
  │ It is mainly for debug and performance tuning use.

https://github.com/torvalds/linux/blob/v5.9/Makefile#L891

ifdef CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_32B
KBUILD_CFLAGS += -falign-functions=32
endif

Definitely a plus for Intel, unsure if it would be beneficial for newer AMD Zen 1/+/2/3/4/etc.

Unless I am misinterpreting Agner's analysis, it appears Zen models would still have best cache hits with -falign-functions=16 (or lower?)
Source: https://www.agner.org/optimize/microarchitecture.pdf

20.2 Instruction fetch
The instruction fetcher can fetch 32 aligned bytes of code per clock cycle from the level-1 code cache, _according to AMD documents_, but the maximum measured throughput is only 16 bytes per clockfor a single thread. The bottleneck is probably an "instruction byte queue" between the fetch unit and the decoder.

The maximum measured fetch rate for two threads per core is 12 bytes per clock per thread. This maximum is obtained when all instructions are 8 bytes long. The fetch rate is lower for other instruction sizes.

jiblime picture jiblime on 1 Nov 2020
👍2 🎉1

Oh neat, saw 5.9 wasn't there but didn't try 5.8. Cheers

telans picture telans on 1 Nov 2020
👍1

Don't know why, but I am unable to boot if I don't use an alternative scheduler, MuQSS or PDS has worked. Does anyone else have the same experience?

jiblime picture jiblime on 20 Nov 2020

i've managed to port the 5.8-1 to 5.9 and it builds fine so far (5.9.0 -> 5.9.12) but the changes made to 5.10 rc's are beyond my skills to make 5.8-1 apply to them cleanly. is anybody up to porting it to 5.10 rc releases given how it is going to be released the following weekend?

here's my patch that applies to 5.9.12 and builds, with the lto compression part taken from @jiblime's gist: 0001-lto-5.9-mss-v3.patch

mssx86 picture mssx86 on 7 Dec 2020

It\s also a LTS kernel [5.10] so I'm going to wait with upgrading till the patches are fixed for it.

barolo picture barolo on 8 Dec 2020
👍2

Hello,
Have you guys tried clang LTO as well?
https://lwn.net/ml/linux-kernel/[email protected]/

mesziman picture mesziman on 23 Dec 2020

@mesziman Yes, I do run such a Clang+LTO Kernel. Make sure to use LLVM=1 and LLVM_IAS=1 with the latest patches. Also some kernel options are throwing out errors (don't strip the external symbols) and I also got two objtool errors during the build process at the end, but haven't noticed any practical issues yet. I haven't tested any benchmarks besides Company of Heroes 2 yet, it performs fine though.

ms178 picture ms178 on 23 Dec 2020

Hi @ms178 which is the latest? I have only seen v7 but that dropped x86 support.is there a newer?

mesziman picture mesziman on 23 Dec 2020

@mesziman the latest patch, v9, does not cover x86, you can do it only on arm kernels. it also takes more than twice the amount of time that gcc does when lto'ing with the end result being larger than gcc lto'd and non-lto kernels.

mssx86 picture mssx86 on 24 Dec 2020

@mesziman You can get the Kernel sources with the x86-LTO patches included from: https://github.com/samitolvanen/linux

Either download from the clang-lto branch with the green Code button on the front page or use the 5.10 sources from the release page. I've tested the first option only though.

You can also chose between full LTO and ThinLTO in the Kernel config (if you use menuconfig, remember to put LLVM=1 and LLVM_IAS=1 on the same line). ThinLTO is preferred as it uses all available cores unlike the FullLTO variant. The performance impact seems to be negligible between the two options according to what I've read. Also make sure to not have the "strip external symbols" Kernel option enabled or else you will get tons of errors. This is a known issue and they are working on a white list to make the Kernel a bit lighter.

ms178 picture ms178 on 24 Dec 2020

@mssx86

(Doesn't work)
https://gist.github.com/jiblime/328856e8b989b679d482ed2b7d63e980

I've tried to update it for 5.10, but it doesn't boot. I mostly removed hunks that were already upstreamed. Interesting note, scripts/link-vmlinux.sh might not need Andi Kleen's change to create .S files. 5.10, 5.8-lto, 5.9.

I needed to use -ffat-lto-objects in the linker flags to build. I initially tried using KCFLAGS="-ffat-lto-objects, so the final kernel would be built with -fno-lto-objects and modules would be built with -fno-fat-lto-objects -ffat-lto-objects (the last option takes precedence), but that didn't work. At the end of the build, the output complains of unknown symbols, orphan sections, missing CRC. Worth noting my GCC now has PIE on by default, so I added fno-pie -no-pie to the Makefile as per the above suggestion.

It's probably trivial to fix up why the symbols aren't being processed correctly, but I don't have the know how.

Pretty exciting to finally have a 'modern' LTS kernel. I'd like to see if I can update that old kgcov PGO patchset for 4.19 to 5.10

@ms178 does that allow building LTO with GCC?

jiblime picture jiblime on 25 Dec 2020

@jiblime No, LTO support is currently exclusive to LLVM/Clang - at least the patchset is far more usable than the ones for GCC as I haven't succeeded in building the Kernel with Andi Kleen's patchset for GCC, but I haven't tried in a long time. From what I've read the Clang folks reached out to a group at Microsoft which was working on LTO-support on the GCC side. But I haven't seen any new efforts of upstreaming the results yet.

ms178 picture ms178 on 25 Dec 2020

using sami tolvanen's v9 clang lto patch and thinlto'ing linux on 5.10.2, i get these warnings during the objtool stage:

  GEN     .version
  CHK     include/generated/compile.h
  GEN     .tmp_initcalls.lds
  LTO     vmlinux.o
  OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: wakeup_long64()+0x61: indirect jump found in RETPOLINE build
vmlinux.o: warning: objtool: .text+0x4e08a: indirect jump found in RETPOLINE build
vmlinux.o: warning: objtool: .text+0x4e0c5: indirect jump found in RETPOLINE build
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen2()+0x12: unsupported stack pointer realignment
vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen2()+0x12: unsupported stack pointer realignment
vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen2()+0x12: unsupported stack pointer realignment
vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen2()+0x12: unsupported stack pointer realignment
vmlinux.o: warning: objtool: aesni_gcm_init_avx_gen4()+0x12: unsupported stack pointer realignment
vmlinux.o: warning: objtool: aesni_gcm_enc_update_avx_gen4()+0x12: unsupported stack pointer realignment
vmlinux.o: warning: objtool: aesni_gcm_dec_update_avx_gen4()+0x12: unsupported stack pointer realignment
vmlinux.o: warning: objtool: aesni_gcm_finalize_avx_gen4()+0x12: unsupported stack pointer realignment
vmlinux.o: warning: objtool: __x86_retpoline_rdi()+0x10: return with modified stack frame
vmlinux.o: warning: objtool: __x86_retpoline_rdi()+0x0: stack state mismatch: cfa1=7+32 cfa2=7+8
vmlinux.o: warning: objtool: eb_relocate_parse_slow()+0x456: stack state mismatch: cfa1=7+128 cfa2=-1+0
vmlinux.o: warning: objtool: eb_copy_relocations()+0x1d5: stack state mismatch: cfa1=7+96 cfa2=-1+0
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
mssx86 picture mssx86 on 27 Dec 2020

@mssx86 I did get far less objtool errors, only two. But I have used Sami's clang-lto branch and disabled RETPOLINE in my config next to a lot of other things, as I try to disable all the things I don't want or need. As the v9 patchset on the LKML lacks the x86-related patches, I assume you got the x86-patches from his branch? Or do you build on ARM?

ms178 picture ms178 on 27 Dec 2020

@ms178 yes, i'm on 64 bit intel and i've ported the 26-commit patchset from his clang-lto to 5.10.2, it needed changes only in two hunks and none of them required any code change, the rest applied just fine.

mssx86 picture mssx86 on 27 Dec 2020

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=79db4d2293eba2ce6265a341bedf6caecad5eeb3
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=414eece95b98b209cef0f49cfcac108fd00b8ced

It's coming

zhuyifei1999 picture zhuyifei1999 on 4 Mar 2021

@zhuyifei1999

It's coming

that's the same patchset that got mentioned above and it still gives the same objtool warnings, and i've seen reports people mentioning not having wifi after suspend etc, hope it gets much more smoother though.

mssx86 picture mssx86 on 4 Mar 2021

with 5.12's release and clang lto's introduction i tried building a full lto'd kernel and i've seen a ~1.2mb raise in the size of my minimal kernel with no modules, it also has given me 3 objtool warnings.

checking andi kleen's linux-misc repo i saw that he has a branch for 5.12's release tree: https://github.com/andikleen/linux-misc/commits/lto-5.12-3

will report once i craft a patch and build 5.12.0 with gcc lto.

mssx86 picture mssx86 on 26 Apr 2021

you can find the unified lto-5.12-3 patchset here.

mssx86 picture mssx86 on 26 Apr 2021

i'm seeing considerably less time being taken for lto to finish this time compared to 5.7 -> 5.8 -> 5.9 patchset. i also got a single objtool warning: vmlinux.o: warning: objtool: crc_pcl()+0x87: sibling call from callable instruction with modified stack frame.

$ du vmlinuz-gcclto vmlinuz-nolto
6576    vmlinuz-gcclto
6088    vmlinuz-nolto
$ du linux-gcclto linux-nolto
30888   linux-gcclto
28660   linux-nolto
$ size linux-gcclto linux-nolto
   text    data     bss     dec     hex filename
18615468        5135928 2314288 26065684        18dbb14 linux-gcclto
16369388        5136240 2465840 23971468        16dc68c linux-nolto
mssx86 picture mssx86 on 26 Apr 2021

@mssx86 Thanks for unifying the patchset. I've just tried it with the 5.12.0-Xanmod-Sources on Ubuntu 21.04 and while it compiles with a objtool and a symbol error, I don't get it to boot. I've tried gcc 10.3. and 11.1 - even with only O2 and march=haswell set, with or without the experimental function option enabled or disabled, it simply won't boot, freezing right after loading from the Ramdisk.

Edit: @zhuyifei1999 Here is my bug report with more details but without earlyprintk/earlycon (I am also testing on bare metal): https://github.com/andikleen/linux-misc/issues/37

ms178 picture ms178 on 29 Apr 2021

Do you have earlyprintk/earlycon? I tested under QEMU/KVM with a serial earlycon, and got: (if you want to do it on a physical machine, vga and efi are the easier ones).

[    3.055405] Kernel/User page tables isolation: enabled
[    3.056280] ------------[ cut here ]------------
[    3.057087] kernel BUG at kernel/sched/core.c:8090!
[    3.057902] invalid opcode: 0000 [#1] PTI
[    3.058569] CPU: 0 PID: 0 Comm: swapper Not tainted 5.12.0 #7
[    3.059501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-20210404_191649-peregrine 04/01/2014
[    3.061202] RIP: 0010:sched_init+0x0/0x2
[    3.061867] Code: 1f 82 48 c7 c7 d5 39 b6 81 e8 82 9f 02 ff 48 85 c0 75 02 0f 0b 48 c7 c7 c0 9a e2 81 e8 b1 59 f2 fe 84 c0 75 02 0f 0b 31 c0 c3 <0f> 0b fa e8 b5 0a f2 fe fb 48 c7 c7 70 e2 1f 82 e9 24 ed f8 fe 48
[    3.065030] RSP: 0000:ffffffff81e03f18 EFLAGS: 00010082
[    3.065911] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000027
[    3.067208] RDX: 000ffffffffff000 RSI: ffffffffff600000 RDI: 0000000001e23067
[    3.068587] RBP: ffff88813fff8000 R08: ffffffff81e1cff8 R09: 0000000000002400
[    3.069812] R10: ffffea0004001240 R11: 00000000001fffff R12: 0000000000000167
[    3.070990] R13: ffffffff8219fba0 R14: 0000000000000000 R15: 0000000000000000
[    3.072320] FS:  0000000000000000(0000) GS:ffffffff81e30000(0000) knlGS:0000000000000000
[    3.073749] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.074728] CR2: ffff88813ffff000 CR3: 0000000001e1c001 CR4: 00000000000606b0
[    3.075911] Call Trace:
[    3.076304]  start_kernel+0x21f/0x3d8
[    3.076926]  secondary_startup_64_no_verify+0xc2/0xcb
[    3.077728] Modules linked in:
[    3.078213] random: get_random_bytes called from init_oops_id.lto_priv.0+0x1d/0x2c with crng_init=0
[    3.078221] ---[ end trace becf9b562df913ba ]---
[    3.080483] RIP: 0010:sched_init+0x0/0x2
[    3.081130] Code: 1f 82 48 c7 c7 d5 39 b6 81 e8 82 9f 02 ff 48 85 c0 75 02 0f 0b 48 c7 c7 c0 9a e2 81 e8 b1 59 f2 fe 84 c0 75 02 0f 0b 31 c0 c3 <0f> 0b fa e8 b5 0a f2 fe fb 48 c7 c7 70 e2 1f 82 e9 24 ed f8 fe 48
[    3.084624] RSP: 0000:ffffffff81e03f18 EFLAGS: 00010082
[    3.085501] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000027
[    3.086891] RDX: 000ffffffffff000 RSI: ffffffffff600000 RDI: 0000000001e23067
[    3.088059] RBP: ffff88813fff8000 R08: ffffffff81e1cff8 R09: 0000000000002400
[    3.089272] R10: ffffea0004001240 R11: 00000000001fffff R12: 0000000000000167
[    3.090600] R13: ffffffff8219fba0 R14: 0000000000000000 R15: 0000000000000000
[    3.092044] FS:  0000000000000000(0000) GS:ffffffff81e30000(0000) knlGS:0000000000000000
[    3.093919] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.095221] CR2: ffff88813ffff000 CR3: 0000000001e1c001 CR4: 00000000000606b0
[    3.096932] Kernel panic - not syncing: Attempted to kill the idle task!
[    3.098507] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

This code is https://elixir.bootlin.com/linux/v5.12/source/kernel/sched/core.c#L8090:

    BUG_ON(&idle_sched_class + 1 != &fair_sched_class ||
           &fair_sched_class + 1 != &rt_sched_class ||
           &rt_sched_class + 1   != &dl_sched_class);

Even though the symbols are in the correct locations (linker script applied correctly):

nolto $ nm -n -f sysv vmlinux | grep -B 1 -A 1 sched_class
.LC16               |ffffffff81b99b69|   d  |            NOTYPE|                |     |.rodata
__begin_sched_classes|ffffffff81b99ce0|   D  |            NOTYPE|                |     |.rodata
idle_sched_class    |ffffffff81b99ce0|   D  |            OBJECT|0000000000000080|     |.rodata
fair_sched_class    |ffffffff81b99d60|   D  |            OBJECT|0000000000000080|     |.rodata
rt_sched_class      |ffffffff81b99de0|   D  |            OBJECT|0000000000000080|     |.rodata
dl_sched_class      |ffffffff81b99e60|   D  |            OBJECT|0000000000000080|     |.rodata
__end_sched_classes |ffffffff81b99ee0|   D  |            NOTYPE|                |     |.rodata
__start_ro_after_init|ffffffff81b99ee0|   D  |            NOTYPE|                |     |.rodata
lto $ nm -n -f sysv vmlinux | grep -B 1 -A 1 sched_class
.LC0                |ffffffff81baf326|   d  |            NOTYPE|                |     |.rodata
__begin_sched_classes|ffffffff81bb1840|   D  |            NOTYPE|                |     |.rodata
idle_sched_class    |ffffffff81bb1840|   D  |            OBJECT|0000000000000080|     |.rodata
fair_sched_class    |ffffffff81bb18c0|   D  |            OBJECT|0000000000000080|     |.rodata
rt_sched_class      |ffffffff81bb1940|   D  |            OBJECT|0000000000000080|     |.rodata
dl_sched_class      |ffffffff81bb19c0|   D  |            OBJECT|0000000000000080|     |.rodata
__end_sched_classes |ffffffff81bb1a40|   D  |            NOTYPE|                |     |.rodata
__start_ro_after_init|ffffffff81bb1a40|   D  |            NOTYPE|                |     |.rodata

gef➤  p &idle_sched_class + 1
$1 = (const struct sched_class *) 0xffffffff81bb18c0 <fair_sched_class>
gef➤  p &fair_sched_class + 1
$2 = (const struct sched_class *) 0xffffffff81bb1940 <rt_sched_class>
gef➤  p &rt_sched_class + 1
$3 = (const struct sched_class *) 0xffffffff81bb19c0 <dl_sched_class>

The disassembly of that entire function just looks like:

gef➤  disas sched_init
Dump of assembler code for function sched_init:
=> 0xffffffff8212cec4 <+0>: ud2    
End of assembler dump.

... optimizer bug?

=sys-devel/gcc-11.1.0
=sys-devel/binutils-2.36.1-r1
zhuyifei1999 picture zhuyifei1999 on 29 Apr 2021

last time i did try lto in the kernel nvidia-drivers was not working. but now people say that it works with clang?.
i like to try it out again . is this still correct : make -j8 AR=gcc-ar NM=gcc-nm KCFLAGS="-march=skylake -O3 -falign-functions=32 -fipa-pta -fno-semantic-interposition -fgraphite-identity -floop-nest-optimize -flto=8 -ffat-lto-objects" ?

it did boot last time tried...

hedmo picture hedmo on 30 Apr 2021

@hedmo Keep in mind that 5.12 only supports LTO with Clang by default. The patch set from Andi Kleen in this thread enables GCC LTO support, but getting it to boot might depend on your luck - I couldn't get it to work. Either way, it should be enabled in the Kernel config as an option under the "general architecture features" category (make LLVM=1 LLVM_IAS=1 menuconfig), as some parts of the Kernel do not compile with it when LTO is set on the command line. I've also seen things falling apart with some Kernel options (e.g. trimming unused kernel symbols with both GCC and Clang leads to tons of modpost errors) but by and large everything does work with Clang and ThinLTO for me, even with additional Polly flags for parallelization and vectorization.

ms178 picture ms178 on 30 Apr 2021

@ms178 can you give me the commands you used?

hedmo picture hedmo on 30 Apr 2021

@hedmo -O3 -march=native -mtune=native -mllvm -polly -mllvm -polly-parallel -fopenmp -mllvm -polly-vectorizer=stripmine -mllvm -polly-omp-backend=LLVM -mllvm -polly-num-threads=24 -mllvm -polly-scheduling=dynamic -mllvm -polly-scheduling-chunksize=1 -fno-math-errno -fno-trapping-math -fno-semantic-interposition -falign-functions=32

ms178 picture ms178 on 1 May 2021

Clang ignores -fsemantic-interposition/-fno-semantic-interposition flag for gcc compatibility

perfect7gentleman picture perfect7gentleman on 1 May 2021

last commit made to the lto-5.12-3 branch have fixed the booting issues for me with 11.1.0, you might want to rebuild with that.
here's the unified patch.

mssx86 picture mssx86 on 1 May 2021
👍1

Clang ignores -fsemantic-interposition/-fno-semantic-interposition flag for gcc compatibility

Thanks, I remember seeing output of clang that this flag is ignored in other build systems like CMAKE and didn't adjust it elsewhere. Fixed.

@Mssx86 Thanks for providing the latest version of the patch, my gcc flags which I use are: -O3 -march=native -mtune=native -feliminate-unused-debug-types -fno-semantic-interposition -falign-functions=32 -floop-nest-optimize -fgraphite-identity -flive-range-shrinkage -fno-math-errno -fno-trapping-math -mtls-dialect=gnu2

and I get the following compiler error with GCC-11.1:
CC [M] drivers/hwmon/ltc4215.o
In file included from ./include/linux/mmzone.h:16,
from ./include/linux/gfp.h:6,
from ./include/drm/drm_managed.h:6,
from drivers/gpu/drm/drm_managed.c:8:
./include/linux/seqlock.h: In function ‘read_seqbegin’:
./include/linux/seqlock.h:840:9: internal compiler error: Speicherzugriffsfehler [memory access fault]
840 | unsigned ret = read_seqcount_begin(&sl->seqcount);
| ^~~~
CC fs/d_path.o
CC fs/stack.o
CC [M] drivers/hwmon/ltc4222.o
CC [M] drivers/hwmon/ltc4245.o
0xd8444f internal_error(char const*, ...)
???:0
0x1332aed c_parse_file()
???:0
0x1330633 c_common_parse_file()
???:0
Please submit a full bug report,

EDIT: I can work around this compiler issue with disabling the ltc4215 and other ltc modules in my config. The compilation process succeeds and deb-pkg will be build, but the Kernel still hangs/crashes while loading from the ramdisk.

ms178 picture ms178 on 1 May 2021

@ms178

my gcc flags which I use are: -O3 -march=native -mtune=native -feliminate-unused-debug-types -fno-semantic-interposition -falign-functions=32 -floop-nest-optimize -fgraphite-identity -flive-range-shrinkage -fno-math-errno -fno-trapping-math -mtls-dialect=gnu2

i don't have any of those flags set, i just apply the graysky's -march and the lto-5.12-3 patchset, and have a very barebones kernel with no modules so i don't see any of those warnings.

mssx86 picture mssx86 on 1 May 2021

Without modyfing the Makefile, O3 and INTEL_NATIVE I now see:

CC drivers/devfreq/governor_passive.o
In file included from ./include/linux/i2c.h:18,
from drivers/leds/leds-lm3642.c:8:
./include/linux/rtmutex.h:33:16: internal compiler error: Ungültiger Maschinenbefehl
33 | struct task_struct owner;
| ^~~
0xd8444f internal_error(char const, ...)
???:0
CC [M] drivers/watchdog/w83977f_wdt.o
0xe7d9f9 variably_modified_type_p(tree_node, tree_node)
???:0
0xebaca7 grokfield(unsigned int, c_declarator, c_declspecs, tree_node, tree_node)
???:0
0xe9c7bf c_parser_declspecs(c_parser, c_declspecs*, bool, bool, bool, bool, bool, bool, bool, c_lookahead_kind)
???:0
CC [M] drivers/crypto/cavium/nitrox/nitrox_debugfs.o
0x1332aed c_parse_file()
???:0
0x1330633 c_common_parse_file()
???:0

When I use O2, it compiles fine (edit: but still doesn't boot). This is with the Xanmod sources by the way which sets O3 by default.

ms178 picture ms178 on 1 May 2021

you can find the latest commit (3fb2a4dd) of the lto-5.12-3 branch here with 09da4fc9 removed so that the patchset can be applied to 5.12.3.

mssx86 picture mssx86 on 13 May 2021

you can find the latest commit (3fb2a4dd) of the lto-5.12-3 branch here with 09da4fc9 removed so that the patchset can be applied to 5.12.3.

Thanks, but I still can't get the Kernel to boot. :(

ms178 picture ms178 on 14 May 2021

By the way, the patch from Sami Tolvanan over here for Clang (https://github.com/ClangBuiltLinux/linux/issues/1369#issuecomment-832307198) [I've modified it to use with CONFIG_LTO_GCC] also helps GCC LTO a bit to bring TRIM_UNUSED_KSYMS forward, but it still errors out later at the end of the build process with the following errors (just grabbed the last ones from the console, there are more Module-License errors):

ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/ltc3815.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/max16064.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/max16601.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/max20730.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/max20751.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/max31785.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/max34440.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/max8688.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/pxe1610.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/tps40422.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/tps53679.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/ucd9000.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/ucd9200.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/xdpe12284.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwmon/pmbus/zl6100.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/intel_powerclamp.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/x86_pkg_temp_thermal.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/intel_soc_dts_iosf.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/intel_soc_dts_thermal.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/int3400_thermal.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/int3402_thermal.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/int3403_thermal.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/processor_thermal_device.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/int3406_thermal.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/intel_bxt_pmic_thermal.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/thermal/intel/intel_pch_thermal.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/pretimeout_panic.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/pcwd_pci.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/wdt_pci.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/pcwd_usb.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/cadence_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/twl4030_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/dw_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/retu_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/acquirewdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/advantechwdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/alim1535_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/alim7101_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/ebc-c384_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/f71808e_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/sp5100_tco.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/sbc_fitpc2_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/eurotechwdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/ib700wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/ibmasr.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/wafer5823wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/i6300esb.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/ie6xx_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/iTCO_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/iTCO_vendor_support.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/it8712f_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/it87_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/hpwdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/kempld_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/sc1200wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/pc87413_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/nv_tco.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/sbc60xxwdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/cpu5wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/sch311x_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/smsc37b787_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/tqmx86_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/via_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/w83627hf_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/w83877f_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/w83977f_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/machzwd.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/sbc_epx_c3.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/mei_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/ni903x_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/nic7018_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/of_xilinx_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/mena21_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/da9052_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/da9055_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/da9062_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/da9063_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/wdat_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/wm831x_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/wm8350_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/max63xx_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/ziirave_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/softdog.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/watchdog/menf21bmc_wdt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/raid0.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/multipath.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/md-cluster.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/bcache/bcache.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/md-mod.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-mod.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-bufio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-bio-prison.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-delay.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-flakey.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-multipath.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-round-robin.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-queue-length.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-service-time.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-historical-service-time.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-io-affinity.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-switch.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/persistent-data/dm-persistent-data.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-zero.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-thin-pool.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-verity.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-cache.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-cache-smq.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-ebs.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-era.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-log-writes.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-integrity.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-zoned.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/md/dm-writecache.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/i5000_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/i5100_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/i5400_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/i7300_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/i7core_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/sb_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/pnd2_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/e752x_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/i82975x_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/i3000_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/i3200_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/ie31200_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/x38_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/skx_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/edac/i10nm_edac.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-timer.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-oneshot.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-heartbeat.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-backlight.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-gpio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-activity.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-default-on.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-transient.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-camera.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-netdev.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-pattern.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/trigger/ledtrig-audio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/led-class-flash.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/led-class-multicolor.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-88pm860x.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-adp5520.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-apu.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-as3645a.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-bd2802.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-blinkm.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-clevo-mail.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-da903x.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-da9052.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-gpio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-ss4200.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lm3530.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lm3532.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lm3533.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lm355x.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lm3601x.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lm36274.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lm3642.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lp3944.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lp3952.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-lp8788.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-max8997.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-mc13783.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-menf21bmc.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-mlxcpld.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-mlxreg.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-mt6323.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-nic78bx.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-pca9532.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-pca955x.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-pca963x.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-pwm.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-regulator.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-sgm3140.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-tca6507.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-ti-lmu-common.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-tlc591xx.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-tps6105x.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-wm831x-status.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/leds-wm8350.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/leds/uleds.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/firmware/efi/efi-pstore.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/firmware/efi/efibc.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/firmware/efi/test/efi_test.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/firmware/efi/capsule-loader.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/firmware/dmi-sysfs.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/atmel-i2c.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/atmel-ecc.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/atmel-sha204a.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/cavium/nitrox/n5pf.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/padlock-aes.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/padlock-sha.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/qat/qat_common/intel_qat.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/qat/qat_c3xxx/qat_c3xxx.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/qat/qat_c62x/qat_c62x.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/qat/qat_dh895xccvf/qat_dh895xccvf.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/qat/qat_c3xxxvf/qat_c3xxxvf.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/qat/qat_c62xvf/qat_c62xvf.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/virtio/virtio_crypto.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/inside-secure/crypto_safexcel.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/crypto/amlogic/amlogic-gxl-crypto.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/wmi.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/wmi-bmof.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel-wmi-sbl-fw-update.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/gpd-pocket-fan.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/ibm_rtl.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel-hid.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_int0002_vgpio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_menlow.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel-vbtn.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/pcengines-apuv2.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/i2c-multi-instantiate.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_ips.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel-rst.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel-smartconnect.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_speed_select_if/isst_if_common.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_speed_select_if/isst_if_mmio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_speed_select_if/isst_if_mbox_pci.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_speed_select_if/isst_if_mbox_msr.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel-uncore-frequency.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_bxtwc_tmu.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_chtdc_ti_pwrbtn.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_mrfld_pwrbtn.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_punit_ipc.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_scu_pltdrv.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_scu_ipcutil.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_telemetry_core.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_telemetry_pltdrv.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/platform/x86/intel_telemetry_debugfs.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-arizona.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-axp288.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-fsa9480.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-gpio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-intel-int3496.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-intel-mrfld.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-max14577.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-max3355.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-max77693.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-max77843.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-max8997.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-palmas.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-ptn5150.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-rt8973a.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-sm5502.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/extcon/extcon-usb-gpio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/powercap/intel_rapl_common.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/powercap/intel_rapl_msr.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwtracing/intel_th/intel_th.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwtracing/intel_th/intel_th_pci.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwtracing/intel_th/intel_th_acpi.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwtracing/intel_th/intel_th_gth.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwtracing/intel_th/intel_th_msu.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwtracing/intel_th/intel_th_pti.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hwtracing/intel_th/intel_th_msu_sink.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/target/target_core_mod.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/target/target_core_iblock.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/target/target_core_file.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/target/target_core_pscsi.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/target/target_core_user.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/target/loopback/tcm_loop.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/target/iscsi/iscsi_target_mod.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/parsers/ar7part.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/parsers/cmdlinepart.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/parsers/redboot.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/chipreg.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/cfi_probe.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/cfi_util.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/cfi_cmdset_0020.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/cfi_cmdset_0002.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/cfi_cmdset_0001.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/gen_probe.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/jedec_probe.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/map_ram.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/map_rom.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/chips/map_absent.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/lpddr/qinfo_probe.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/lpddr/lpddr_cmds.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/map_funcs.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/l440gx.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/amd76xrom.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/esb2rom.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/ichxrom.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/ck804xrom.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/physmap.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/sbc_gxx.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/pci.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/nettel.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/scb2_flash.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/plat-ram.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/maps/intel_vr_nor.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/devices/slram.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/devices/phram.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/devices/pmc551.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/devices/mtdram.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/devices/block2mtd.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/onenand/onenand.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/onenand/generic.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/nand.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/sm_common.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/cafe_nand.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/denali.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/denali_pci.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/diskonchip.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/nandsim.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/gpio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/plat_nand.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/r852.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/mxic_nand.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/raw/arasan-nand-controller.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nand/nandcore.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/mtd.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/mtd_blkdevs.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/mtdblock.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/mtdblock_ro.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/ftl.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/nftl.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/inftl.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/rfd_ftl.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/ssfdc.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/sm_ftl.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/mtdoops.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/mtdpstore.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/mtdswap.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/ubi/ubi.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/ubi/gluebi.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/mtd/hyperbus/hyperbus-core.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/uio/uio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/uio/uio_pdrv_genirq.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/uio/uio_dmem_genirq.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/uio/uio_pci_generic.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/masters/matrox_w1.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/masters/ds2490.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/masters/ds2482.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/masters/ds1wm.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/masters/w1-gpio.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/masters/sgi_w1.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_therm.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_smem.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2405.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2408.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2413.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2406.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2423.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2430.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2431.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2805.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2433.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2438.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds250x.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2780.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds2781.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds28e04.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/slaves/w1_ds28e17.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/w1/wire.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/dca/dca.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/hid.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/uhid.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/hid-generic.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/hid-led.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/hid-sensor-hub.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/hid-sensor-custom.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/usbhid/usbhid.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/usbhid/usbkbd.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/usbhid/usbmouse.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/intel-ish-hid/intel-ishtp.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/intel-ish-hid/intel-ish-ipc.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/intel-ish-hid/intel-ishtp-hid.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/hid/intel-ish-hid/intel-ishtp-loader.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/rpmsg/rpmsg_core.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/rpmsg/rpmsg_char.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/rpmsg/rpmsg_ns.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/rpmsg/qcom_glink.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/rpmsg/qcom_glink_rpm.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/rpmsg/virtio_rpmsg_bus.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/soundwire/soundwire-bus.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/soundwire/soundwire-generic-allocation.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/soundwire/soundwire-cadence.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/soundwire/soundwire-intel.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/ntb/ntb.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/ntb/hw/idt/ntb_hw_idt.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/ntb/hw/intel/ntb_hw_intel.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/ntb/hw/mscc/ntb_hw_switchtec.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/ntb/test/ntb_pingpong.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/ntb/test/ntb_tool.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/ntb/test/ntb_perf.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/ntb/test/ntb_msi_test.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/ntb/ntb_transport.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/counter/counter.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soundcore.o
ERROR: modpost: missing MODULE_LICENSE() in sound/core/snd.o
ERROR: modpost: missing MODULE_LICENSE() in sound/core/snd-hwdep.o
ERROR: modpost: missing MODULE_LICENSE() in sound/core/snd-timer.o
ERROR: modpost: missing MODULE_LICENSE() in sound/core/snd-hrtimer.o
ERROR: modpost: missing MODULE_LICENSE() in sound/core/snd-pcm.o
ERROR: modpost: missing MODULE_LICENSE() in sound/core/snd-pcm-dmaengine.o
ERROR: modpost: missing MODULE_LICENSE() in sound/core/snd-rawmidi.o
ERROR: modpost: missing MODULE_LICENSE() in sound/drivers/snd-aloop.o
ERROR: modpost: missing MODULE_LICENSE() in sound/pci/snd-intel8x0.o
ERROR: modpost: missing MODULE_LICENSE() in sound/pci/ac97/snd-ac97-codec.o
ERROR: modpost: missing MODULE_LICENSE() in sound/pci/hda/snd-hda-codec.o
ERROR: modpost: missing MODULE_LICENSE() in sound/pci/hda/snd-hda-codec-generic.o
ERROR: modpost: missing MODULE_LICENSE() in sound/pci/hda/snd-hda-codec-realtek.o
ERROR: modpost: missing MODULE_LICENSE() in sound/pci/hda/snd-hda-codec-hdmi.o
ERROR: modpost: missing MODULE_LICENSE() in sound/pci/hda/snd-hda-intel.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/snd-usb-audio.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/snd-usbmidi-lib.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/misc/snd-ua101.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/usx2y/snd-usb-usx2y.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/usx2y/snd-usb-us122l.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/caiaq/snd-usb-caiaq.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/6fire/snd-usb-6fire.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/hiface/snd-usb-hiface.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/bcd2000/snd-bcd2000.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/line6/snd-usb-line6.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/line6/snd-usb-pod.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/line6/snd-usb-podhd.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/line6/snd-usb-toneport.o
ERROR: modpost: missing MODULE_LICENSE() in sound/usb/line6/snd-usb-variax.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/snd-soc-acpi.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/snd-soc-core.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-ac97.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-alc5623.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-dmic.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-hdac-hdmi.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-max9759.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-max98373.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-max98373-i2c.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-mt6351.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-mt6358.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-mt6660.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-nau8540.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-nau8810.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-nau8822.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-nau8824.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-hdmi-codec.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-pcm1789-i2c.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-pcm1789-codec.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-pcm512x.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-pcm512x-i2c.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rl6231.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt1011.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt1015.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt1308-sdw.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt5616.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt5631.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt5640.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt5659.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt5682.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt5682-i2c.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt5682-sdw.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt700.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt711.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-rt715.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-spdif-rx.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-spdif-tx.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-sta32x.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-sta350.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-sti-sas.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tas2552.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tas2562.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tas5086.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tas571x.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tas5720.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tas6424.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tda7419.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tas2770.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tfa9879.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tlv320aic23.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tlv320aic23-i2c.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tlv320aic31xx.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tlv320aic32x4.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tlv320aic32x4-i2c.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tlv320aic3x.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tlv320adcx140.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tscs42xx.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tscs454.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-ts3a227e.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-uda1334.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8510.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8523.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8524.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8580.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8711.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8728.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8731.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8737.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8741.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8750.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8753.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8776.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8782.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8804.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8804-i2c.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8903.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8904.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8960.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8962.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8974.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8978.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wm8985.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-wsa881x.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-simple-amplifier.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/codecs/snd-soc-tpa6130a2.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/generic/snd-soc-simple-card-utils.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/generic/snd-soc-simple-card.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/dwc/designware_i2s.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-fsl-audmix.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-fsl-asrc.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-fsl-sai.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-fsl-ssi.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-fsl-spdif.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-fsl-esai.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-fsl-micfil.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-fsl-mqs.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-fsl-easrc.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/fsl/snd-soc-imx-audmux.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/intel/common/snd-soc-acpi-intel-match.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/intel/catpt/snd-soc-catpt.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/intel/boards/snd-soc-sof_rt5682.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/intel/boards/snd-soc-sst-haswell.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/intel/boards/snd-soc-sst-sof-pcm512x.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/sof/intel/snd-sof-acpi-intel-byt.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/sof/intel/snd-sof-intel-ipc.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/sof/snd-sof.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/sof/snd-sof-acpi.o
ERROR: modpost: missing MODULE_LICENSE() in sound/soc/sof/xtensa/snd-sof-xtensa-dsp.o
ERROR: modpost: missing MODULE_LICENSE() in sound/hda/snd-hda-core.o
ERROR: modpost: missing MODULE_LICENSE() in sound/hda/ext/snd-hda-ext-core.o
ERROR: modpost: missing MODULE_LICENSE() in sound/hda/snd-intel-dspcfg.o
ERROR: modpost: missing MODULE_LICENSE() in sound/hda/snd-intel-sdw-acpi.o
ERROR: modpost: missing MODULE_LICENSE() in sound/ac97_bus.o
ERROR: modpost: missing MODULE_LICENSE() in net/core/pktgen.o
ERROR: modpost: missing MODULE_LICENSE() in net/core/failover.o
ERROR: modpost: missing MODULE_LICENSE() in net/802/p8022.o
ERROR: modpost: missing MODULE_LICENSE() in net/802/psnap.o
ERROR: modpost: missing MODULE_LICENSE() in net/802/stp.o
ERROR: modpost: missing MODULE_LICENSE() in net/802/garp.o
ERROR: modpost: missing MODULE_LICENSE() in net/802/mrp.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_police.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_gact.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_mirred.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_sample.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_ipt.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_nat.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_pedit.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_simple.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_skbedit.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_csum.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_mpls.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_vlan.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_bpf.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_connmark.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_ctinfo.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_skbmod.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_tunnel_key.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_ct.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/act_gate.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_cbq.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_htb.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_hfsc.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_red.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_gred.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_ingress.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_dsmark.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_sfb.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_sfq.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_tbf.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_teql.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_prio.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_multiq.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_atm.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_netem.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_drr.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_plug.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_ets.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_mqprio.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_skbprio.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_choke.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_qfq.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_codel.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_fq_codel.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_cake.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_fq.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_hhf.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_cbs.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_etf.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/sch_taprio.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_u32.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_route.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_fw.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_rsvp.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_tcindex.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_rsvp6.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_basic.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_flow.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_cgroup.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_bpf.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_flower.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/cls_matchall.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/em_cmp.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/em_nbyte.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/em_u32.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/em_meta.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/em_text.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/em_ipset.o
ERROR: modpost: missing MODULE_LICENSE() in net/sched/em_ipt.o
ERROR: modpost: missing MODULE_LICENSE() in net/netlink/netlink_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nfnetlink.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nfnetlink_acct.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nfnetlink_queue.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nfnetlink_log.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nfnetlink_osf.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_netlink.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nfnetlink_cttimeout.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nfnetlink_cthelper.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_amanda.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_ftp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_h323.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_irc.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_broadcast.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_netbios_ns.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_snmp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_pptp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_sane.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_sip.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conntrack_tftp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_log_common.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_log_netdev.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_nat.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_nat_amanda.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_nat_ftp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_nat_irc.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_nat_sip.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_nat_tftp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_synproxy_core.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_conncount.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_dup_netdev.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_tables.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_compat.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_connlimit.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_numgen.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_ct.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_flow_offload.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_limit.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_nat.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_objref.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_queue.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_quota.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_reject.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_reject_inet.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_tunnel.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_counter.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_log.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_masq.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_redir.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_hash.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_fib.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_fib_inet.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_fib_netdev.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_socket.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_osf.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_tproxy.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_xfrm.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_synproxy.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_chain_nat.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_dup_netdev.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nft_fwd_netdev.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_flow_table.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/nf_flow_table_inet.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/x_tables.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_tcpudp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_mark.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_connmark.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_set.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_nat.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_AUDIT.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_CHECKSUM.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_CLASSIFY.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_CONNSECMARK.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_CT.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_DSCP.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_HL.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_HMARK.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_LED.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_LOG.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_NETMAP.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_NFLOG.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_NFQUEUE.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_RATEEST.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_REDIRECT.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_MASQUERADE.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_SECMARK.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_TPROXY.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_TCPMSS.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_TCPOPTSTRIP.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_TEE.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_TRACE.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_IDLETIMER.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_addrtype.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_bpf.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_cluster.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_comment.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_connbytes.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_connlabel.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_connlimit.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_conntrack.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_cpu.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_dccp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_devgroup.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_dscp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_ecn.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_esp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_hashlimit.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_helper.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_hl.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_ipcomp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_iprange.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_ipvs.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_l2tp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_length.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_limit.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_mac.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_multiport.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_nfacct.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_osf.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_owner.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_cgroup.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_physdev.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_pkttype.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_policy.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_quota.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_rateest.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_realm.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_recent.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_sctp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_socket.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_state.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_statistic.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_string.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_tcpmss.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_time.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/xt_u32.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_bitmap_ip.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_bitmap_ipmac.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_bitmap_port.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_ip.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_ipmac.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_ipmark.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_ipport.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_ipportip.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_ipportnet.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_mac.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_net.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_netport.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_netiface.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_netnet.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_hash_netportnet.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipset/ip_set_list_set.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_rr.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_wrr.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_lc.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_wlc.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_fo.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_ovf.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_lblc.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_lblcr.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_dh.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_sh.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_mh.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_sed.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_nq.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_ftp.o
ERROR: modpost: missing MODULE_LICENSE() in net/netfilter/ipvs/ip_vs_pe_sip.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_defrag_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_socket_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_tproxy_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_log_arp.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_log_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_reject_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_nat_h323.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_nat_pptp.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_nat_snmp_basic.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nft_reject_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nft_fib_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nft_dup_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_flow_table_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/ip_tables.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/iptable_filter.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/iptable_mangle.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/iptable_nat.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/iptable_raw.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/ipt_ah.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/ipt_rpfilter.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/ipt_CLUSTERIP.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/ipt_ECN.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/ipt_REJECT.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/ipt_SYNPROXY.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/arp_tables.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/arpt_mangle.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/arptable_filter.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/netfilter/nf_dup_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/ip_tunnel.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/ipip.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/fou.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/gre.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/ip_gre.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/udp_tunnel.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/ip_vti.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/ah4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/esp4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/esp4_offload.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/ipcomp.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/xfrm4_tunnel.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tunnel4.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/inet_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/udp_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/raw_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_bic.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_cdg.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_cubic.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_dctcp.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_westwood.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_highspeed.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_hybla.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_htcp.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_vegas.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_nv.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_veno.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_scalable.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_lp.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_yeah.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv4/tcp_illinois.o
ERROR: modpost: missing MODULE_LICENSE() in net/xfrm/xfrm_algo.o
ERROR: modpost: missing MODULE_LICENSE() in net/xfrm/xfrm_user.o
ERROR: modpost: missing MODULE_LICENSE() in net/xfrm/xfrm_ipcomp.o
ERROR: modpost: missing MODULE_LICENSE() in net/xfrm/xfrm_interface.o
ERROR: modpost: missing MODULE_LICENSE() in net/unix/unix_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6_tables.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6table_filter.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6table_mangle.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6table_raw.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6table_nat.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nf_defrag_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nf_socket_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nf_tproxy_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nf_log_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nf_reject_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nf_dup_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nft_reject_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nft_dup_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nft_fib_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/nf_flow_table_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_ah.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_eui64.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_frag.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_ipv6header.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_mh.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_hbh.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_rpfilter.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_rt.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_srh.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_NPT.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_REJECT.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/netfilter/ip6t_SYNPROXY.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/ah6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/esp6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/esp6_offload.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/ipcomp6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/xfrm6_tunnel.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/tunnel6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/mip6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/ila/ila.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/ip6_vti.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/sit.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/ip6_tunnel.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/ip6_gre.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/fou6.o
ERROR: modpost: missing MODULE_LICENSE() in net/ipv6/ip6_udp_tunnel.o
ERROR: modpost: missing MODULE_LICENSE() in net/bpfilter/bpfilter.o
ERROR: modpost: missing MODULE_LICENSE() in net/packet/af_packet_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/8021q/8021q.o
ERROR: modpost: missing MODULE_LICENSE() in net/mpls/mpls_gso.o
ERROR: modpost: missing MODULE_LICENSE() in net/mpls/mpls_router.o
ERROR: modpost: missing MODULE_LICENSE() in net/mpls/mpls_iptunnel.o
ERROR: modpost: missing MODULE_LICENSE() in net/xdp/xsk_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/mptcp/mptcp_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/llc/llc.o
ERROR: modpost: missing MODULE_LICENSE() in net/llc/llc2.o
ERROR: modpost: missing MODULE_LICENSE() in net/tls/tls.o
ERROR: modpost: missing MODULE_LICENSE() in net/key/af_key.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/nft_meta_bridge.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/nft_reject_bridge.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/nf_conntrack_bridge.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/nf_log_bridge.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebtables.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebtable_broute.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebtable_filter.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebtable_nat.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_802_3.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_among.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_arp.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_ip.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_ip6.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_limit.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_mark_m.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_pkttype.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_stp.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_vlan.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_arpreply.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_mark.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_dnat.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_redirect.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_snat.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_log.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/netfilter/ebt_nflog.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/bridge.o
ERROR: modpost: missing MODULE_LICENSE() in net/bridge/br_netfilter.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/dsa_core.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_8021q.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_ar9331.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_brcm.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_dsa.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_gswip.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_ksz.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_rtl4_a.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_lan9303.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_mtk.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_ocelot.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_qca.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_sja1105.o
ERROR: modpost: missing MODULE_LICENSE() in net/dsa/tag_trailer.o
ERROR: modpost: missing MODULE_LICENSE() in net/appletalk/appletalk.o
ERROR: modpost: missing MODULE_LICENSE() in net/x25/x25.o
ERROR: modpost: missing MODULE_LICENSE() in net/lapb/lapb.o
ERROR: modpost: missing MODULE_LICENSE() in net/sunrpc/sunrpc.o
ERROR: modpost: missing MODULE_LICENSE() in net/sunrpc/auth_gss/auth_rpcgss.o
ERROR: modpost: missing MODULE_LICENSE() in net/sunrpc/auth_gss/rpcsec_gss_krb5.o
ERROR: modpost: missing MODULE_LICENSE() in net/rxrpc/rxrpc.o
ERROR: modpost: missing MODULE_LICENSE() in net/atm/atm.o
ERROR: modpost: missing MODULE_LICENSE() in net/atm/clip.o
ERROR: modpost: missing MODULE_LICENSE() in net/atm/br2684.o
ERROR: modpost: missing MODULE_LICENSE() in net/atm/lec.o
ERROR: modpost: missing MODULE_LICENSE() in net/atm/mpoa.o
ERROR: modpost: missing MODULE_LICENSE() in net/l2tp/l2tp_core.o
ERROR: modpost: missing MODULE_LICENSE() in net/l2tp/l2tp_ppp.o
ERROR: modpost: missing MODULE_LICENSE() in net/l2tp/l2tp_ip.o
ERROR: modpost: missing MODULE_LICENSE() in net/l2tp/l2tp_netlink.o
ERROR: modpost: missing MODULE_LICENSE() in net/l2tp/l2tp_eth.o
ERROR: modpost: missing MODULE_LICENSE() in net/l2tp/l2tp_debugfs.o
ERROR: modpost: missing MODULE_LICENSE() in net/l2tp/l2tp_ip6.o
ERROR: modpost: missing MODULE_LICENSE() in net/decnet/netfilter/dn_rtmsg.o
ERROR: modpost: missing MODULE_LICENSE() in net/decnet/decnet.o
ERROR: modpost: missing MODULE_LICENSE() in net/phonet/phonet.o
ERROR: modpost: missing MODULE_LICENSE() in net/phonet/pn_pep.o
ERROR: modpost: missing MODULE_LICENSE() in net/dccp/dccp.o
ERROR: modpost: missing MODULE_LICENSE() in net/dccp/dccp_ipv4.o
ERROR: modpost: missing MODULE_LICENSE() in net/dccp/dccp_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/dccp/dccp_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/sctp/sctp.o
ERROR: modpost: missing MODULE_LICENSE() in net/sctp/sctp_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/rds/rds.o
ERROR: modpost: missing MODULE_LICENSE() in net/rds/rds_tcp.o
ERROR: modpost: missing MODULE_LICENSE() in net/tipc/tipc.o
ERROR: modpost: missing MODULE_LICENSE() in net/tipc/diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/6lowpan/6lowpan.o
ERROR: modpost: missing MODULE_LICENSE() in net/6lowpan/nhc_dest.o
ERROR: modpost: missing MODULE_LICENSE() in net/6lowpan/nhc_fragment.o
ERROR: modpost: missing MODULE_LICENSE() in net/6lowpan/nhc_hop.o
ERROR: modpost: missing MODULE_LICENSE() in net/6lowpan/nhc_ipv6.o
ERROR: modpost: missing MODULE_LICENSE() in net/6lowpan/nhc_mobility.o
ERROR: modpost: missing MODULE_LICENSE() in net/6lowpan/nhc_routing.o
ERROR: modpost: missing MODULE_LICENSE() in net/6lowpan/nhc_udp.o
ERROR: modpost: missing MODULE_LICENSE() in net/ieee802154/6lowpan/ieee802154_6lowpan.o
ERROR: modpost: missing MODULE_LICENSE() in net/ieee802154/ieee802154.o
ERROR: modpost: missing MODULE_LICENSE() in net/ieee802154/ieee802154_socket.o
ERROR: modpost: missing MODULE_LICENSE() in net/mac802154/mac802154.o
ERROR: modpost: missing MODULE_LICENSE() in net/ceph/libceph.o
ERROR: modpost: missing MODULE_LICENSE() in net/batman-adv/batman-adv.o
ERROR: modpost: missing MODULE_LICENSE() in net/psample/psample.o
ERROR: modpost: missing MODULE_LICENSE() in net/ife/ife.o
ERROR: modpost: missing MODULE_LICENSE() in net/openvswitch/openvswitch.o
ERROR: modpost: missing MODULE_LICENSE() in net/openvswitch/vport-gre.o
ERROR: modpost: missing MODULE_LICENSE() in net/vmw_vsock/vsock.o
ERROR: modpost: missing MODULE_LICENSE() in net/vmw_vsock/vsock_diag.o
ERROR: modpost: missing MODULE_LICENSE() in net/vmw_vsock/vmw_vsock_virtio_transport.o
ERROR: modpost: missing MODULE_LICENSE() in net/vmw_vsock/vmw_vsock_virtio_transport_common.o
ERROR: modpost: missing MODULE_LICENSE() in net/vmw_vsock/vsock_loopback.o
ERROR: modpost: missing MODULE_LICENSE() in net/nsh/nsh.o
ERROR: modpost: missing MODULE_LICENSE() in net/hsr/hsr.o
ERROR: modpost: missing MODULE_LICENSE() in net/qrtr/qrtr.o
ERROR: modpost: missing MODULE_LICENSE() in net/qrtr/ns.o
ERROR: modpost: missing MODULE_LICENSE() in net/qrtr/qrtr-smd.o
ERROR: modpost: missing MODULE_LICENSE() in net/qrtr/qrtr-tun.o
ERROR: modpost: missing MODULE_LICENSE() in net/qrtr/qrtr-mhi.o
ERROR: modpost: "cast_s4" [arch/x86/crypto/cast6-avx-x86_64.ko] undefined!
ERROR: modpost: "cast_s1" [arch/x86/crypto/cast6-avx-x86_64.ko] undefined!
ERROR: modpost: "cast_s2" [arch/x86/crypto/cast6-avx-x86_64.ko] undefined!
ERROR: modpost: "cast_s3" [arch/x86/crypto/cast6-avx-x86_64.ko] undefined!
ERROR: modpost: "cast_s4" [arch/x86/crypto/cast5-avx-x86_64.ko] undefined!
ERROR: modpost: "cast_s1" [arch/x86/crypto/cast5-avx-x86_64.ko] undefined!
ERROR: modpost: "cast_s2" [arch/x86/crypto/cast5-avx-x86_64.ko] undefined!
ERROR: modpost: "cast_s3" [arch/x86/crypto/cast5-avx-x86_64.ko] undefined!
ERROR: modpost: "camellia_sp00444404" [arch/x86/crypto/camellia-x86_64.ko] undefined!
ERROR: modpost: "camellia_sp44044404" [arch/x86/crypto/camellia-x86_64.ko] undefined!
ERROR: modpost: "camellia_sp30333033" [arch/x86/crypto/camellia-x86_64.ko] undefined!
ERROR: modpost: "camellia_sp22000222" [arch/x86/crypto/camellia-x86_64.ko] undefined!
ERROR: modpost: "camellia_sp03303033" [arch/x86/crypto/camellia-x86_64.ko] undefined!
ERROR: modpost: "camellia_sp10011110" [arch/x86/crypto/camellia-x86_64.ko] undefined!
ERROR: modpost: "camellia_sp02220222" [arch/x86/crypto/camellia-x86_64.ko] undefined!
ERROR: modpost: "camellia_sp11101110" [arch/x86/crypto/camellia-x86_64.ko] undefined!

ms178 picture ms178 on 14 May 2021

@ms178 i am not getting any of the errors you're describing with andikleen's patchset, though i have no modules. also, i don't think what you're doing is a healthy approach as the way gcc and clang does lto are fundamentally different so i doubt you can succeed that way.

i don't know if it helps you in any way, shape or form but these are my binutils and gcc versions and configuration (bare in mind that this is an x86_64 musl 1.2.2 host and i'm not running gentoo):

 0 ~: gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-apathy-linux-musl/11.1.0/lto-wrapper
Target: x86_64-apathy-linux-musl
Configured with: ../configure --prefix=/usr --libexecdir=/usr/libexec --mandir=/usr/share/man --infodir=/usr/share/info --host=x86_64-apathy-linux-musl --build=x86_64-apathy-linux-musl --target=x86_64-apathy-linux-musl --with-pkgversion=apathy --enable-checking=release --enable-languages=c,c++,lto --enable-cxx-flags='-w -pipe -O2 -mtune=native -march=native' --with-linker-hash-style=gnu --with-system-zlib --enable-__cxa_atexit --enable-bootstrap --enable-linker-build-id --enable-lto --enable-plugin --enable-shared --enable-threads=posix --enable-tls --without-included-gettext --without-isl --disable-default-pie --disable-default-ssp --disable-fixed-point --disable-libmpx --disable-libmudflap --disable-libsanitizer --disable-libstdcxx-pch --disable-multilib --disable-nls --disable-rpath --disable-static --disable-symvers --disable-werror
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.1.0 (apathy)
 0 ~: ld.bfd -v
GNU ld (GNU Binutils) 2.36.1

you can try doing a make localmodconfig without a .config in the root so that it reads from /proc/config.gz then slim down your kernel a bit with make menuconfig or make nconfig by manually going through each page and disabling everything you don't need, maybe even bake in the modules.

mssx86 picture mssx86 on 14 May 2021

@mssx86 Well, problems were to be expected with my experiment of today which I just wanted to share with the community here. The result was interesting as it brought some progress even with GCC while not fixing the issue entirely (trimmed KSYSMS are a known issue with both Clang LTO and GCC LTO, just Clang's ThinLTO does work reliably with Sami's patch and that option now, FullLTO ran into the same boot issue).

Regarding my boot problems, Andi just told me that he was able to reproduce it and has it on his radar to debug further. My Kernel is already trimmed down heavily and way slimmer than anything Ubuntu ships with at default and building without modules had no effect, building it with less aggressive compiler flags had no effects either.

ms178 picture ms178 on 14 May 2021

What's the easiest way of applying LTO patches to a gentoo-src kernel?

barolo picture barolo on 17 Jun 2021

@barolo not aware whether the current andikleen lto branches apply to current kernels but what i do is cloning the repo with --depth=n+1 -b lto-X.X-X where n is the number of lto commits, then generate patches with git format-patch -n.

mssx86 picture mssx86 on 17 Jun 2021

@barolo not aware whether the current andikleen lto branches apply to current kernels but what i do is cloning the repo with --depth=n+1 -b lto-X.X-X where n is the number of lto commits, then generate patches with git format-patch -n.

Do they apply cleanly to a corresponding gentoo-src kernel?
I remember having some issues and pretty much did the inverse, cloned andi's patched kernel and applied gentoo's patches to it.

barolo picture barolo on 17 Jun 2021

Do they apply cleanly to a corresponding gentoo-src kernel?

only one way to find out like i said in the beginning of my first post.

mssx86 picture mssx86 on 17 Jun 2021

Do they apply cleanly to a corresponding gentoo-src kernel?

only one way to find out like i said in the beginning of my first post.

Your way worked beautifully, applied to 5.12.10-gentoo with everything built-in, GCC 11.
Thanks!
The weird thing is that it's 1MB bigger than non-lto one, which makes me think that something isn't right.

barolo picture barolo on 26 Jun 2021

@barolo

Your way worked beautifully, applied to 5.12.10-gentoo with everything built-in, GCC 11.

glad to hear.

The weird thing is that it's 1MB bigger than non-lto one, which makes me think that something isn't right.

that is also the case with my kernel config w/ gcc 11.1.0 stable, 5.8 patchset reduced ~1.5 megs iirc.

mssx86 picture mssx86 on 26 Jun 2021

Unfortunately that kernel is unstable for me, I get occasional hard locks, possibly related to gpu drivers [AMD], gonna try clang.

barolo picture barolo on 27 Jun 2021

Does anyone have idea how to enable clang build for Andi's kernel? I have only options for GCC

barolo picture barolo on 28 Jun 2021

@barolo you don't need andikleen's patchset for clang lto, clang lto patchset is the work of sami tolvanen from google and 5.12+ already have the support for it in the tree.

to enable it, you need to invoke all make instances with LLVM=1 LLVM_IAS=1. you also need clang and lld 11.0.0+ otherwise you cannot pass the CLANG_VERSION >=110000 and LD_IS_LLD checks for the HAS_LTO_CLANG to be set so that you can enable clang lto.

mssx86 picture mssx86 on 28 Jun 2021
❤1

Clang one seems to be stable, but again, it's even bigger.

barolo picture barolo on 29 Jun 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

prototype99 picture prototype99  Â·  3Comments
InBetweenNames picture InBetweenNames  Â·  17Comments
pchome picture pchome  Â·  14Comments
jelinekto picture jelinekto  Â·  11Comments
ElDavoo picture ElDavoo  Â·  10Comments