Zstd: Add zstd support to kmod

[tpgxyz]

Hi,

Zstandard is doing fine and making great progress in adoption in various software.

With kernel supporting zstd there is a missing support in kmod. Without this mod{probe,info} does not recognize ko.zst kernel modules compressed with zstd so loading such modules is not possible

[terrelln]

If anyone wants to work on this, I'm happy to review and test the code.

[Cyan4973]

Should this rather be a topic tracked within kmod issue board itself ?

[tpgxyz]

@Cyan4973 kmod developers did not respond to that call

[Cyan4973]

What about this patch ?
http://lists.openwrt.org/pipermail/openwrt-devel/2017-December/010228.html ?

[tpgxyz]

@Cyan4973 this is a patch for btrfs not for kmod (https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git/)

[Cyan4973]

This topic is more appropriate on the kmod project.

[tpgxyz]

Hi,

thanks to @abucodonosor for adding support to kmod for zstd kernel compressed modules.
Would be nice if you guys @Cyan4973 @terrelln could add this patch https://gitweb.frugalware.org/frugalware-current/raw/master/source/base/kmod/zstd-support.patch
to /contrib/kmod and give a shout out :)

We at OpenMandriva are updated kmod with it and currently running tests:
https://github.com/OpenMandrivaAssociation/kmod
For now no regressions found, boot speed inceraced over kernel with XZ compressed modules

[terrelln]

I'd be happy to add it. If someone would like to put up a PR with the patch + a README.md with some info (like measured gains) that would be great!

[abucodonosor]

@terrelln

at this point, the patch was not meant to go upstream, it was more meant to have something working so full gzip vs xz vs zstd kernel+modules+initramfs can be tested.

I know kmod folks kind doesn't want to add that support more like they don't
want lzo or lz4 support for some reason.

I think the right way to get that upstream would be to push first for your remaining
ZSTD kernel patches ( somehow v2 is from back 2017? ), then after that is in we can
improve the kmod patch and actually get the Kbuild folks involved too since then we can do a full benchmark of gzip vs xz vs zstd without out-of-tree-kernel patches.
( Yes I know modules can be compressed without your patches but then we cannot
have a full benchmark )

As for performance, I didn't really measure much right now, but @tpgxyz is right
compared to an XZ kernel, modules, initramfs the ZSTD kernel+modules+initramfs
seems to be 2.6 to 3.8 secs faster booting on my testing HW. I'll do some testing on my Epyc server soon too.

[terrelln]

I think the right way to get that upstream would be to push first for your remaining
ZSTD kernel patches ( somehow v2 is from back 2017? ), then after that is in we can
improve the kmod patch and actually get the Kbuild folks involved too since then we can do a full benchmark of gzip vs xz vs zstd without out-of-tree-kernel patches.
( Yes I know modules can be compressed without your patches but then we cannot
have a full benchmark )

I just resubmitted the compressed initramfs/kernel patches last week. However, I just noticed they didn't get picked up by the mailing list, so I resent them just now. You can follow along here https://lkml.org/lkml/2020/3/25/991.

[abucodonosor]

@terrelln

cool, thx :-). Will give these a try, was about to build some testing git kernels later
on anyway.

Btw, I think the decompress functions from patch 3/8, with small modifications, can be copied for kmod. I think your code looks much better, much more optimized. After all, I am not a ZSTD developer.

[abucodonosor]

@terrelln

I have tested your v3 and now v4 on some boxes, from Laptops to Epyc server.
Not a problem found, good work :-).

I've added a comment and a Tested-by: line on LKML
https://lkml.org/lkml/2020/4/1/374

If you need any testing or similar feel free to always CC or direct email me about.

Thanks for your work on this :-)

[terrelln]

Thanks for testing @abucodonosor!