Kubeadm: RequiredIPVSKernelModulesAvailable warning seems confusing

Created on 6 Jul 2018  路  37Comments  路  Source: kubernetes/kubeadm

What keywords did you search in kubeadm issues before filing this one?

ip_vs, RequiredIPVSKernelModulesAvailable

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
    aws ec2
  • OS (e.g. from /etc/os-release):
    debian stretch
  • Kernel (e.g. uname -a):
    Linux debian 4.9.0-6-amd64 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
  • Others:

What happened?

ipvs is installed but modules not loaded yet, kubeadm show warning RequiredIPVSKernelModulesAvailable

# lsmod | grep ip_vs
$ cat kubeadm.yml
kubeProxy:
  config:
    mode: ipvs
    ipvs:
      scheduler: lc
$ sudo kubeadm init --config kubeadm.yml
...
I0706 07:18:56.609843    1084 kernel_validator.go:96] Validating kernel config
    [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

 Pulling images required for setting up a Kubernetes cluster
...

But after cluster is up, ipvs modules are actually loaded by kube-proxy and i see Services VIPs in ipvsadm -ln

What you expected to happen?

This is confusing.
Maybe warning should only be shown when such modules are not exist in system?

areecosystem kinbug lifecyclactive sicluster-lifecycle sinode statneeds-more-information
Source
picture sepich
👍1

Most helpful comment

I see the same warning when joining a new node to the cluster. None of the modules are loaded into the kernel to start with, but after running kubeadm join, they are all there:

Before:

# lsmod | grep ^ip_vs
(nothing)

After:

# lsmod | grep ^ip_vs
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 147456  6 ip_vs_wrr,ip_vs_rr,ip_vs_sh

Is it a case of the preflight check not understanding that modules can be dynamically loaded when needed?

I'm running v1.11.0 from the apt repo on Ubuntu 16.04.

picture cu on 17 Jul 2018
👍3

All 37 comments

@sepich

please, create a shell script from this and show me what output do you get:

#!/bin/sh
cut -f1 -d" " /proc/modules
echo -----
version=`cut -f1 -d" " /proc/sys/kernel/osrelease`
echo $version
echo -----
cut -f1 -d" " /lib/modules/$version/modules.builtin
neolit123 picture neolit123 on 6 Jul 2018

Please note that it is CI VM which is created, booted up, and then terminated after tests.

Right after boot

root@ip-10-99-134-230:~# bash 1.sh
xt_nat
xt_tcpudp
veth
vxlan
ip6_udp_tunnel
udp_tunnel
iptable_mangle
xt_mark
ipt_MASQUERADE
nf_nat_masquerade_ipv4
nf_conntrack_netlink
nfnetlink
xfrm_user
xfrm_algo
iptable_nat
nf_conntrack_ipv4
nf_defrag_ipv4
nf_nat_ipv4
xt_addrtype
iptable_filter
xt_conntrack
nf_nat
nf_conntrack
br_netfilter
bridge
stp
llc
overlay
intel_rapl
cirrus
sb_edac
edac_core
crct10dif_pclmul
crc32_pclmul
ghash_clmulni_intel
ttm
drm_kms_helper
drm
intel_rapl_perf
evdev
serio_raw
pcspkr
button
ip_tables
x_tables
autofs4
ext4
crc16
jbd2
crc32c_generic
fscrypto
ecb
mbcache
ata_generic
crc32c_intel
aesni_intel
xen_netfront
xen_blkfront
aes_x86_64
glue_helper
lrw
gf128mul
ablk_helper
cryptd
ata_piix
psmouse
i2c_piix4
libata
scsi_mod
floppy
-----
4.9.0-6-amd64
-----
kernel/arch/x86/ia32/ia32_aout.ko
kernel/arch/x86/platform/intel/iosf_mbi.ko
kernel/mm/zpool.ko
kernel/mm/zbud.ko
kernel/fs/binfmt_script.ko
kernel/fs/exportfs/exportfs.ko
kernel/fs/nls/nls_base.ko
kernel/fs/pstore/pstore.ko
kernel/crypto/crypto.ko
kernel/crypto/crypto_wq.ko
kernel/crypto/crypto_algapi.ko
kernel/crypto/aead.ko
kernel/crypto/crypto_blkcipher.ko
kernel/crypto/crypto_hash.ko
kernel/crypto/akcipher.ko
kernel/crypto/kpp.ko
kernel/crypto/cryptomgr.ko
kernel/crypto/hmac.ko
kernel/crypto/crypto_null.ko
kernel/crypto/md5.ko
kernel/crypto/sha1_generic.ko
kernel/crypto/sha256_generic.ko
kernel/crypto/aes_generic.ko
kernel/crypto/crct10dif_common.ko
kernel/crypto/crct10dif_generic.ko
kernel/crypto/lzo.ko
kernel/crypto/rng.ko
kernel/block/deadline-iosched.ko
kernel/block/cfq-iosched.ko
kernel/drivers/acpi/processor.ko
kernel/drivers/acpi/hed.ko
kernel/drivers/acpi/acpi_extlog.ko
kernel/drivers/base/firmware_class.ko
kernel/drivers/base/regmap/regmap-i2c.ko
kernel/drivers/base/regmap/regmap-spi.ko
kernel/drivers/char/agp/agpgart.ko
kernel/drivers/char/agp/amd64-agp.ko
kernel/drivers/char/agp/intel-agp.ko
kernel/drivers/char/agp/intel-gtt.ko
kernel/drivers/char/agp/sis-agp.ko
kernel/drivers/char/agp/via-agp.ko
kernel/drivers/char/tpm/tpm.ko
kernel/drivers/char/tpm/tpm_tis_core.ko
kernel/drivers/char/tpm/tpm_tis.ko
kernel/drivers/connector/cn.ko
kernel/drivers/cpufreq/cpufreq_performance.ko
kernel/drivers/cpufreq/cpufreq_ondemand.ko
kernel/drivers/dma/virt-dma.ko
kernel/drivers/dma/hsu/hsu_dma.ko
kernel/drivers/firmware/dmi-sysfs.ko
kernel/drivers/hwmon/hwmon.ko
kernel/drivers/i2c/i2c-core.ko
kernel/drivers/input/input-core.ko
kernel/drivers/input/input-leds.ko
kernel/drivers/input/mousedev.ko
kernel/drivers/input/keyboard/atkbd.ko
kernel/drivers/input/misc/xen-kbdfront.ko
kernel/drivers/input/touchscreen/of_touchscreen.ko
kernel/drivers/input/serio/serio.ko
kernel/drivers/input/serio/i8042.ko
kernel/drivers/input/serio/libps2.ko
kernel/drivers/iommu/iova.ko
kernel/drivers/iommu/amd_iommu_v2.ko
kernel/drivers/leds/led-class.ko
kernel/drivers/macintosh/mac_hid.ko
kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko
kernel/drivers/pinctrl/intel/pinctrl-intel.ko
kernel/drivers/pinctrl/intel/pinctrl-broxton.ko
kernel/drivers/pinctrl/intel/pinctrl-sunrisepoint.ko
kernel/drivers/rtc/rtc-cmos.ko
kernel/drivers/thermal/thermal_sys.ko
kernel/drivers/tty/serial/serial_core.ko
kernel/drivers/tty/serial/8250/8250.ko
kernel/drivers/tty/serial/8250/8250_base.ko
kernel/drivers/tty/serial/8250/8250_pci.ko
kernel/drivers/tty/serial/8250/8250_dw.ko
kernel/drivers/tty/serial/8250/8250_mid.ko
kernel/drivers/video/backlight/backlight.ko
kernel/drivers/video/console/fbcon.ko
kernel/drivers/video/console/bitblit.ko
kernel/drivers/video/console/softcursor.ko
kernel/drivers/video/console/tileblit.ko
kernel/drivers/video/console/fbcon_rotate.ko
kernel/drivers/video/console/fbcon_cw.ko
kernel/drivers/video/console/fbcon_ud.ko
kernel/drivers/video/console/fbcon_ccw.ko
kernel/drivers/video/fbdev/xen-fbfront.ko
kernel/drivers/video/fbdev/core/fb.ko
kernel/drivers/video/fbdev/core/cfbfillrect.ko
kernel/drivers/video/fbdev/core/cfbcopyarea.ko
kernel/drivers/video/fbdev/core/cfbimgblt.ko
kernel/drivers/video/fbdev/core/sysfillrect.ko
kernel/drivers/video/fbdev/core/syscopyarea.ko
kernel/drivers/video/fbdev/core/sysimgblt.ko
kernel/drivers/video/fbdev/core/fb_sys_fops.ko
kernel/drivers/xen/xenbus/xenbus_probe_frontend.ko
kernel/arch/x86/video/fbdev.ko
kernel/net/802/fddi.ko
kernel/net/ipv4/tcp_cubic.ko
kernel/net/ipv6/ipv6.ko
kernel/net/ipv6/mip6.ko
kernel/net/ipv6/inet6_hashtables.ko
kernel/net/mpls/mpls_gso.ko
kernel/net/packet/af_packet.ko
kernel/net/unix/unix.ko
kernel/lib/bitrev.ko
kernel/lib/crc-t10dif.ko
kernel/lib/crc32.ko
kernel/lib/ucs2_string.ko
kernel/lib/fonts/font.ko
kernel/lib/lz4/lz4_decompress.ko
kernel/lib/lzo/lzo_compress.ko
kernel/lib/lzo/lzo_decompress.ko
kernel/lib/xz/xz_dec.ko
kernel/lib/zlib_deflate/zlib_deflate.ko
kernel/lib/zlib_inflate/zlib_inflate.ko


Then i can load modules by accessing them:

root@ip-10-99-134-230:~# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
root@ip-10-99-134-230:~# lsmod | grep ip_vs
ip_vs                 147456  0
libcrc32c              16384  1 ip_vs
nf_conntrack          114688  7 ip_vs,nf_conntrack_ipv4,nf_conntrack_netlink,nf_nat_masquerade_ipv4,xt_conntrack,nf_nat_ipv4,nf_nat

And then script output changes:

After accessing

root@ip-10-99-134-230:~# bash 1.sh
ip_vs
libcrc32c
xt_nat
xt_tcpudp
veth
vxlan
ip6_udp_tunnel
udp_tunnel
iptable_mangle
xt_mark
ipt_MASQUERADE
nf_nat_masquerade_ipv4
nf_conntrack_netlink
nfnetlink
xfrm_user
xfrm_algo
iptable_nat
nf_conntrack_ipv4
nf_defrag_ipv4
nf_nat_ipv4
xt_addrtype
iptable_filter
xt_conntrack
nf_nat
nf_conntrack
br_netfilter
bridge
stp
llc
overlay
intel_rapl
cirrus
sb_edac
edac_core
crct10dif_pclmul
crc32_pclmul
ghash_clmulni_intel
ttm
drm_kms_helper
drm
intel_rapl_perf
evdev
serio_raw
pcspkr
button
ip_tables
x_tables
autofs4
ext4
crc16
jbd2
crc32c_generic
fscrypto
ecb
mbcache
ata_generic
crc32c_intel
aesni_intel
xen_netfront
xen_blkfront
aes_x86_64
glue_helper
lrw
gf128mul
ablk_helper
cryptd
ata_piix
psmouse
i2c_piix4
libata
scsi_mod
floppy
-----
4.9.0-6-amd64
-----
kernel/arch/x86/ia32/ia32_aout.ko
kernel/arch/x86/platform/intel/iosf_mbi.ko
kernel/mm/zpool.ko
kernel/mm/zbud.ko
kernel/fs/binfmt_script.ko
kernel/fs/exportfs/exportfs.ko
kernel/fs/nls/nls_base.ko
kernel/fs/pstore/pstore.ko
kernel/crypto/crypto.ko
kernel/crypto/crypto_wq.ko
kernel/crypto/crypto_algapi.ko
kernel/crypto/aead.ko
kernel/crypto/crypto_blkcipher.ko
kernel/crypto/crypto_hash.ko
kernel/crypto/akcipher.ko
kernel/crypto/kpp.ko
kernel/crypto/cryptomgr.ko
kernel/crypto/hmac.ko
kernel/crypto/crypto_null.ko
kernel/crypto/md5.ko
kernel/crypto/sha1_generic.ko
kernel/crypto/sha256_generic.ko
kernel/crypto/aes_generic.ko
kernel/crypto/crct10dif_common.ko
kernel/crypto/crct10dif_generic.ko
kernel/crypto/lzo.ko
kernel/crypto/rng.ko
kernel/block/deadline-iosched.ko
kernel/block/cfq-iosched.ko
kernel/drivers/acpi/processor.ko
kernel/drivers/acpi/hed.ko
kernel/drivers/acpi/acpi_extlog.ko
kernel/drivers/base/firmware_class.ko
kernel/drivers/base/regmap/regmap-i2c.ko
kernel/drivers/base/regmap/regmap-spi.ko
kernel/drivers/char/agp/agpgart.ko
kernel/drivers/char/agp/amd64-agp.ko
kernel/drivers/char/agp/intel-agp.ko
kernel/drivers/char/agp/intel-gtt.ko
kernel/drivers/char/agp/sis-agp.ko
kernel/drivers/char/agp/via-agp.ko
kernel/drivers/char/tpm/tpm.ko
kernel/drivers/char/tpm/tpm_tis_core.ko
kernel/drivers/char/tpm/tpm_tis.ko
kernel/drivers/connector/cn.ko
kernel/drivers/cpufreq/cpufreq_performance.ko
kernel/drivers/cpufreq/cpufreq_ondemand.ko
kernel/drivers/dma/virt-dma.ko
kernel/drivers/dma/hsu/hsu_dma.ko
kernel/drivers/firmware/dmi-sysfs.ko
kernel/drivers/hwmon/hwmon.ko
kernel/drivers/i2c/i2c-core.ko
kernel/drivers/input/input-core.ko
kernel/drivers/input/input-leds.ko
kernel/drivers/input/mousedev.ko
kernel/drivers/input/keyboard/atkbd.ko
kernel/drivers/input/misc/xen-kbdfront.ko
kernel/drivers/input/touchscreen/of_touchscreen.ko
kernel/drivers/input/serio/serio.ko
kernel/drivers/input/serio/i8042.ko
kernel/drivers/input/serio/libps2.ko
kernel/drivers/iommu/iova.ko
kernel/drivers/iommu/amd_iommu_v2.ko
kernel/drivers/leds/led-class.ko
kernel/drivers/macintosh/mac_hid.ko
kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko
kernel/drivers/pinctrl/intel/pinctrl-intel.ko
kernel/drivers/pinctrl/intel/pinctrl-broxton.ko
kernel/drivers/pinctrl/intel/pinctrl-sunrisepoint.ko
kernel/drivers/rtc/rtc-cmos.ko
kernel/drivers/thermal/thermal_sys.ko
kernel/drivers/tty/serial/serial_core.ko
kernel/drivers/tty/serial/8250/8250.ko
kernel/drivers/tty/serial/8250/8250_base.ko
kernel/drivers/tty/serial/8250/8250_pci.ko
kernel/drivers/tty/serial/8250/8250_dw.ko
kernel/drivers/tty/serial/8250/8250_mid.ko
kernel/drivers/video/backlight/backlight.ko
kernel/drivers/video/console/fbcon.ko
kernel/drivers/video/console/bitblit.ko
kernel/drivers/video/console/softcursor.ko
kernel/drivers/video/console/tileblit.ko
kernel/drivers/video/console/fbcon_rotate.ko
kernel/drivers/video/console/fbcon_cw.ko
kernel/drivers/video/console/fbcon_ud.ko
kernel/drivers/video/console/fbcon_ccw.ko
kernel/drivers/video/fbdev/xen-fbfront.ko
kernel/drivers/video/fbdev/core/fb.ko
kernel/drivers/video/fbdev/core/cfbfillrect.ko
kernel/drivers/video/fbdev/core/cfbcopyarea.ko
kernel/drivers/video/fbdev/core/cfbimgblt.ko
kernel/drivers/video/fbdev/core/sysfillrect.ko
kernel/drivers/video/fbdev/core/syscopyarea.ko
kernel/drivers/video/fbdev/core/sysimgblt.ko
kernel/drivers/video/fbdev/core/fb_sys_fops.ko
kernel/drivers/xen/xenbus/xenbus_probe_frontend.ko
kernel/arch/x86/video/fbdev.ko
kernel/net/802/fddi.ko
kernel/net/ipv4/tcp_cubic.ko
kernel/net/ipv6/ipv6.ko
kernel/net/ipv6/mip6.ko
kernel/net/ipv6/inet6_hashtables.ko
kernel/net/mpls/mpls_gso.ko
kernel/net/packet/af_packet.ko
kernel/net/unix/unix.ko
kernel/lib/bitrev.ko
kernel/lib/crc-t10dif.ko
kernel/lib/crc32.ko
kernel/lib/ucs2_string.ko
kernel/lib/fonts/font.ko
kernel/lib/lz4/lz4_decompress.ko
kernel/lib/lzo/lzo_compress.ko
kernel/lib/lzo/lzo_decompress.ko
kernel/lib/xz/xz_dec.ko
kernel/lib/zlib_deflate/zlib_deflate.ko
kernel/lib/zlib_inflate/zlib_inflate.ko

It is plain debian stretch with apt install ipvsadm

sepich picture sepich on 6 Jul 2018

the validation code requires the following modules to be present:
ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, nf_conntrack_ipv4

you have ip_vs and nf_conntrack_ipv4 but the rest are missing.

i do not understand these missing modules and IPVS in general.

the validation was added here:
https://github.com/kubernetes/kubernetes/pull/59566

/cc @stewart-yu @luxas
/cc @kubernetes/sig-cluster-lifecycle-bugs
/cc @kubernetes/sig-node-bugs

neolit123 picture neolit123 on 6 Jul 2018
👍3

cc @m1093782566 @stewart-yu

dims picture dims on 6 Jul 2018

Does your cluster still using ipvs proxy, even that occur thoes warning messages? @sepich
IMO, those warning are pre-check when you use kubeadm to creating cluster. But actually, all ipvs modules will loaded by kube-proxy, please see :https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go#L469

stewart-yu picture stewart-yu on 7 Jul 2018

Does your cluster still using ipvs proxy, even that occur thoes warning messages?

Yes, i see Services in ipvsadm

those warning are pre-check when you use kubeadm to creating cluster. But actually, all ipvs modules will loaded by kube-proxy

And that is exactly reason it seems confusing - why this warning is needed then?

sepich picture sepich on 7 Jul 2018

And that is exactly reason it seems confusing - why this warning is needed then?

i guess a preflight warning about missing critical IPVS modules is nice to have - i.e. if required module is missing on the system.
but then it becomes a question of what modules are really required?

this logic here:
https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go#L472
throws a warning by the proxier on missing modules from this list:
https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go#L157

neolit123 picture neolit123 on 7 Jul 2018

but then it becomes a question of what modules are really required?

Agree, we should figure it out and I'd like to file a PR to fix this. :)

xlgao-zju picture xlgao-zju on 11 Jul 2018

i would hold on this @xlgao-zju , because even the recent blog post about IPVS included the whole list of modules that is discussed in this thread. i reviewed the blog before it was push, but to my understanding the list in there is the defacto valid list.

https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/
^ @timothysc

what we don't know here is why some of the modules are not required and why it works for @sepich without them. @stewart-yu do you know why? did you see the above blog post too?

are these modules optional?:
ip_vs_rr, ip_vs_wrr, ip_vs_sh

neolit123 picture neolit123 on 11 Jul 2018

I see the same warning when joining a new node to the cluster. None of the modules are loaded into the kernel to start with, but after running kubeadm join, they are all there:

Before:

# lsmod | grep ^ip_vs
(nothing)

After:

# lsmod | grep ^ip_vs
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 147456  6 ip_vs_wrr,ip_vs_rr,ip_vs_sh

Is it a case of the preflight check not understanding that modules can be dynamically loaded when needed?

I'm running v1.11.0 from the apt repo on Ubuntu 16.04.

cu picture cu on 17 Jul 2018
👍3

Hi, I have the same problem with my 4.15.18-7-pve kernel.

Modules existing, loaded and working fine:

# modprobe -- br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4
# lsmod | grep '^\(ip_vs\|ip_vs_rr\|ip_vs_wrr\|ip_vs_sh\|nf_conntrack_ipv4\)'
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  16
ip_vs                 151552  22 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack_ipv4      16384  7

even after that I have this warning:

[preflight] running pre-flight checks
 [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

kube-proxy uses iptables mode by default, but if I forcefully set parameter mode: "ipvs" in it's configmap it is start using ipvs without any problems.

According this PR I found out that modules are checking inside modules.builtin file. But my one have no records for those modules:

# kernelVersion=$(cat /proc/sys/kernel/osrelease)
# cat /lib/modules/${kernelVersion}/modules.builtin | grep ipvs
<nothing>

As can I see those modules existing in my modules path

# find /lib/modules/${kernelVersion} -name ip_vs.ko -o -name ip_vs_rr.ko -o -name ip_vs_wrr.ko -o -name ip_vs_sh.ko -o -name nf_conntrack_ipv4.ko
/lib/modules/4.15.18-7-pve/kernel/net/ipv4/netfilter/nf_conntrack_ipv4.ko
/lib/modules/4.15.18-7-pve/kernel/net/netfilter/ipvs/ip_vs.ko
/lib/modules/4.15.18-7-pve/kernel/net/netfilter/ipvs/ip_vs_rr.ko
/lib/modules/4.15.18-7-pve/kernel/net/netfilter/ipvs/ip_vs_sh.ko
/lib/modules/4.15.18-7-pve/kernel/net/netfilter/ipvs/ip_vs_wrr.ko

Verdict:

  • ipvs modules existing in my system, I can use them, but they're not described in modules.builtin.
  • Kubernetes check for IPVS is wrong, because saying that modules not accessible.
  • Targeted components: kubeadm and kube-proxy
kvaps picture kvaps on 18 Oct 2018
👍1

Please, review my attempt to fix this issue: using 'modinfo ' to check if required module is installed.

bart0sh picture bart0sh on 20 Oct 2018

@kvaps
https://github.com/kubernetes/kubernetes/pull/63850 fixes https://github.com/kubernetes/kubernetes/issues/63801. The added check is just an OR operation. I suggest you to read that part of code. Where the modules are placed varies on operation systems and releases. Some are in builtin.modules and some in kernel/net/netfilter/ipvs/.
I think the kube-proxy works fine on this check.

islinwb picture islinwb on 23 Oct 2018
👍1

@islinwb, seems there is same problem, it's checking modules only inside modules.builtin file, and this check is not working in my case .

kvaps picture kvaps on 23 Oct 2018

@kvaps
run 

cut -f1 -d "" /proc/modules | grep ipvs

in your machine have a try?

stewart-yu picture stewart-yu on 23 Oct 2018

@kvaps, Can anybody check if this PR fixes the issue?

bart0sh picture bart0sh on 23 Oct 2018

@stewart-yu , sure

# cut -f1 -d " " /proc/modules | grep ip_vs
ip_vs_sh
ip_vs_wrr
ip_vs_rr
ip_vs
kvaps picture kvaps on 23 Oct 2018

In word, RequiredIPVSKernelModulesAvailable() first part : https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L47-L59 will check the kernel modules

var ipvsModules = []string{
    "ip_vs",
    "ip_vs_rr",
    "ip_vs_wrr",
    "ip_vs_sh",
    "nf_conntrack_ipv4",
}

If missing any kerner modules, https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L62 the condition are true, we will check if exists builtin modules :https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L63-L84. If builtin modules missing, we throw warning : https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L85-L90
So, if all kernel modules in your machine, this condition in https://github.com/kubernetes/kubernetes/blob/master/pkg/util/ipvs/kernelcheck_linux.go#L62 are false, can not throwwarning

stewart-yu picture stewart-yu on 23 Oct 2018

So missing nf_conntrack_ipv4, please install it, and have a try @kvaps

stewart-yu picture stewart-yu on 23 Oct 2018

It's not missing:

cut -f1 -d " " /proc/modules | grep nf_conntrack_ipv4

you just asked me for grep only ipvs string before

kvaps picture kvaps on 23 Oct 2018
👍1

@kvaps, Can anybody check if this PR fixes the issue?

I'm working on that

kvaps picture kvaps on 23 Oct 2018

What happened is clear:

  1. kubeadm first checks the _loaded_ modules;
    and if ipvs modules are not all loaded, it will check the builtin modules;
  2. kube-proxy checks builtin modules;
    and also loads modules, then check if they are loaded.

The kubeadm doesn't automatically load these modules.
If you load these modules manually, then the kubeadm check should pass.

islinwb picture islinwb on 23 Oct 2018

From your waring describe in https://github.com/kubernetes/kubeadm/issues/975#issuecomment-430990149, all kernel module missing (the second line)

[preflight] running pre-flight checks
 [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

But you install all kernel module first, so strange

stewart-yu picture stewart-yu on 23 Oct 2018

It's not missing:

cut -f1 -d " " /proc/modules | grep nf_conntrack_ipv4

you just asked me for grep only ipvs string before

sorry, my mistakes

stewart-yu picture stewart-yu on 23 Oct 2018

Here is my take on it:
Current code of RequiredIPVSKernelModulesAvailable check doesn't check if modules are installed. It only checks if they're loaded or builtin. This confuses users as even if modules are not loaded and not builtin kube-proxy can load them if they are installed. The check forces users to manually load modules, which is not needed as kube-proxy can do it.

bart0sh picture bart0sh on 23 Oct 2018

Here is my take on it:
Current code of RequiredIPVSKernelModulesAvailable check doesn't check if modules are installed. It only checks if they're loaded or builtin. This confuses users as even if modules are not loaded and not builtin kube-proxy can load them if they are installed. The check forces users to manually load modules, which is not needed as kube-proxy can do it.

yes, you are right :+1:

stewart-yu picture stewart-yu on 23 Oct 2018

@islinwb @stewart-yu My bad, I was loading modules wrong, using this command:

modprobe ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4

This is wrong, because in this case modprobe reads only first argument, and return no errors. This behavior is different unlike rmmod which removes all modules from it's arguments.

If I do:

for i in ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4; do modprobe $i; done

all modules are loaded fine, and there is no warning from kubeadm, but after I join, kube-proxy still not uses ipvs by default:

1 server_others.go:295] Flag proxy-mode="" unknown, assuming iptables proxy
1 server_others.go:148] Using iptables Proxier.

My kube-proxy config really have mode: "", if I switch it to mode: "ipvs", then ipvs is working fine. And even if modules are not loaded, it is loading them automatically.

Is that normal behavior? - In my opinion not, because in case when mode is not set it should use ipvs by default, and only if it's not accessible it should fallback to iptables, am I right?

@bart0sh I still need little bit time for testing your PR

kvaps picture kvaps on 23 Oct 2018

@kvaps

Seems you got the key :)

Please keep in mind that ipvs mode is still not the default although it's already GA. So, you should specify the "ipvs" mode in your kube-proxy config file.

m1093782566 picture m1093782566 on 23 Oct 2018
👍1

@bart0sh I've tried your patch, it's working fine for me, no warning in kubeadm even if modules unloaded in my system. Cheers!

@m1093782566 thanks for clarification

kvaps picture kvaps on 23 Oct 2018

@kvaps Thank you for testing it!

bart0sh picture bart0sh on 23 Oct 2018

@islinwb @stewart-yu My bad, I was loading modules wrong, using this command:

modprobe ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4

This is wrong, because in this case modprobe reads only first argument, and return no errors. This behavior is different unlike rmmod which removes all modules from it's arguments.

If I do:

for i in ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4; do modprobe $i; done

all modules are loaded fine, and there is no warning from kubeadm, but after I join, kube-proxy still not uses ipvs by default:

1 server_others.go:295] Flag proxy-mode="" unknown, assuming iptables proxy
1 server_others.go:148] Using iptables Proxier.

My kube-proxy config really have mode: "", if I switch it to mode: "ipvs", then ipvs is working fine. And even if modules are not loaded, it is loading them automatically.

Is that normal behavior? - In my opinion not, because in case when mode is not set it should use ipvs by default, and only if it's not accessible it should fallback to iptables, am I right?

@bart0sh I still need little bit time for testing your PR

ipvs kube-proxy is not default model. so the behavior using iptables is normally

stewart-yu picture stewart-yu on 9 Nov 2018

IMO, the warning messages not confused, just throw warning, and not auto-installed modules for user. That's OK now.
If ipvs became default kube-proxy model someday, we can add code about auto-installed modules.
BTW, some disscuss in the issuse, may just related to local ENV, not about RequiredIPVSKernelModulesAvailable :slightly_smiling_face:

stewart-yu picture stewart-yu on 9 Nov 2018

IMO, the warning messages not confused, just throw warning, and not auto-installed modules for user. That's OK now.

the warnings message says "the IPVS proxier will not be used", which is 100% incorrect. It can be used as you've just pointed out. It is confusing.

bart0sh picture bart0sh on 21 Nov 2018
👍2

FYI, I am here because of the "...IPVS proxier will not be used..." message during kubeadm join .... and it is confusing. But, wouldn't just changing the message to something like the following help clear things up?

[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier may not be used because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support was found: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]. However, these modules may be loaded automatically by kube-proxy for you if they are available on your system.
To verify IPVS support:

  1. Run lsmod | grep 'ip_vs\|nf_conntrack' and verify each of the above modules are listed.

If they are not listed, you can use the following methods to load them:

  1. For each missing module run 'modprobe $modulename' (e.g., 'modprobe ip_vs', 'modprobe ip_vs_rr', ...)
  2. If 'modprobe $modulename' returns an error, you will need to install the missing module support for your kernel. (see http://linktosomepagewithhelponinstalling.foo)
keithdadkins picture keithdadkins on 2 Dec 2018

@keithdadkins thank you for the less confusing warning message! I updated it as you've suggested in this PR

bart0sh picture bart0sh on 13 Feb 2019
👍2

/lifecycle active

bart0sh picture bart0sh on 15 Feb 2019
👍2

I'm still hoping to get this issue properly fixed. Will update the fix soon. Feel free to reopen this issue if you think it's not fixed yet.

bart0sh picture bart0sh on 5 Mar 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ggee picture ggee  路  4Comments
shenshouer picture shenshouer  路  3Comments
luxas picture luxas  路  3Comments
helphi picture helphi  路  3Comments
RakeshNagarajan picture RakeshNagarajan  路  4Comments