Kind: [WSL2] Sync failed errors in kube-proxy for Service with SessionAffinity: ClientIP

[anyname2]

What happened:
iptables fail to be updated on the nodes after a Service with sessionAffinity: ClientIP is created.
The issue manifests in requests beeing dropped to any Services that were created after the Service with session affinity.

kube-proxy pod is logging the following error:

E0720 14:29:10.934607       1 proxier.go:1507] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.8.3 (legacy): Couldn't load match `recent':No such file or directory

Error occurred at line: 96
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
)
I0720 14:29:10.934636       1 proxier.go:779] Sync failed; retrying in 30s

What you expected to happen:
iptables to be updated correctly so that requests could be routed to any Service in the cluster.

How to reproduce it (as minimally and precisely as possible):
Create a Service with sessionAffinity: ClientIP

apiVersion: v1
kind: Service
metadata:
  labels:
    alertmanager: main
  name: alertmanager-main
  namespace: monitoring
spec:
  ports:
  - name: web
    port: 9093
    targetPort: web
  selector:
    alertmanager: main
    app: alertmanager
  sessionAffinity: ClientIP

Anything else we need to know?:
Issue is reproducible with both kubeProxyMode: iptables (default) and kubeProxyMode: ipvs

Environment:

• kind version:
  
  
  
  • kind v0.8.1 go1.13.8 linux/amd64
  
  
  • kind v0.9.0-alpha+95753c11434213 go1.15beta1 linux/amd64
  
  
• Kubernetes version:
  
  
  
  • Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5", GitCommit:"e0fccafd69541e3750d460ba0f9743b90336f24f", GitTreeState:"clean", BuildDate:"2020-05-01T02:11:15Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  
  
  • also tried v1.18.2 (default with kind v0.8.1) and v1.18.6 (default with kind v0.9.0-alpha)
  
  
• Docker version: Docker Desktop with WSL2

Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.104-microsoft-standard
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 25GiB
 Name: docker-desktop
 ID: D4I2:L4Y5:PGPS:CEUY:H3TU:C33L:HASQ:VZKB:53SE:SHQG:OOQV:BZMQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 48
  Goroutines: 57
  System Time: 2020-07-20T15:35:34.5632783Z
  EventsListeners: 3
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

• OS: Windows 10 (Build: 19041.388)

[aojea]

hmm, I think that is missing one kernel module, If I'm correct it should be xt_recent
@PatrickLang you are the WSL2 expert, how is possible to include this module?

[BenTheElder]

kind is not going to mess with your kernel modules so bug => support

If docker desktop is missing a module, that's probably hard to fix as an end user, but they might be willing to seeing as they also offer running the docker desktop VM as a single fixed-version kubernetes node instead of just dockerd.

[BenTheElder]

/kind external

[tallaxes]

Yes, it looks like the current WSL2 Kernel is built without xt_recent, needed by iptables -m recent ... which kube-proxy uses to implement sessionAffinity: ClientIP. Custom Kernel built with CONFIG_NETFILTER_XT_MATCH_RECENT=y fixed it for me. Submitted https://github.com/microsoft/WSL2-Linux-Kernel/pull/198 (4.19.y) and https://github.com/microsoft/WSL2-Linux-Kernel/pull/199 (5.4.y)

[BenTheElder]

thanks @tallaxes !

[WSLUser]

If someone wants to compile the latest 5.8 kernel instead for WSL2 with this option enabled, take a look here https://github.com/WSLUser/WSL2-Linux-Kernel/blob/WSLUser-5.8.5-config/Microsoft/config-wsl. I too have a PR but my config is based on 5.8.7 currently and will soon be updated to 5.8.12. Follow https://wsl.dev/wsl2-kernel-zfs/ for steps for compiling your own kernel.

[hawk29]

CONFIG_NETFILTER_XT_MATCH_RECENT=y

I am sorry but a newbie question. I have come across the same issue using docker-desktop. I have downloaded and installed the latest docker-desktop but to no avail. Is there a release where this will be embedded for end-users or do we have to compile on our own?

Client:
Debug Mode: false
Plugins:
scan: Docker Scan (Docker Inc., v0.3.4)

Server:
Containers: 86
Running: 80
Paused: 0
Stopped: 6
Images: 24
Server Version: 19.03.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.128-microsoft-standard
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 24.77GiB
Name: docker-desktop
ID: 4IEZ:4LGJ:N7EI:P4FA:XJYC:5TTB:X7HG:FCPV:BTFP:YTO2:M75E:QDKH
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[tallaxes]

@hawk29 - that would be a question to WSL2 maintainers; as far as I can tell it is not included in any recent releases. (And I don't see _any_ PR merging activity at microsoft/WSL2-Linux-Kernel - so maybe they just don't accept contributions ...)

FWIW, in tallaxes/WSL2-Linux-Kernel fork I have configured GitHub Action to build it, so you should be able to get built Kernel image from there - without worrying about downloading/running "mystery meat" bits - since the build process is transparent. The Kernel image is captured as build artifact - click on build run, scroll to Artifacts, look for bzImage. Then follow instructions for configuring global options in .wslconfig, setting kernel key to point to the custom kernel. (Obviously, use at your own risk, #include <disclamer.h> ...)