Kube-router: Network policies are applied for pods that are not running (completed jobs)

[zerkms]

What happened?

I deployed an application and found out that I cannot connect its ports. All looked fine but still - connection was actively refused. After some debugging I have found that on that node there are 2 pods with the same IP assigned: one is the pod I'm talking about, and the other is a pod from a job that has successfully completed 15 hours ago.

Both have 10.51.5.17 IP assigned.

The job does have network policies configured, the application I just deployed - does not. Yet, there are iptables rules applied for the job still, even though it's completed.

Chain KUBE-POD-FW-AIWVY7TMPWP7PXQH (3 references)
    pkts      bytes target     prot opt in     out     source               destination
    3087   201342 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* rule for stateful firewall for pod */ ctstate RELATED,ESTABLISHED
       0        0 ACCEPT     all  --  *      *       0.0.0.0/0            10.51.5.17           /* rule to permit the traffic traffic to pods when source is the pod's local node */ ADDRTYPE match src-type LOCAL
       0        0 KUBE-NWPLCY-SB6L4XPQ7W7HDE22  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* run through nw policy default-ingress-deny */
       0        0 NFLOG      all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* rule to log dropped traffic POD name:backup-sync-adm-1617704100-hmrzs namespace: adm-backup-hq */ mark match ! 0x10000/0x10000 limit: avg 10/min burst 10 nflog-group 100
       0        0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* rule to REJECT traffic destined for POD name:backup-sync-adm-1617704100-hmrzs namespace: adm-backup-hq */ mark match ! 0x10000/0x10000 reject-with icmp-port-unreachable
       0        0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK and 0xfffeffff

What did you expect to happen?

I suspect that if assigning the same IP is possible - then rules from the stopped pod should be cleaned up.

How can we reproduce the behavior you experienced?
Steps to reproduce the behavior:

1. Schedule enough jobs on the same node so that IPs wrapped around and started from the beginning, also - make sure you apply some network policy in those jobs namespace
2. Now in different namespace deploy something else, and ensure the the just scheduled pod gets the same IP as one of previously running jobs
3. iptables -L -nvx --- see the iptables rules are still there

*Screenshots / Architecture Diagrams / Network Topologies *
If applicable, add those here to help explain your problem.

* System Information (please complete the following information):*

• Kube-Router Version (kube-router --version): 1.1.1
• Kube-Router Parameters: [e.g. --run-router --run-service-proxy --enable-overlay --overlay-type=full etc.]

--run-router=true
--run-firewall=true
--run-service-proxy=true
--bgp-graceful-restart=true
--kubeconfig=/var/lib/kube-router/kubeconfig
--hairpin-mode=true

• Kubernetes Version (kubectl version) : 1.19.6
• Cloud Type: [e.g. AWS, GCP, Azure, on premise] baremetal
• Kubernetes Deployment Type: [e.g. EKS, GKE, Kops, Kubeadm, etc.] kubeadm
• Kube-Router Deployment Type: [e.g. DaemonSet, System Service] daemonset
• Cluster Size: [e.g. 200 Nodes] 15

* Logs, other output, metrics *
Please provide logs, other kind of output or observed metrics here.

Additional context
Add any other context about the problem here.

[aauren]

We've just made some significant changes to network policy which should have an impact on this bug. Can you please see if you can reproduce with kube-router version 1.2.1?

If you can, please respond here and I'll re-open this issue.

[chendotjs]

/reopen

still encounter this problem with 1.2.2-dirty

version info:

➜  ~ docker run -it --entrypoint sh docker.io/cloudnativelabs/kube-router
~ # /usr/local/bin/kube-router -V
Running /usr/local/bin/kube-router version v1.2.2-dirty, built on 2021-04-20T15:56:16+0000, go1.15.8

[aauren]

Before re-opening this, it would be helpful to have more information. Can you provide what the environment looks like when the error happened?

The following things would be helpful:

• a view of the pod and its network policy (even just querying for the blank policy is fine)
• a view of the iptables chains that show the new rules (which includes the KUBE-POD-FW and KUBE-NWPLCY chains for the associated pod in full
• a view of the network policy that was applied previously to the pod that used to own the IP address
• the full kube-router log around the time of the incident
• a sense of timing in terms of when the previous pod was scheduled to when the new pod (containing the old pod's IP address) was scheduled
• can you also share the pod network subnet mask? Both the entire pod network, and the subnet that you provision to nodes?

[aauren]

Another helpful piece of information is whether or not the chain sticks around in iptables when the pod is terminated? Or if it only shows up when a new pod that re-uses an IP is scheduled.

[zerkms]

And fyi: I have not upgraded kube-router yet (I do it probably every several months, and the timeline hasn't reached yet), so I cannot tell if I can reproduce it either, sorry.

@aauren but if there is a docker image available to test this + MTU fix I definitely could do both in one shot.

[zerkms]

Okay, I have installed the https://hub.docker.com/layers/150186454/cloudnativelabs/kube-router-git/amd64-fix_multiple_reductions_in_mtu/images/sha256-d1516a1f45eac43e613944cc5e9975bd3ef94778961fa3592a3d4492e7c917b2?context=explore image.

This bug is not fixed: the iptables rules are not cleaned up after the pod completes. Manifests:

kind: Namespace
apiVersion: v1
metadata:
  name: test-iptables
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-ingress-deny
  namespace: test-iptables
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
kind: Job
apiVersion: batch/v1
metadata:
  name: test-job
  namespace: test-iptables
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: script
          image: ubuntu:focal
          command:
            - /bin/bash
            - -c
            - |-
              set -eux
              set -o pipefail

              sleep 30

Then while the container runs:

# iptables -L -nvx |grep 134
       0        0 ACCEPT     all  --  *      *       0.0.0.0/0            10.53.68.134         /* rule to permit the traffic traffic to pods when source is the pod's local node */ ADDRTYPE match src-type LOCAL
       0        0 KUBE-POD-FW-ACOT6K7HVF7NBHBA  all  --  *      *       0.0.0.0/0            10.53.68.134         PHYSDEV match --physdev-is-bridged /* rule to jump traffic destined to POD name:test-job-hdf52 namespace: test-iptables to chain KUBE-POD-FW-ACOT6K7HVF7NBHBA */
       0        0 KUBE-POD-FW-ACOT6K7HVF7NBHBA  all  --  *      *       0.0.0.0/0            10.53.68.134         /* rule to jump traffic destined to POD name:test-job-hdf52 namespace: test-iptables to chain KUBE-POD-FW-ACOT6K7HVF7NBHBA */
       0        0 KUBE-POD-FW-ACOT6K7HVF7NBHBA  all  --  *      *       0.0.0.0/0            10.53.68.134         /* rule to jump traffic destined to POD name:test-job-hdf52 namespace: test-iptables to chain KUBE-POD-FW-ACOT6K7HVF7NBHBA */

should show as expected - rules to be present. But after job completest - these rules must be removed as well.

Can it please be reopened.

I have not tested changes in the MTU for tunnels yet.

[chendotjs]

Thanks for responses @zerkms @aauren

I am not sure whether kube-router should check Pod phase like calico does here ?

Hope this can help.

[aauren]

@chendotjs that is helpful, thanks for the links.

I'm actually working on a separate PR to improve the efficiency of pod handling within the NetworkPolicy controller in kube-router. This work can be rolled into that. While pod status phase is something that we look for currently, I doubt that we are handling all of the possible stages well.

I'll take a look at that early next week.