Kube-router: use xtables.lock

[Daryltp]

When deploying networkpolicies to two namespaces in the same cluster on Azure Kubernetes Service simultaneously we have observed the following error. It appears to be due to resource contention on the iptables command. The issue is that both deployments will complete "successfully" but the traffic flows the policy allows will fail and the error is only apparent when you view the logs of the kube-router pod/s.

E0801 10:27:45.538722       1 network_policy_controller.go:182] Error syncing network policy for the update to network policy: <NAMESPACE_NAME>/<POLICY_NAME> Error: Aborting sync. Failed to sync network policy chains: Failed to run iptables command: running [/sbin/iptables -t filter -N KUBE-NWPLCY-YYN4V3F74PSONPOB --wait]: exit status 4: iptables: Resource temporarily unavailable.

[murali-reddy]

@Daryltp thanks for reporting the issue. kube-router need to hold xtables.lock before trying to run iptables commands

kube-router manifests need to mount xtables.lock, go-iptables library that kube-router uses can lock to prevent concurrency issues.

[serbrech]

I took a quick look at this.
the iptables are being updated through the go-iptables package in the different controllers :

• network_routes_controller
• network_services_controller
• pod_egress

we cannot guarantee that other services running on the host would not try to update the iptables as at the same time too.

Would the file lock implementation from the go-iptable package work across containers?

What would be the prefered approach in kube-router?
Take the lock in each controller individually? or have a shared lock at the base of the service that the different controller use?
It seems that the go-iptable package does a no-op when the file is already locked (would block). How should that be handled?

[serbrech]

I see now that the go-iptables library always runs with --wait, so would mounting the xtables.lock file be enough to resolve this?

-w, --wait
Wait for the xtables lock. To prevent multiple instances of the program from running concurrently, an attempt will be made to obtain an exclusive lock at launch. By default, the program will exit if the lock cannot be obtained. This option will make the program wait until the exclusive lock can be obtained.

[roffe]

I've added the following to my kube-router deployments to allows usage of xtables

        volumeMounts:
        ...
        - mountPath: /var/run/xtables.lock
          name: xtables
      volumes:
      ...
      - name: xtables
        hostPath:
          path: /run/xtables.lock

[soumeng09]

The volume mount doesn't really solve the issue.
We see kube-router stuck in "controller still performing bootup full-sync" loop for a while then restarts with messages "Network Policy Controller heartbeat missed" and "Shutting down the controllers".

We also see kube-proxy on the same node giving errors like -
"Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 4: Another app is currently holding the xtables lock. Stopped waiting after 5s."

From the messages it looks like a race condition between kube-proxy and kube-router on iptables lock.

We've been seeing this issue persistently in our clusters (AKS cluster with around 20 nodes).

[murali-reddy]

We see kube-router stuck in "controller still performing bootup full-sync" loop for a while then restarts with messages "Network Policy Controller heartbeat missed" and "Shutting down the controllers".

@soumeng09 can you please share kube-router logs? Lock is held by iptables command just for the duration when command is run. There is possibility of race condition but both kube-router and kube-proxy launches iptables command with -w option. So there is less chance of failing to execute.

It woule be good to look at the both kube-router and kube-proxy logs to understand what sort of contention is happening. Also is it possible that some other process is holding up the lock?