Lokomotive: Expose --conntrack-max-per-core kube-proxy flag

[johananl]

kube-proxy runs as a privileged DaemonSet and, among the rest, configures the net.netfilter.nf_conntrack_max kernel parameter at run time. This parameter determines the size of the netfilter conntrack table. Exhausting this table has an adverse effect on traffic, which means that on nodes which handle a lot of connections this parameter may need to be tuned.

The --conntrack-max-per-core kube-proxy flag controls the value which kube-proxy sets in the kernel. The default value is 32768, which means that by default kube-proxy sets net.netfilter.nf_conntrack_max to 32768 * cpu_cores.

We have seen use cases where the default is not enough. Therefore, we should consider exposing this flag in the cluster configuration.

To test this functionality, setting --conntrack-max-per-core to 100000 and running sysctl net.netfilter.nf_conntrack_max e.g. on a 48-core machine should give the following output:

net.netfilter.nf_conntrack_max = 4800000

One thing to note is that setting the flag on kube-proxy will affect all node pools so we should consider whether that makes sense.

[invidian]

Maybe we could set that with ignition instead of kube-proxy? would it make sense?

[surajssd]

But using ignition means everytime you want to tweak the config you have to rebuild the worker pool.

[johananl]

I think the fastest/easiest solution would be to add the flag using Helm. As mentioned above this creates a flexibility problem in case we want to use a different value for different worker pools, however at least it can cover some of the use cases. For example, as long as the memory impact isn't too big one could set the global value to the maximum needed in the cluster.

Ideally we should be able to control this at the worker pool level, however AFAIK we don't have a mechanism for per-pool control plane config right now and I'd rather avoid forcing worker replacement until we have a faster provisioning time on Packet and better automation for minimizing downtime.

[johananl]

We've received user feedback about some cases where it's preferable to have a per-pool setting even if it means replacing nodes. I think we could actually have both: We add the global knob which affects kube-proxy, and in addition we document how to set the kernel param via CLC snippets on a per-pool level.

Some thoughts which occurred to me:

• We might want to quantify the implication of setting a global value on memory consumption. If it's negligible, maybe it makes sense to set a global value based on the maximum value required for the most traffic-intensive pool in a cluster.
• If we document per-pool config via CLC snippets, we should verify that the kube-proxy flag doesn't override the host-level setting (kube-proxy may tune the kernel params by default, even when no related flag is specified in the args).

[surajssd]

+1 for both.

[invidian]

Created PR to address this: https://github.com/kinvolk/lokomotive/pull/1187.