Kubeadm: mutable upgrades: upgrade the kube-proxy DS after all control-plane components

[neolit123]

during mutable upgrades, upgrade kube-proxy only after the kube-apiserver on all nodes have been upgraded.

currently kubeadm does the following:

• upgrades the control-plane on one node (via "apply")
• upgrades kube-proxy and coredns
• upgrades the rest of the control-plane nodes and worker nodes
• upgrades kubelets everywhere

the problem with that is that it can create issues for a period of time when kube-proxy's version is ahead of a kube-apiserver version in an HA setup.

https://kubernetes.io/docs/setup/release/version-skew-policy/#kube-proxy

there is also another problem where kube-proxy would be updated before the kubelets on nodes and the policy suggests they should match.

this would require:

• "apply" phases so that upgrades of the addons are performed last after kube-* control-plane components are upgraded:
  https://github.com/kubernetes/kubeadm/issues/1318
• changes in the upgrade documentation:
  https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

[neolit123]

cc @fabriziopandini

[yvespp]

Don't know how hard it would be but could the proxy maybe be converted to a static pod manifest to solve this?
The update of it could then be moved from kubeadm upgrade apply <vesions> on the master to kubeadm upgrade node.

1. Remove manifest of proxy to stop it
2. Stop kubelet
3. Update config and manifest
4. Start kubelet

This should also play nicely with immutable nodes.

[neolit123]

having kube-proxy as a static pod would definitely help around the fact kubelet and kube-proxy versions need to be in sync during upgrade.

there are pros on cons of both DS vs static pods. in general we are likely to see users preferring to run replicas of kube-proxy on all nodes (ala DS), and for multi-platform support variants of the DS can be deployed, which is what the kubeadm for Windows deployment is doing today.

this also opens some questions around instance specific configuration, and how to manage being possible to configure kube-proxy differently on different nodes:
https://github.com/kubernetes/enhancements/pull/1439

at this point however it is quite the breaking change to user expectations - e.g. if they have patches for the podspec of the DS to pass extra flags or modify pod resources. wondering if we are pass the point of changing this.

This should also play nicely with immutable nodes.

cluster-api discussion from today that mentions nodeselectors / labels and multiple DSes as one way of solving the problems for immutable upgrades:
https://kubernetes.slack.com/archives/C8TSNPY4T/p1604950077231800

[fabriziopandini]

@neolit123 thanks for raising this point.
I think that we should focus on solutions that are not making the user experience more complex, by adding more steps to the upgrade sequence.

If I got this right v of kube-proxy is f(v of API server, v of Kubelet on the node), and so I'm kind of intrigued about the idea of transforming kube-proxy into a node-specific manifest because:

• This allows having "self-consistent" control plane nodes, which is a requirement for mutable upgrades.
• This makes the workers less sensible while upgrading (this can take some time in a big cluster).

This is something that should be discussed with focus to:

• Transitioning current cluster to the new model
• Possible additional maintenance burden due to the new model.

WRT to the latest point, the idea of an operator managing automatically the merge of cluster level and node level setting would be a perfect fit for preserving a nice user experience no matter of how kube-proxy is deployed....

[neolit123]

Ok given there is interest in the ststic pod approach we should raise it
for discussion in the next sig meeting. Its tuesday next week.

[randomvariable]

Just to return to this, have been reviewing the Draft STIG for Kubernetes, and CNTR-K8-000440 states the following:

Kubernetes kubelet static PodPath must not enable static pods.
with check content:
"On the Master and Worker nodes, change to the /etc/sysconfig/ directory and run the command:
grep -i staticPodPath kubelet
If any of the nodes return a value for staticPodPath, this is a finding."

FWIW, I have requested that we give feedback for the following:

1. Change the offensive language
2. Add a kubeadm compatible check that greps the /var/lib/kublet/config.yaml
3. Remove the text for control plane nodes, as other bits of the document explicitly expect the control plane components to be run as static pods.

However, I can understand that they will want to stand by not enabling static pods on nodes, which means we may not want to pursue converting the kube-proxy deployment to a static pod.

[neolit123]

Just to return to this, have been reviewing the Draft STIG for Kubernetes, and CNTR-K8-000440 states the following:

not enabling static pods at all is disagreement with the k8s project maintainers basically.
i'm not familiar with this website and their work.

• Change the offensive language
• Add a kubeadm compatible check that greps the /var/lib/kublet/config.yaml
• Remove the text for control plane nodes, as other bits of the document explicitly expect the control plane components to be run as static pods.

are these requested changes for them to do?

However, I can understand that they will want to stand by not enabling static pods on nodes, which means we may not want to pursue converting the kube-proxy deployment to a static pod.

i don't like kube-proxy as a static pod mainly because it imposes tight coupling with the kubeadm binary. i.e. it get's promoted from a optional addon that may eventually get applied via kubectl and templated manifests to something that is hardcoded or available via an interface that requires root (i.e. an addon interface that deploys static pods).

[randomvariable]

i'm not familiar with this website and their work.

It's part of the the US DOD

are these requested changes for them to do?

Yes, I've requested the changes.

i don't like kube-proxy as a static pod mainly because it imposes tight coupling with the kubeadm binary.

Ah, ok. I thought that maybe work had started on converting it to a static pod.

[pacoxu]

How about creating a job to do the upgrade of kube-proxy?

The job will wait for all apiserver versions to the newer one, and then upgrade kube-proxy.

We have used similar logic in some cluster installation for special components like cni initial and storage components.

[neolit123]

The job will wait for all apiserver versions to the newer one, and then upgrade kube-proxy.

but that would still happen before the kubelet upgrade on the node?
according to the policy the versions of kube-proxy and kubelet should match.

at this point i'm -1 on moving kube-proxy to a static pod.
perhaps an external kube-proxy operator can manage kube-proxy running as a static pod and also manage it's upgrade properly.

the k-sigs/cluster-addons had some ideas around a kube-proxy operator and we do like to move kube-proxy management outside of kubeadm eventually.

but also it seems the kubelet/kube-proxy skew does not matter for the time being:
https://groups.google.com/g/kubernetes-sig-architecture/c/QX-4qq2krm0/m/bJwosPI_AAAJ

[pacoxu]

Agree, an external kube-proxy operator would be a better option.

[neolit123]

i'm going to go ahead and close this after the discussion here.

• once we support the kubeadm upgrade apply phases we can break down the upgrade process of doing CP upgrade first and then doing something like upgrade apply phase kube-proxy potentially.
  https://github.com/kubernetes/kubeadm/issues/1318
• the operator can handle this better that the step-by-step docs.
  https://github.com/kubernetes/kubeadm/issues/2318 https://github.com/kubernetes/kubeadm/issues/1698

if kube-apiserver / kube-proxy start having more drift in features the skew can become a problem here...but the system should fix itself after some time. what should be a concern is potential downtime while unsupported skew is in effect during upgrade.
let's log a separate issue if we see cases of that.

[neolit123]

@pacoxu

How about creating a job to do the upgrade of kube-proxy?
The job will wait for all apiserver versions to the newer one, and then upgrade kube-proxy.

that is not a bad idea.
are you willing to draft this is a small proposal in a MD / Doc file so that we can review it?

we need to investigate the corner cases around it and how to detect the apiserver versions.

to me this feels safer:

once we support the kubeadm upgrade apply phases we can break down the upgrade process of doing CP upgrade first and then doing something like upgrade apply phase kube-proxy potentially.

[pacoxu]

Is this target of 1.22？

Add to my TODO list then. I'd like to write it next week.

[neolit123]

given we haven't seen actual problems due to the proxy / server skew during upgrade it's not urgent.
you can write your proposal later if you'd like.

ok, i'm going to re-open this and tag with 1.22 milestone.