Kubeadm: provide workarounds for the kubelet self-signed serving certificate

[neolit123]

TL;DR kubeadm manages a kubelet that self-signs it's serving certificate. this creates a blocker for metric server users that wish to scrape the kubelet as a server.

related issue:
https://github.com/kubernetes/kubeadm/issues/1223

related enhancement:
https://github.com/kubernetes/enhancements/issues/267

official docs:
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#certificate-rotation

Note: The CSR approving controllers implemented in core Kubernetes do not approve node serving certificates for security reasons. To use RotateKubeletServerCertificate operators need to run a custom approving controller, or manually approve the serving certificate request

this ticket is to track documenting a workaround in our TS guide:
https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/

or alternatively as a MD file / guide in this repository under /docs.

[neolit123]

someone shared this operator:
https://github.com/kubernetes/kubeadm/issues/1635#issuecomment-509559813

[fabriziopandini]

@neolit123 is https://github.com/kubernetes/kubeadm/issues/1753 going to fix this as well?

[neolit123]

@fabriziopandini no, this is about the kubelet self-signed serving cert.
some users want it signed with the CA, instead.

[NeilW]

Helm workaround is to turn off TLS checking for the metrics server

$ helm install --set 'args={--kubelet-insecure-tls}' --namespace kube-system metrics stable/metrics-server

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/lifecycle frozen

[zedtux]

Looking for this so that deploying metrics-server would work securely and out-of-the-box!

[blepoutr]

Hello

Looking for this so that deploying metrics-server would work securely and out-of-the-box!
Indeed, and also to allow proper APIserver to kubelet configuration ( which is now not validating Kubelet serving cert with any CA per doc : https://v1-16.docs.kubernetes.io/docs/concepts/architecture/master-node-communication/#apiserver-to-kubelet )

[NeilW]

I've written up the process and how to get metrics server to work securely with a kubeadm deployed Kubernetes system here:

https://www.brightbox.com/blog/2020/09/15/secure-kubernetes-metrics/

Hope you find it useful.

[neolit123]

I've written up the process and how to get metrics server to work securely with a kubeadm deployed Kubernetes system here:

https://www.brightbox.com/blog/2020/09/15/secure-kubernetes-metrics/

Hope you find it useful.

Thanks for the writeup, Neil.
Related to 'serverTLSBootstrap: true' it is important to note that this certificate will expire after 1 year. If serving cerificate rotation is enabled, the user needs to approve the CSR when that happens. Thus, users created the controllers mentioned above.

I wish we had a guide for using rbac proxy too.

[blepoutr]

Hello @NeilW

Thanks for the very good article .
One variant seems to sign the Serving/Server csr of the Metrics server with Kubernetes.
See https://github.com/jenting/secure-metrics-server/blob/master/gen-metrics-server-cert-key.sh.

[NeilW]

I've added another post about auto-signing the kubelet certificates using "rubber stamp" and why we believe that is an acceptable risk on kubeadm installed clusters.

https://www.brightbox.com/blog/2020/09/24/auto-signing-kubernetes-server-certificates/

[randomvariable]

The situation has changed somewhat since this was opened.

The current state is that cloud providers are or are going to implement CSR signers that verify the identity of the node via an out of band mechanism, and that would provide a stronger guarantee than the rubber stamp controller alone.

We should probably direct users to look at the relevant cloud provider docs to see what their options are, and suggest they can use a rubber stamp or custom controller given the subject access review etc...

[neolit123]

i can summarize the state in the docs this cycle.

[neolit123]

opened PR for the kubeadm docs:
https://github.com/kubernetes/website/pull/27071