Kubeadm: Support certificate rotation

[jbeda]

[FEATURE REQUEST]

We need some way to rotate certificates. Right now we mint certs that expire after 1 year and we have no built in provision to rotate them.

We should either have components manage rotating their own certificates (and writing the certs back to the kubeconfig) or have a way to easily do this from the outside without disrupting a running cluster. Notifying users when rotation is necessary should be part of our design here too.

[pipejakob]

See also https://github.com/kubernetes/kubernetes/issues/4672

[jimmycuadra]

See also https://github.com/kubernetes/kubernetes/issues/25379

[luxas]

@timothysc @pipejakob @mikedanese @jcbsmpsn @jbeda
Should we make this a goal for v1.8 with self-hosting and rotateable kubelet certs?

[jcbsmpsn]

I think there are still a few steps left to accomplishing this:

• https://github.com/kubernetes/kubernetes/pull/41912 and https://github.com/kubernetes/kubernetes/pull/45059 should give support for certificate rotation of kubelet certificates. Out for review now, hopefully they will make 1.7 behind a feature gate, then GA Beta in 1.8.

Remaining:

• When the kubelet boots on a master, and doesn't point to an API server, it has a self signed cert that isn't rotated.
• The API server has a signed certificate that needs to be rotated.
• The CA for the cluster should be rotated/rotatable too.
• https://github.com/kubernetes/kubernetes/issues/4672 covers offering the rotation logic to all clients
• Offering a mechanism to do this on demand seems like a useful perk (my admin just left the company and I'd like to rotate all cert/keys in the cluster)

All these might be a stretch for 1.8.

[jcbsmpsn]

/assign @jcbsmpsn

[luxas]

Sub-goal of v1.8 is getting kubelet client cert rotation enabled, ref https://github.com/kubernetes/kubeadm/issues/386
Other than that, I'm moving this out for further improvements in v1.9 as new auth features regarding this are available.

[nathanleclaire]

It would be cool if somehow this could be configured to run automatically, via cron or something like that, like Docker Swarm mode does. Swarm mode is constantly refreshing certificates, which, to be fair, has its own set of tradeoffs (example: if a worker node panics/partitions and stops refreshing certs for a while, it's locked out of the cluster. not sure if it still happens, but that was one issue we ran into when the feature was new)

[jcbsmpsn]

@nathanleclaire Not sure what you mean. When certificate rotation is enabled it does rotate the certificates automatically. Currently only the certificate used by the node is done. More work to do for other certificates used by Kubernetes.

[nathanleclaire]

@jcbsmpsn Ah, cool. Still catching up on all the fast-moving new Kube stuff :)

[jamiehannaford]

Now that kubelet supports auto-rotation, should we enable this feature gate in 1.9? Would we also need to feature gate it? If so, maybe we can make it generic (e.g. AllowAutoTLSRotation) to pave the way for other components in the future.

[luxas]

We already enable kubelet client cert autorotation. We'll enable kubelet servingcert rotation if possible in v1.9

[luxas]

We'll enable kubelet servingcert rotation if possible in v1.9

Not possible yet AFAIK.

In the interim we'll do #548

[timothysc]

Ideally we could have a phase command to help rotate.
/cc @mattmoyer and @mikedanese (re: other googlers)

[stealthybox]

Suggested MVP is that you would run the phase certs rotate command per node.
@timothysc suggests we have a daemonset with hostpath mounts assist in this rotation/upgrade.

[SpComb]

Related to kubelet server certs, once the kubelet uses a kube-issued server cert instead of the current self-signed server cert, then the kube-apiserver also needs --kubelet-certificate-authority to validate those kubelet server certs.

[timothysc]

/assign @liztio

[liztio]

I've done some testing on the client cert upgrades. My methodology:

1. Join a node to the cluster using kubeadm join
2. make a csr for the existing private key with openssl req
3. Sign the csr on the master using the k8s ca, but set it to expire in two days (-days 2)
4. copy the issued cert back, replace it on the worker node
5. upgrade the node (in my case, 1.11.0 -> 1.11.1).

Based on my reading of the merged features, I should've seen a new certificate for the client node be issued, but so far that hasn't happened. I'll look into this in more detail tomorrow.