Kubeadm: Make sure that Kubelet Certificate Rotation is enabled in v1.8

[luxas]

As https://github.com/kubernetes/features/issues/266 and https://github.com/kubernetes/features/issues/267 mature, we should enable them in the kubeadm-specific kubelet arg list (https://github.com/kubernetes/release/blob/master/debian/xenial/kubeadm/channel/stable/etc/systemd/system/kubelet.service.d/10-kubeadm.conf)

At least client cert rotation will be beta in v1.8. We should enable that behavior by default.

cc @kubernetes/sig-auth-feature-requests @jcbsmpsn

[ericchiang]

Since this is part of the 1.8 milestone, going to note that this feature will not be turned on by default for the kubelet.

https://github.com/kubernetes/kubernetes/pull/51045#discussion_r134328227

CSRs aren't properly garbage collected as of 1.8, so having the kubelets create some arbitrary number of CSRs on clusters that don't have an external controller to clean them up is problematic. Hoping to address this in 1.9.

[luxas]

We're planning to enable it via the feature gate. I saw that CSRs aren't GC'd yet, but if I understood it correctly, the expiration time is one year, right?
It makes a lot of sense to enable this now that we're migrating from kubeadm join doing the TLS bootstrap => /etc/kubernetes/kubelet.conf in v1.7 to kubeadm join writing /etc/kubernetes/bootstrap-kubelet.conf in v1.8, which kubelet uses to do the TLS bootstrapping itself. As we're doing this transition anyway, I think it's reasonable to enable now. (Also see https://github.com/kubernetes/kubernetes/pull/52188)

[ericchiang]

We're planning to enable it via the feature gate.

Feature gate is enabled by default. There was a new flag added as part of https://github.com/kubernetes/kubernetes/pull/51045

This defaults to false because the actual CSR kubernetes resources in the certificates API are never removed, even if they're expired, denied, or even ignored for some period of time.

[luxas]

@ericchiang Gotcha. Then we're planning to set --rotate-certificates for the reasons I mentioned above :)
Makes sense?

[ericchiang]

Yep!

[mattmoyer]

Just to clarify because I wasn't familiar with the 1.8 behavior...

CSRs aren't garbage collected in 1.8, but the default lifetime is 1 year and rotation only happens after between 70% (~8 months) and 90% (~10 months) of the lifetime of the certificate. So in the worst case a cluster of N nodes will accumulate N CSRs every 8.4 months or so. This means after 3 years you still only have around 4N CSRs.

This is where the default certificate lifetime is set:
https://github.com/kubernetes/kubernetes/blob/b2b079b95a867b560fd327cba56e7d3878f43dc3/cmd/kube-controller-manager/app/options/options.go#L113

This is where the 70%-90% calculation is done:
https://github.com/kubernetes/kubernetes/blob/4d409a4d9e9772193cb24944bd76b90aaca01473/pkg/kubelet/certificate/certificate_manager.go#L311-L319