Kubeadm: deprecate and remove '--use-api' for cert renewal

[neolit123]

kubeadm has a flag under the kubeadm alpha certs renew command called --use-api that allows users to renew a certificate by sending a CSR and blocking until a privileged user (e.g. admin) approves it, the KCM can then sign the certificate using the root CA:

the CSR API is changing to support and require signer names:
https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/20190607-certificates-api.md#signers

the KCM is no longer going to be able to sign arbitrary certificates and external signers will be required. the legacy signers may be removed in certificates.v1.

actions:

• [x] deprecate --use-api
  assigned: @neolit123
  PR: https://github.com/kubernetes/kubernetes/pull/88827
  cycle: 1.18

• [x] remove --use-api once certficates/v1 is out?
  assigned: @neolit123
  PR: https://github.com/kubernetes/kubernetes/pull/90143
  cycle: 1.19? watching this issue.

• [x] remove flag from the docs:
  https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#create-certificate-signing-requests-csr
  PR: https://github.com/kubernetes/website/pull/26841
  cycle: missed 1.19

---

for more details see the following links and the discussion in this same ticket:
https://github.com/kubernetes/kubernetes/pull/88246#issuecomment-592588131
https://github.com/kubernetes/enhancements/pull/1400/files
https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/20190607-certificates-api.md
https://github.com/kubernetes/kubernetes/pull/86933
https://github.com/kubernetes/kubernetes/pull/86476

[munnerz]

/cc

[neolit123]

marking as important soon and moving to 1.18 as confirmed that the kubeadm CSRs fallback to legacy.

there is also a bug in the API based renewal and currently it does not work.
we can try adding the related changes post CF as a bug-fix.

[munnerz]

Which CSRs are defaulting to legacy? Kubelet client & serving certificates
should all default so long as the request submitted is according to spec
for those types.

Also worth noting that the signer will refuse to sign these certificates
as well if you set the signerName to not-legacy and the request submitted
does not match the requirements for the chosen signerName, as specified in
the KEP:
https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/20190607-certificates-api.md#signers

So it’s important the CSRs being submitted are actually valid for the
signerName, once you begin setting it 🙂

On Sat, 29 Feb 2020 at 14:13, Lubomir I. Ivanov [e-mail]
wrote:

marking as important soon and moving to 1.18 as confirmed that the kubeadm
CSRs fallback to legacy.

there is also a bug in the API based renewal and currently it does not
work.
we can try adding the related changes post CF as a bug-fix.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubeadm/issues/2047?email_source=notifications&email_token=AABRWP3GUMBF6HE4XDYF2A3RFELZJA5CNFSM4K6GJFA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENL3CZA#issuecomment-592949604,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABRWP5KP5KOZPIBC3UIXL3RFELZJANCNFSM4K6GJFAQ
.

[neolit123]

Which CSRs are defaulting to legacy? Kubelet client & serving certificates
should all default so long as the request submitted is according to spec
for those types.

kubeadm does not manage or renew the kubelet client and serving certificates, it enables auto-rotation for the client cert and the serving cert is self-signed.

user have the option to use the API to renew some other certificates, such as:

  apiserver                Generates the certificate for serving the Kubernetes API
  apiserver-etcd-client    Generates the client apiserver uses to access etcd
  apiserver-kubelet-client Generates the Client certificate for the API server to connect to kubelet
...

i did not fully test this, but looking at the defaulting for signerName, our CSRs will fall under legacy:
https://github.com/munnerz/kubernetes/blob/d7e10f98697644367f689be79189583b9163dcab/pkg/apis/certificates/v1beta1/defaults.go#L46-L60

So it’s important the CSRs being submitted are actually valid for the
signerName, once you begin setting it

to enable custom signerNames do we need to define a new ClusterRole and a ClusterRoleBinding, as seen here?
https://github.com/munnerz/kubernetes/blob/d7e10f98697644367f689be79189583b9163dcab/test/integration/certificates/admission_approval_test.go#L107-L147

[neolit123]

waiting for CI to build the latest k/k master and i will confirm my understanding.

i think kubeadm users usually approve CSRs for the above certs using the admin credentials, so in that case the RBAC should be an non-issue.

for other users with external signers we might want to document how to setup the new RBAC in the kubeadm certs page to grant approval for the new signerName.

[BenTheElder]

cc

[neolit123]

@munnerz i think i'm seeing an oddity, or something that i cannot find in the KEP change - why a certficate might not get issued post approval.

• all kube components are at 56987164f8aa20.
• i modified kubeadm to have signerName to CSRs specs e.g. somedomain.io/csr-signer
• created a CSR that looks like this:

Name:               kubeadm-cert-kube-apiserver-t9zjk
Labels:             <none>
Annotations:        <none>
CreationTimestamp:  Sat, 29 Feb 2020 21:46:22 +0200
Requesting User:    kubernetes-admin
Status:             Pending
Subject:
  Common Name:    kube-apiserver
  Serial Number:  
Subject Alternative Names:
         DNS Names:     node-name
                        kubernetes
                        kubernetes.default
                        kubernetes.default.svc
                        kubernetes.default.svc.cluster.local
                        localhost
         IP Addresses:  10.96.0.1
                        172.17.0.2
                        172.17.0.2
                        172.17.0.2
Events:  <none>

• signed the CSR with kubectl certificate approve ... using the same kubernetes-admin (system:masters) credential and the status changed to approved.
• however the certificate is then not being issued:

certificate signing request kubeadm-cert-kube-apiserver-t9zjk is approved, waiting to be issued

I0229 19:48:08.723049 7133 request.go:1068] Response Body: {"kind":"CertificateSigningRequestList","apiVersion":"certificates.k8s.io/v1beta1","metadata":{"selfLink":"/apis/certificates.k8s.io/v1beta1/certificatesigningrequests","resourceVersion":"15116"},"items":[{"metadata":{"name":"kubeadm-cert-kube-apiserver-t9zjk","generateName":"kubeadm-cert-kube-apiserver-","selfLink":"/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/kubeadm-cert-kube-apiserver-t9zjk","uid":"8ff60f77-6357-488d-ac2b-20a6a1d8073a","resourceVersion":"15116","creationTimestamp":"2020-02-29T19:46:22Z","managedFields":[{"manager":"kubeadm","operation":"Update","apiVersion":"certificates.k8s.io/v1beta1","time":"2020-02-29T19:46:22Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{}},"f:spec":{"f:request":{},"f:usages":{}}}},{"manager":"kubectl","operation":"Update","apiVersion":"certificates.k8s.io/v1beta1","time":"2020-02-29T19:47:40Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:signerName":{}},"f:status":{"f:conditions":{}}}}]},"spec":{"request":"CSR_BYTES_HERE","signerName":"somedomain.io/csr-signer","usages":["server auth"],"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"status":{"conditions":[{"type":"Approved","reason":"KubectlApprove","message":"This CSR was approved by kubectl certificate approve.","lastUpdateTime":"2020-02-29T19:47:40Z"}]}}]}
I0229 19:48:08.724043 7133 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubeadm/v1.19.0 (linux/amd64) kubernetes/5698716" 'https://172.17.0.2:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dkubeadm-cert-kube-apiserver-t9zjk&resourceVersion=15116&timeout=6m4s&timeoutSeconds=364&watch=true'
I0229 19:48:08.724074 7133 csr.go:124] certificate signing request kubeadm-cert-kube-apiserver-t9zjk is approved, waiting to be issued
I0229 19:48:08.725696 7133 round_trippers.go:443] GET https://172.17.0.2:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dkubeadm-cert-kube-apiserver-t9zjk&resourceVersion=15116&timeout=6m4s&timeoutSeconds=364&watch=true 200 OK in 1 milliseconds
I0229 19:48:08.725823 7133 round_trippers.go:449] Response Headers:
I0229 19:48:08.725847 7133 round_trippers.go:452] Content-Type: application/json
I0229 19:48:08.725855 7133 round_trippers.go:452] Date: Sat, 29 Feb 2020 19:48:08 GMT

and WaitForCertificate blocks until timeout.

cc @liggitt

[liggitt]

kube-controller-manager only signs certificates for the defined kubernetes.io signerNames, currently:

• kubernetes.io/kube-apiserver-client
• kubernetes.io/kube-apiserver-client-kubelet
• kubernetes.io/kubelet-serving
• kubernetes.io/legacy-unknown

Specifying other signerNames allows external signers to make use of the CSR API. See https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/20190607-certificates-api.md#signers for details.

If you specify another signerName, and aren't running a signer that will issue certificates for that signerName, it is expected that the CSR would just sit in the approved+unissued state.

[neolit123]

unless i'm mistaken, the kubeadm certs renew --use-api feature now must require an external signer.

even if one of the predefined signer names are picked (other than legacy) then some of the certificates we are trying to renew will simply not match the rest of the criteria.

e.g. for the cert/key pair (--tls-cert-file / --tls-private-key-file) of kube-apiserver, a CM kube-apiserver with usages server auth does not fit in any of:
https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/20190607-certificates-api.md#signers

for now i see kubeadm keeping signerName empty, until certsificates.v1 when it may be required, in which case we will move kubeadm to a custom signerName and require users to manage an external signer.

the action for this ticket seems to be to update docs and usage of --use-api to warn users.

[liggitt]

I would recommend explicitly categorizing each certificate by signer, to make it clear which ones are expected to be issued by the kube-controller-manager in the future (the API client and kubelet client ones), and which ones are currently uncategorized and would require an external signer in the future (the etcd certificates, etc). The uncategorized ones would use the legacy signer name for now.

[neolit123]

ok, makes sense. thanks.

[liggitt]

Something else to be aware of is that the beta CSR API is the only beta API currently required to bring up clusters and run conformance e2e jobs (because of the kubelet->client cert renewal).

The plan is to promote it to v1 in 1.19, update the kubelet to use v1, and run a conformance e2e job that runs with all beta APIs disabled. See https://github.com/kubernetes/enhancements/blob/master/keps/sig-architecture/20191023-conformance-without-beta.md for details.

I would avoid adding new ways kubeadm relies on the v1beta1 API in ways that would prevent starting a cluster if v1beta1 was not available. My understanding is that the current kubeadm use is alpha-level and not the default.

[neolit123]

I would avoid adding new ways kubeadm relies on the v1beta1 API in ways that would prevent starting a cluster if v1beta1 was not available.

ack.

My understanding is that the current kubeadm use is alpha-level and not the default.

yes, kubeadm's on demand certificate renewal support is alpha-grade, under a alpha certs renew and the --use-api flag is opt-in. by default the on-disk CA key/cert are using for renewal.

currently we don't have e2e tests for the on-demand cert renewal with or without the flag.

[neolit123]

so we had a talk today (yesterday, 05.03.2020) on the kubeadm office hours meeting, and we ended up with an agreement that due to the requirement of external signers for renewing the common CP certs, we'd prefer to deprecate the --use-api flag.

overall this deprecation is fine as it falls under experimental functionality.
thanks again for explaining the change to me!

PR is here:
https://github.com/kubernetes/kubernetes/pull/88827

/retitle deprecate and remove '--use-api' for cert renewal

[neolit123]

looks like we missed updating the docs for the flag removal:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#create-certificate-signing-requests-csr