Kubeadm: Make the apiserver -> kubelet communication secure

[luxas]

Generate these files in the cert phase as well:

• kubelet-ca.crt
• kubelet-ca.key
• apiserver-kubelet.crt
• apiserver-kubelet.key

Pass to apiserver:

• --kubelet-certificate-authority=kubelet-ca.crt
• --kubelet-client-certificate=apiserver-kubelet.crt
• --kubelet-client-key=apiserver-kubelet.key

Pass to kubelet:

• --client-ca-file=kubelet-ca.crt

Difficulty: How do we transfer the kubelet-ca.crt file from master to node?
Can we somehow transfer two ca:s with a kubeconfig and the token JWS flow...?

Or do we have to extend our custom Go struct for now to pass two CA certs until we have something like kubeconfig v2?

Thoughts? @jbeda @pires

@mikedanese Did I get it right?

[pires]

Why not use the same CA for server and file certs? This way, bot master and worker nodes will know how to validate server and client certificates.

[luxas]

This is special, since there are a lot of certificates signed from ca.crt, so if the kubelets trust certificates generated from ca.crt (=any client cert really), all kubelets will trust any client cert to exec, get logs and so forth, which isn't so great.

Imagine I got a client cert which is restricted to only get pods and services in a given namespace (signed from ca.crt). Then I could escalate those privileges and get the logs on any node if we set --client-ca-file=kubelet-ca.crt. So we need a dedicated CA where there's only client certs for the API Server(s), so only the API Servers are authenticated to talk to the kubelets.

As we discussed in the last sig meeting, it's not an optimal solution, but the best solution we have currently.

cc @kubernetes/sig-auth-misc @roberthbailey

[pires]

I'm not sure I follow. CA is used to sign and validate certificates (identities). After you trust a certificate, you are able to identify the client and look for the respective permissions. I may be missing something here.

[deads2k]

Imagine I got a client cert which is restricted to only get pods and services in a given namespace (signed from ca.crt). Then I could escalate those privileges and get the logs on any node if we set --client-ca-file=kubelet-ca.crt. So we need a dedicated CA where there's only client certs for the API Server(s), so only the API Servers are authenticated to talk to the kubelets.

I don't think this is true. The identity from the client cert is then used for an authorization check which determines if the action is allowed. The kubelet runs this check using a subjectaccessreview.v1beta1.authorization.k8s.io check against the core API server which uses the same authorizer as the API server. Perhaps you accidentally set your cluster up with an unrestrictive policy instead of RBAC?

[liggitt]

The identity from the client cert is then used for an authorization check which determines if the action is allowed. The kubelet runs this check using a subjectaccessreview.v1beta1.authorization.k8s.io check against the core API server which uses the same authorizer as the API server.

see https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization for setting up a kubelet that authorizes requests

[roberthbailey]

@cjcullen @mikedanese

[luxas]

Perhaps you accidentally set your cluster up with an unrestrictive policy instead of RBAC?

@deads2k Yes, that information above was what I got in the light of no RBAC.
Do you mean we don't have to do this if we set up some rules in the SubjectAccessReview API that authorizes only apiservers? How would that work?

[deads2k]

@deads2k Yes, that information above was what I got in the light of no RBAC.
Do you mean we don't have to do this if we set up some rules in the SubjectAccessReview API that authorizes only apiservers? How would that work?

SAR exists already and it uses the same authorizer as the API server. If you set up RBAC with safe rules, your kubelet will be safe too without any additional effort.

[luxas]

Yes, but this is about apiserver contacting the kubelet and wanting to access the logs or exec some pod.
And the point is that only the apiservers, no other client certificate that normally has access to the apiserver can authorize to the kubelet.

I'm not sure if I'm missing something or if we're talking about two different things. I want to prevent my admin.conf file from accessing every kubelet, the only one that can access the kubelet should be the apiserver that has a special certificate signed by a second CA cert, see above.

[deads2k]

I'm not sure if I'm missing something or if we're talking about two different things. I want to prevent my admin.conf file from accessing every kubelet, the only one that can access the kubelet should be the apiserver that has a special certificate signed by a second CA cert, see above.

I disagree that it should be a separate signer. I think a single signer is a reasonable state to be in and that it is safe. We should not get lazy with authorization and attempt to fix it via authentication that still doesn't resolve the underlying authorization problem.

The kubelet authorization rules which @liggitt referenced are prescriptive and safe.

[luxas]

Ok, then I haven't heard or read at all about that kubelet authorization mechanism that only allows apiservers to access kubelets.

It was just said in the meeting that this was the only way to go so therefore I thought I'd write it up and make sure we proceed on it.

Very cool if we have a better alternative :) Do you have an example RBAC/SAR configuration that only allows apiservers to contact kubelets?

[deads2k]

Very cool if we have a better alternative :) Do you have an example RBAC/SAR configuration that only allows apiservers to contact kubelets?

The default, OOTB configuration. See https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization

[deads2k]

If you grant a user cluster-admin rights, then that user already has full API control, so also having kubelet access doesn't grant him anything new. That kubelet operates at his pleasure and runs exactly what he wants. I suspect you've built an admin.conf that is a cluster-admin.

[luxas]

Ok, so if we're setting --authorization-mode=Webhook and --client-ca-file=ca.pem (the CA used for everything), kubelet will only authenticate client certificates of the ca.pem cert _and_ only authorize cluster-admins to access its API, right?

So basically the --authorization-mode=Webhook flag will make it deny all requests except for cluster-admins?

[deads2k]

So basically the --authorization-mode=Webhook flag will make it deny all requests except for cluster-admins?

The specific checks are laid out in the doc I linked, but we didn't make a role for them (not something you normal distribute and API servers are cluster-admins), so its effectively cluster-admin only for now.

[luxas]

Yes, so what I meant was that without any specific rules for this matter; only cluster-admins are authorized (unless a cluster-admin specifically grants other people that same priv)?

[deads2k]

Yes, so what I meant was that without any specific rules for this matter; only cluster-admins are authorized (unless a cluster-admin specifically grants other people that same priv)?

correct.

[luxas]

Ok, that's really cool! Thanks for informing me, I've started to think about auth[n|z] for like two weeks ago, so haven't had time to catch up on all mechanisms, but I'm up to speed soon. Thank you.

This makes it indeed a lot easier for us with less certs.
I'm gonna add --authorization-mode=Webhook to the kubelet as soon as we have RBAC working for kubeadm at HEAD

[liggitt]

The default, OOTB configuration. See https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization

Authorization doesn't default on yet, since it uses v1beta1 APIs, but that documentation is correct for how to enable it. (edit: the APIs are going to v1 in 1.6, I expect deployments to default this on by 1.7)

If you grant a user cluster-admin rights, then that user already has full API control, so also having kubelet access doesn't grant him anything new. That kubelet operates at his pleasure and runs exactly what he wants. I suspect you've built an admin.conf that is a cluster-admin.

I would agree.

@luxas the changes on your side (from the referenced doc) would be:

1. enable X509 client certificate authentication to the kubelet’s HTTPS endpoint:

   • start the kubelet with the --client-ca-file flag (I'd expect you to provide the cluster CA here)
   • start the apiserver with --kubelet-client-certificate and --kubelet-client-key flags (you'll need to create this certificate... the username is up to you, make sure it's in the system:masters group)

2. Enable kubelet authorization:

   • ensure the authorization.k8s.io/v1 API group is enabled in the API server (I think it is by default)
   • start the kubelet with the --authorization-mode=Webhook, --kubeconfig, and --require-kubeconfig flags

With that configuration (and the default RBAC policy), only the cluster admin superuser and apiserver will be authorized to make kubelet API requests to the secured kubelet port (there's still the readonly kubelet port, but that's a different issue)

[liggitt]

I see the apiserver-related changes in https://github.com/luxas/kubernetes/commit/071844e35f954b493de5846d7cc797cb25ca3a11 but not the kubelet ones... where is the config for the kubelet done?

[yann-soubeyrand]

Also, it seems the API server still doesn't verify the kubelet certificate (the --kubelet-certificate-authority flag have to be used). Am I wrong?

[tswift242]

@luxas, I'm using kubernetes 1.10.x and, as @yann-soubeyrand pointed out, kubeadm still doesn't set the --kubelet-certificate-authority argument for the API server. Looking over the diff for https://github.com/kubernetes/kubernetes/pull/40292, I don't see any mention of --kubelet-certificate-authority. Can kubeadm be enhanced to set this by default as well?

Even if it's not set by default, it's not clear which file should be pointed to for that flag. I tried using /etc/kubernetes/pki/ca.crt and I got an error.

[luxas]

Right, we can't make the connections from the api server to the kubelet servers verified, as each kubelet has its own self-signed cert. We might consider a manual approving flow wrt kubelet serving certs in the future but that's not secured by default at the moment.

[jrfeenst]

Cat the kubelet get a signed server cert when it joins the cluster? Aren't there already certificate signing requests being sent from kubeadm/kubelet to the apiserver?

[liggitt]

Cat the kubelet get a signed server cert when it joins the cluster?

It can, if there is an approval process in place to approve those serving certificates. Today, nothing in the cluster has sufficient knowledge to know if a particular kubelet is entitled to a serving certificate for a particular address.

Aren't there already certificate signing requests being sent from kubeadm/kubelet to the apiserver?

For client certificates, yes. For serving certificates, no. Serving certificate requests is planned to be promoted to beta in 1.12, but still requires the presence of a CSR approver for those requests.

[mark-casey]

Aren't there already certificate signing requests being sent from kubeadm/kubelet to the apiserver?

For client certificates, yes. For serving certificates, no. Serving certificate requests is planned to be promoted to beta in 1.12, but still requires the presence of a CSR approver for those requests.

@liggitt @luxas Once "serving certificate requests" are available... Would approval of CSR[s] for Kubelet serving certs be a kubeadm phase? Or in other words, would the "CSR approver" be a person or some future functionality?

[liggitt]

@liggitt @luxas Once "serving certificate requests" are available... Would approval of CSR[s] for Kubelet serving certs be a kubeadm phase? Or in other words, would the "CSR approver" be a person or some future functionality?

since kubelets can request new certs in response to IP changes or cert expiration, an approval process should not be limited to cluster setup. Currently, each deployment method would be required to set up an approver. For example, a cloud provider could set up an approver controller that approves requests for serving certificates for nodes if they are limited to the addresses (hostnames and IPs) assigned to that node in metadata.