Kops: [METRICS-SERVER] New kubelet-certificate-authority flag available (v0.3.2)

[luarx]

1. Describe IN DETAIL the feature/behavior/change you would like to see.
Don't use kubelet-insecure-tls flag by default in https://github.com/kubernetes/kops/blob/master/addons/metrics-server/v1.8.x.yaml because of security reasons.

Maybe using the new --kubelet-certificate-authority flag this could be done.

2. Feel free to provide a design supporting your feature request.
This should be changed here https://github.com/kubernetes/kops/blob/master/addons/metrics-server/v1.8.x.yaml

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Symbianx]

I'm trying to use this new flag on my deployment. However I'm struggling to find the CA that signs the kubelet certificates. Do you know where that is?

[luarx]

Check kubelet process that is running, sometimes it has specified the CA file as a flag

[Symbianx]

Our kubelet processes didn't specify it. After looking more into it, we found out each host had its own self signed CA for the kubelets. Is there any way to share a CA between all kubelets using kops?

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Nilubkal]

/reopen

[k8s-ci-robot]

@Nilubkal: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Nilubkal]

Thanks @rifelpet for reopening ! It seems that i cannot pass the flag --kubelet-certificate-authority to the kube-apiserver when using a template to create the cluster. In the https://github.com/kubernetes/kops/blob/master/pkg/apis/kops/componentconfig.go there is no such option ?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[agilgur5]

Having the same issue trying to validate the kubelet cert with metrics-server. --kubelet-insecure-tls exposes a MITM attack vector and so is a bit 😬 that this the default option for the add-on.

I tried using --kubelet-certificate-authority myself, but no luck, can see a very detailed, fairly exhaustive description at https://github.com/kubernetes-sigs/metrics-server/issues/541.

Do any kops folks know how to get rid of the --kubelet-insecure-tls option and properly validate the kubelet cert? Per the linked issue, I tried --kubelet-certificate-authority=/srv/kubernetes/ca.crt (after mounting hostPath with root privileges to test) as well as downloading the cert from S3 in /pki/issued/ca/ and adding it as a mounted Configmap as well, but it is the same as /srv/kubernetes/ca.crt.

I noticed on the host that there's a file /var/lib/kubelet/pki/kubelet.crt that contains two different certs, but I'm not sure if those are related or used. /var/lib/kubelet/pki/kubelet.crt isn't passed as an arg to kubelet anywhere.

A similar issue around this had been opened here as well: https://github.com/kubernetes/kops/issues/7083 . Unsure if there are more too.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[jmcdonagh]

Surprised this is still an issue... any chance this can get looked at for 1.20?

[jmcdonagh]

/remove-lifecycle rotten

[olemarkus]

It is a question on how to provision the certificates on each node. In kops there is no central mechanism for issuing certificates that will work on every node (including the control plane).
The only approach I can think of is creating a new CA that is stored in the secret store, and then each node during boot fetch that CA and provision their own cert using that shared CA. The CA signing keys would however be accessible on each node and therefore not increase security by much.

[agilgur5]

@olemarkus why does this require more provisioning?

As far as I understand and as I wrote in my comment above, it seems to be asking for the kubelet's CA in order to verify the kubelet's cert. Per my comment and the issue I linked to in metrics-server, I was unable to get it to validate using the same CA the kubelet uses.

[olemarkus]

Kubelet has two certificate chains. One used as client authenticating to the API server. The other is a serving certificate used for the HTTP API.
You are trying to use the client CA cert, which is why you cannot get this working. At the moment the serving cert is self-signed generated by the kubelet _on each node_. This will never work with metrics-server as it cannot access the CA cert of each node, nor can it work with multiple CAs.

[olemarkus]

Submitted #10022
Since metrics-server assumes the same CA as kube-apiserver is used by default, we don't even have to use --kubelet-certificate-authority flag.

[jmcdonagh]

@olemarkus nice work. i think this will make more than a few people happy!