Rke: Enable API bearer tokens (including service account tokens) to be used to authenticate to the kubelet’s HTTPS endpoint

[guangbochen]

RKE version: 0.1.7
This is a feature request:

Prometheus operator requires service account tokens authentication of kubelets to be enabled for it own and cadvisor metrics, can we please add this config to rke with enabled on default and optional to turn off by cluster admin.

--authentication-token-webhook=true
--authorization-mode=Webhook

Reference:
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/

https://github.com/coreos/prometheus-operator/issues/976

[cjellick]

To accomplish this, the k8s authentication webhook framework would need an endpoint to call back to. We would need to expose the Rancher authentication proxy as an an endpoint capable of functioning as one. I've opened the following issue in the rancher repo:
https://github.com/rancher/rancher/issues/13474

[gvisca]

I'm working with version 0.1.6 of RKE and I managed to get kubelet metrics with prometheus operator with this kubelet config

 kubelet:
   image: rancher/hyperkube:v1.10.1-rancher2
   extra_args:
     authentication-token-webhook: true

adding authorization-mode=Webhook in extra-args prevent rke to deploy the cluster

Hope this helps

[cjellick]

I don't really understand what you did here, @galal-hussein @moelsayed @ibuildthecloud do you any of you understand what they did and how it allow the prom operator to work?

[clkao]

It seems --authorization-mode=Webhook would make the installer fail, as nodes/proxy is not allowed from kube-apiserver and system:nodes. The installer will stop before properly configuring cni.

If you initialize the cluster first and then add --authorization-mode=Webhook to cluster.yml after creating the proper clusterrole and clusterrolebinding, running rke up again configures kubelet properly. Unfortunately adding the role manifest to addons_include doesn't work, probably because they are created too late. perhaps we need something like bootstrap_addons.

[jiuchongxiao]

did you solved it?@guangbochen

[jiuchongxiao]

@gvisca how to solve the prevent ?

[wangwj]

rke0.1.7
kubelet:
extra_args:
authentication-token-webhook: true
authorization-mode:Webhook

FATA[0105] [workerPlane] Failed to bring up Worker Plane: Failed to verify healthcheck: Service [kubelet] is not healthy on host [192.168.181.95]. Response code: [403], response body: Forbidden (user=system:node, verb=get, resource=nodes, subresource=proxy)
@gvisca

[gvisca]

Sorry, for my bad english !

I first tried to launch rke with options

kubelet:
   image: rancher/hyperkube:v1.10.1-rancher2
   extra_args:
     authentication-token-webhook: true
     authorization-mode: Webhook

to make prometheus work (as specified here https://github.com/coreos/prometheus-operator/issues/976)
but got the Forbidden error @wangwj mentioned

So I tried with these options

kubelet:
   image: rancher/hyperkube:v1.10.1-rancher2
   extra_args:
     authentication-token-webhook: true

then rke is working fine and prometheus is happy ...

[jiuchongxiao]

i did as @gvisca said

kubelet:
   image: rancher/hyperkube:v1.10.1-rancher2
   extra_args:
     authentication-token-webhook: true

but the rke was wrong just the same.
how to solve the problem,do you have some idea? @gvisca @superseb @sangeethah @soumyalj@adingilloRancher

[pdhung]

Does RKE support "--authorization-mode=Webhook" for kubelet? I tried to add to kubelet extra_args this flag but then it got error.
Enabling tokens authentication alone without proper "--authorization-mode=Webhook" settings means that every service account can now connect to kubelet and control it. This is potentially bad.

[pdhung]

@clkao: I saw your post, but unfortunately got error when I tried. Maybe I missed something. I am also not sure about how many components needing access to kubelet and what roles/permission they need. Can you please help me to list the needed roles and bindings in order for "--authorization-mode=Webhook" to work properly?

[clkao]

@pdhung just run rke without --authorization-mode=Webhook and with just one node. Once the cluster is working, apply rbac and rerun rke with --authorization-mode=Webhook on all nodes.

[ceenobyte]

@clkao I tried exactly as you said, but I still get a failure at the verificaton of the healthcheck service for kubelet during rke deployment. May I ask you, which version of rke you are using?