Kubeadm: Configure secure serving for controller-manager & scheduler

[luxas]

xref: #1285

insecure serving is deprecated. We should switch to secure serving.
controller-manager logs in v1.13.0-rc.1:

I1128 12:40:04.680584       1 serving.go:318] Generated self-signed cert in-memory
I1128 12:40:05.697603       1 controllermanager.go:151] Version: v1.13.0-rc.1
I1128 12:40:05.699084       1 secure_serving.go:116] Serving securely on [::]:10257
I1128 12:40:05.699649       1 deprecated_insecure_serving.go:51] Serving insecurely on 127.0.0.1:10252

scheduler logs in v1.13.0-rc.1:

I1128 12:40:04.787327       1 serving.go:318] Generated self-signed cert in-memory
W1128 12:40:05.278190       1 authentication.go:373] failed to read in-cluster kubeconfig for delegated authentication: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W1128 12:40:05.278217       1 authentication.go:235] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
W1128 12:40:05.278227       1 authentication.go:238] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
W1128 12:40:05.278252       1 authorization.go:177] failed to read in-cluster kubeconfig for delegated authorization: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W1128 12:40:05.278411       1 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
W1128 12:40:05.285538       1 authorization.go:47] Authorization is disabled
W1128 12:40:05.285562       1 authentication.go:55] Authentication is disabled
I1128 12:40:05.285579       1 deprecated_insecure_serving.go:49] Serving healthz insecurely on 127.0.0.1:10251
I1128 12:40:05.286312       1 secure_serving.go:116] Serving securely on [::]:10259

@sttts What do we need to do here?

[luxas]

cc @timothysc @liztio

[sttts]

You have to pass --authentication-kubeconfig and --authorization-kubeconfig and then configure RBAC to

• allow to read the kube-system/extension-apiserver-authentication configmap: https://github.com/kubernetes/kubernetes/blob/1588af4031e746e3e48855256054133e8707f018/test/integration/serving/serving_test.go#L133
• allow to do delegated authorization via SubjectAccessReview: https://github.com/kubernetes/kubernetes/blob/1588af4031e746e3e48855256054133e8707f018/test/integration/serving/serving_test.go#L116

[alexbrand]

To be able to scrape metrics on the secure port of the scheduler, we need to set the --authentication-kubeconfig and authorization-kubeconfig flags. We already do this on the controller manager.

In the case of the scheduler, where the flags are not set, any request to /metrics on the secure port are rejected, because they are not authentication/authorized. By setting the flag, we tell the scheduler to perform tokenaccessreview and subjectaccessreview for requests coming in on the secure port.

[lbogdan]

All RBAC setup was already done in https://github.com/kubernetes/kubernetes/pull/72491 , so it looks like it's only a matter of adding --authentication-kubeconfig and --authorization-kubeconfig to kube-scheduler. If no one is working on this, I'd like to take a shot at it.

[neolit123]

@lbogdan sorry for the delayed reply.

I'd like to take a shot at it.

that would be appreciated, but mind that we are in code freeze soon, so a PR for this can be merged after 1.14 is out.

[fl-max]

For those that stumble upon this and are using Kubeadm, the Kubeconfig is already generated and is mounted into the scheduler pod at /etc/kubernetes/scheduler.conf. You simply need to add authentication-kubeconfig & authorization-kubeconfig to your configuration like so:

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
...
scheduler:
  extraArgs:
    authentication-kubeconfig: "/etc/kubernetes/scheduler.conf"
    authorization-kubeconfig: "/etc/kubernetes/scheduler.conf"

[fabriziopandini]

/assign

[dannymk]

Hmmm...

I added those flags to: /etc/kubernetes/manifests/kube-scheduler.yaml
...
- --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
- --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
...

I also added the kubernetes generated ca to the system:
sudo cp /etc/kubernetes/pki/ca.crt /usr/local/share/ca-certificates/kubernetes-local-ca.crt
sudo update-ca-certificates

That did the trick for me.