Metrics-server: X509 certificate issue

[ticruz38]

I am able to start metric-server on my cluster, however when trying to access the logs with kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" I get:
error: You must be logged in to the server (Unauthorized)
the logs inside metrics server pods are like this:

I0727 13:33:23.905320       1 serving.go:273] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)                                                                                                                          │
│ [restful] 2019/07/27 13:33:26 log.go:33: [restful/swagger] listing is available at https://:8443/swaggerapi                                                                                                                                │
│ [restful] 2019/07/27 13:33:26 log.go:33: [restful/swagger] https://:8443/swaggerui/ is mapped to folder /swagger-ui/                                                                                                                       │
│ I0727 13:33:26.284542       1 serve.go:96] Serving securely on [::]:8443                                                                                                                                                                   │
│ W0727 13:33:47.904111       1 x509.go:172] x509: subject with cn=kubernetes-proxy is not in the allowed list: [system:auth-proxy]                                                                                                          │
│ E0727 13:33:47.904472       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: subject with cn=kubernetes-proxy is not allowed, x509: certificate signed by unknown authority]

Can it be a simple RBAC misconfiguration at some point?

Setting up --kubelet-insecure-tls does'nt help

I am using k3s version 0.7.0 on baremetals server running Ubuntu at Scaleway

[jorneilander]

Ok, so here's my research when I encountered the same issue:

K3S has the following API-server flag (default):
--requestheader-allowed-names=system:auth-proxy

I'm 'guessing' this is a cluster role but I'm not 100% sure yet since it doesn't exist within the K3S cluster by default. Looking at the logs I found that basically the API server is complaining that the CN in the TLS-cert used to identify the kubectl top request is not allowed (aka not in system:auth-proxy). Why it's using cn=kubernetes-proxy instead of the account mentioned in ~/.kube/config is unknown to me.

Anyhow, the quick fix is as follows:
Edit your /etc/systemd/system/k3s.service ExecStart-bit to look like the following:

ExecStart=/usr/local/bin/k3s \
    server \
    --kube-apiserver-arg="requestheader-allowed-names=system:auth-proxy,kubernetes-proxy"

Then run systemctl daemon-reload and restart K3S using systemctl restart k3s.

You should now see this setting pop-up in when you run:
kubectl get configmap -n kube-system "extension-apiserver-authentication" -o yaml under: requestheader-allowed-names.

Now all you have to do is kill/restart your metrics-server pod, wait for a few minutes for it to scrape metrics (by default every 60s), and you should now be able to run kubectl top [pod|node].

Since this is good enough for me I'll leave it here, but I am damn curious as to why/how it's using cn=kubernetes-proxy or why the cert used to identify that CN isn't signed by requestheader-client-ca-file.

[ticruz38]

Thanks for your clear answer, it did the tricks!
Following the referenced issue, I found out that this problem occurs only for upgraded k3s cluster. As I did upgrade all the way from 0.3.0
It would probably be a good idea to open an issue on k3s repo as well