Metrics-server: [Kubeadm] "x509: certificate signed by unknown authority

[memorais]

If we start metric-server as documentation states on a Kubernetes cluster created with kubeadm tool, we have certificate validation errors as showed below:

$ kubectl logs -f --namespace=kube-system metrics-server-7cc9c5496f-lsllm
I0601 15:46:27.425923 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0601 15:46:27.425977 1 heapster.go:72] Metrics Server version v0.2.1
I0601 15:46:27.426181 1 configs.go:61] Using Kubernetes client with master "https://10.96.0.1:443" and version
I0601 15:46:27.426192 1 configs.go:62] Using kubelet port 10255
I0601 15:46:27.426882 1 heapster.go:128] Starting with Metric Sink
I0601 15:46:27.931908 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0601 15:46:28.334906 1 heapster.go:101] Starting Heapster API server...
[restful] 2018/06/01 15:46:28 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/06/01 15:46:28 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0601 15:46:28.335923 1 serve.go:85] Serving securely on 0.0.0.0:443
E0601 15:46:31.627056 1 authentication.go:64] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes"), x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
E0601 15:46:32.084203 1 authentication.go:64] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes"), x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]

I did not found any workaround for this.

Anyone already run over this problem?

I'm using Kubernetes v1.10.1

Thanks.

[DirectXMan12]

Please file this again the kubeadm repo if it's an issue with deploying via kubeadm.

[stvcooke]

@DirectXMan12 This isn't an issue with deploying the metrics-server via kubeadm, the cluster was up and running after being initialized by kubeadm. When the metrics-server yaml files were applied via kubectl, the "Unable to authenticate request" errors were logged.

We're encountering the same error.

[DirectXMan12]

So, from what I can see, the issue doesn't appear to be with metrics-server -- it appears to be with the CA certificates provisioned by kubeadm. Namely, it appears that kubeadm has not set up something correctly with its CA certificates. It looks like the apiserver's CA doesn't match the CA injected into the pod automatically. Are other pods using in-cluster auth working properly?

[MarkDeckert]

This can be added to the deployment yaml as a temporary workaround if you have no other recourse (WARNING: insecure)
command:
- /metrics-server
- --kubelet-insecure-tls

[jpds]

I'm also seeing this with a kops deployed cluster (1.10.3).

[amitsaha]

For kops, v0.2.1 version of metrics-server works. Filed an issue there: https://github.com/kubernetes/kops/issues/5770

[ilham9649]

@MarkDeckert after change the deployment to use command :

• /metrics-server
• --kubelet-insecure-tls

the error become :
E1113 00:01:42.520821 1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-0-42-219.ap-southeast-1.compute.internal: unable to fetch metrics from Kubelet ip-10-0-42-219.ap-southeast-1.compute.internal (ip-10-0-42-219.ap-southeast-1.compute.internal): Get https://ip-10-0-42-219.ap-southeast-1.compute.internal:10250/stats/summary/: dial tcp: i/o timeout, unable to fully scrape metrics from source kubelet_summary:ip-10-0-63-150.ap-southeast-1.compute.internal: unable to fetch metrics from Kubelet ip-10-0-63-150.ap-southeast-1.compute.internal (ip-10-0-63-150.ap-southeast-1.compute.internal): Get https://ip-10-0-63-150.ap-southeast-1.compute.internal:10250/stats/summary/: dial tcp: i/o timeout]

[DirectXMan12]

It looks like the default address preferences in v0.3.x don't work for you. You should check the address preferences configured on your API server (--kubelet-preferred-address-types), and then mirror that option to metrics-server. While the default options work in many cases, certain classes of setups need to use different Kubelet addressing setups (e.g. preferring internal IPs to hostnames, or preferring external hostnames).

[wstrater]

I am having a similar issue with contacting the API and tried different versions and different command line flags.

I am getting the following error:

Get https://ip-172-20-45-179.us-west-2.compute.internal:10250/stats/summary/: x509: certificate signed by unknown authority]

To me that is an issue with the trust store used by Metrics Server. I looked at the deployment and I don't see any CA certificates being mounted. How can it trust the API without a trust store? I see kube-apiserver mounting a ca-trust and specifying a private key and certificate in /srv/kubernetes.

I am using Kops 1.11.0, Kubernetes 1.11.6 and Metrics Server 0.3.1. Switching to 0.2.1 didn't install for me.

Thanks, Wes.

[wstrater]

I just checked commit a823af80d438d642c29e038ca5336004b2a8b97e and it appears the Docker file was modified to not mount ca-certificates.crt. Not sure why but the error I am getting makes sense.

Wes.

[wolkjesmaker]

@wstrater also getting this issue on vm's using kubeadm.
Using - --kubelet-insecure-tls solves this but that's something I do NOT want to use.

@DirectXMan12 could you please tell us why.

[Baykonur]

@wolkjesmaker, have a look at this and the comments around it. Hope it helps.

[wolkjesmaker]

@Baykonur
I can't get it to work like that. Only can approve csr for my master node, slaves still have the same problem.

Edit: I might have made a stupid mistake, I'll update this soon.

[Baykonur]

Strange, you don’t see the CSRs for nodes?

[rahulyadav130]

unable to fetch metrics from Kubelet kubemaster.local (kubemaster.local): Get https://kubemaster.local:10250/stats/summary/: dial tcp: lookup kubemaster.local on 10.32.0.10:53: no such host]

[serathius]

@rahulyadav130 This issue is about problem with certificate problem, your problem looks like a DNS configuration one. Feel free to open a separate issue for it.