Metrics-server: Constant metrics-server X509 certificate issues in an RKE cluster

[jiphex]

I deployed this RKE cluster about 1 year ago, it's been upgraded a number of times since.

❯ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T16:54:35Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

❯ rke --version
rke version v0.2.8

I've never got the metrics-server to work, either using the standard RKE deployment or by trying to deploy the stable/helm chart. I always seem to get the same problems:

1. Running kubectl api-resources shows the resources, plus an error about the metrics APIs:

❯ kubectl api-resources
NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND
bindings                                                                      true         Binding
...elided API resources...
volumeattachments                              storage.k8s.io                 false        VolumeAttachment
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: Unauthorized

1. The metrics-server Pod logs always show the following:

❯ kubectl logs metrics-server-64b595d5f6-jbl5n
I1002 16:37:08.512143       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I1002 16:37:08.835162       1 secure_serving.go:116] Serving securely on [::]:8443
E1002 16:37:33.926319       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E1002 16:37:33.932079       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E1002 16:37:36.626354       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
..... lots and lots of this ...
E1002 16:38:07.079502       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E1002 16:38:09.001200       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:130.246.7.36: unable to get CPU for container "app" in pod dafni-nice-cloud-api-service/review-dafni-682-c3ka62-6d7d775c9d-jxm7j on node "130.246.7.36", discarding data: missing cpu usage metric, unable to fully scrape metrics from source kubelet_summary:130.246.7.60: [unable to get a valid timestamp for metric point for container "app" in pod dafni-nid-nid3-datastore-api-service/review-dafni-1756-qg0w72-7c86cd4699-mdg2m on node "130.246.7.60", discarding data: no non-zero timestamp on either CPU or memory, unable to get CPU for container "app" in pod dafni-nid-nid3-datastore-api-service/review-stress-tes-je6jlw-55d5dcd485-vbkjt on node "130.246.7.60", discarding data: missing cpu usage metric], unable to fully scrape metrics from source kubelet_summary:130.246.7.27: [unable to get a valid timestamp for metric point for container "app" in pod dafni-nid-nid3-datastore-api-service/review-dafni-1786-2vcbtv-7cfdd5fdbd-tk26z on node "130.246.7.27", discarding data: no non-zero timestamp on either CPU or memory, unable to get a valid timestamp for metric point for container "metrics-server" in pod kube-system/metrics-server-6f6c4f75d5-kvkwl on node "130.246.7.27", discarding data: no non-zero timestamp on either CPU or memory]]

I've tried this with a few versions of Rancher's metrics-server image, plus the current one from gcr.io/google_containers/metrics-server-amd64. The current arguments to my metrics-server are as follows:

containers:
- command:
    - /metrics-server
    - --cert-dir=/tmp
    - --logtostderr
    - --secure-port=8443
    - --kubelet-insecure-tls
    - --kubelet-preferred-address-types=InternalIP
    volumeMounts:
    - mountPath: /tmp
      name: tmp
  ...
 volumes:
  - emptyDir: {}
    name: tmp

I've always had this problem with this cluster, both before and after an expired certificate issue forced a certificate rotation. I've tried deleting the metrics-server and replacing it in a few different ways, this has also included removing the service account and recreating it. I'm just stuck, I can't work out where this certificate error is now coming from, or even really what certificate it's complaining about or trying to authenticate.

[errordeveloper]

I've seen this in EKS also, cannot confirm yet what causes it, but it could be related to HTTPS proxy uses in this particular setup.

Once I'll figure out where authentication.go lives (it's not in this repo), I'll try to add debug messages as to what certificate the error message relates to (it's hugely ambiguous the way it's logged right now).

@DirectXMan12 any hits would help! In my case metrics-server seems happy and HPA works, but all kubectl requests are failing with above error.

[errordeveloper]

In my case issue turned to be an external factor, it's a complex setup and I don't have control over or insight into many of the elements at play.

[forsberg]

@errordeveloper If you download the source code for one of the releases of metrics-server, you will find it under vendor/k8s.io/apiserver/pkg/endpoints/filters. Pretty sure it's this code, although in some earlier version (line number doesn't match in the link)

[mvisonneau]

I have this exact behaviour on EKS 👍

[errordeveloper]

@mvisonneau my hunch is that what fixed the problem was upgrade, IIRC when it didn't work that cluster was on 1.11, and then it got upgraded to either 1.12 or 1.13 and this issue gone away. But don't take my word for it, I didn't analyse/debug it as I was simply happy the issue was gone.

[mvisonneau]

Interesting, thanks for the heads up @errordeveloper

I am already running on the latest version of EKS 1.14.7 - eks.3 and metrics-server 0.3.6 though :(

[errordeveloper]

Maxime, did you try creating a new cluster? It would be good to find a way
to reproduce this!

On Tue, 3 Dec 2019, 4:39 pm Maxime VISONNEAU, [e-mail]
wrote:

Interesting, thanks for the heads up @errordeveloper
https://github.com/errordeveloper

I am already running on the latest version of EKS 1.14.7 - eks.3 and
metrics-server 0.3.6 though :(

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/metrics-server/issues/319?email_source=notifications&email_token=AAB5MS6F3LW3YRPL4IYXHSTQW2DTVA5CNFSM4I4Y2FJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF2AHIA#issuecomment-561251232,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAB5MS5SG3BPDKETKSLGJ4LQW2DTVANCNFSM4I4Y2FJA
.

[mvisonneau]

I did another attempt on an new 1.14.8/eks.5 cluster, and got the same results!

[errordeveloper]

Maxime, do you have instructions to reproduce? Last time I tried with a
vanilla eksctl cluster and metrics-server installed by helm, I couldn't
reproduce the issue. It only came up in someone else's cluster I was
working with at the time.

On Wed, 11 Dec 2019 at 20:41, Maxime VISONNEAU [e-mail]
wrote:

I did another attempt on an new 1.14.8 - eks.5 cluster, and got the same
results!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/metrics-server/issues/319?email_source=notifications&email_token=AAB5MS3XUOVEE4CTUZKZHZ3QYFF5XA5CNFSM4I4Y2FJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGUP5ZI#issuecomment-564723429,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAB5MS73ZUJPQVKBG5XKRX3QYFF5XANCNFSM4I4Y2FJA
.

[mvisonneau]

I retried using basic configuration of eksctl + chart values and managed to reproduce:

~$ eksctl create cluster \
--name test \
--version 1.14 \
--region eu-west-1 \
--nodegroup-name test \
--node-type t3a.medium \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4 \
--managed

~$ helm upgrade -i metrics-servers stable/metrics-server

[errordeveloper]

Thank you, Maxime! I don't have the time to look into this myself, but
hopefully someone else will find this and be able to dig into it.

On Thu, 12 Dec 2019 at 17:49, Maxime VISONNEAU [e-mail]
wrote:

I retried using basic configuration of eksctl + chart values and managed
to reproduce:

~$ eksctl create cluster \
--name test \
--version 1.14 \
--region eu-west-1 \
--nodegroup-name test \
--node-type t3a.medium \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4 \
--managed

~$ helm upgrade -i metrics-servers stable/metrics-server

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/metrics-server/issues/319?email_source=notifications&email_token=AAB5MS2GHANEXRGFUTXFEZTQYJ2SZA5CNFSM4I4Y2FJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGXPBWQ#issuecomment-565113050,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAB5MS6AYAG6FCWHATWVNCDQYJ2SZANCNFSM4I4Y2FJA
.

[costimuraru]

Having the same issue in EKS

[mvisonneau]

I do not have AWS support but might be interesting to open a ticket with them too 🤷‍♂

[errordeveloper]

/cc @tabern @mhausenblas

[BlenderDude]

TLDR: I added the username property to the role mapping.

After a long night of trial and error, I finally got my metrics server to work on EKS without the x509 errors and WITHOUT specifying --kubelet-insecure-tls. I had a setup where I wanted codebuild to use helm to deploy to my cluster using a role that would link into the cluster's iam authenticator. I wanted to get HPA setup so I needed the metrics server.

I had my aws-auth configmap setup like this

mapRoles: |
- groups:
  - system:bootstrappers
  - system:nodes
  rolearn: arn:aws:iam::[redacted]:role/EKSNode
  username: system:node:{{EC2PrivateDNSName}}
- groups:
  - system:masters
  rolearn: arn:aws:iam::[redacted]:role/kubernetes-codebuild

I setup a trust relationship with my personal IAM account and the k8s codebuild role so I could try it myself. Using my user iam account, it worked just fine, but when assuming the role it was throwing Unauthorized to kubectl and the server logs showed the x509 errors as described above.

The one notable config property missing from the role mapping is the username property. I left this out because I thought the role name would be fine by itself. _Apparently not!_ I added the username property (along with a SessionName suffix, but that is irrelevant) and now the metrics server doesn't throw x509's anymore. My current aws-auth config map looks like this

mapRoles: |
- groups:
  - system:bootstrappers
  - system:nodes
  rolearn: arn:aws:iam::[redacted]:role/EKSNode
  username: system:node:{{EC2PrivateDNSName}}
- groups:
  - system:masters
  rolearn: arn:aws:iam::[redacted]:role/kubernetes-codebuild
  username: codebuild:{{SessionName}}

To be clear, _I have absolutely no idea why this works_. I just thought my experience might be helpful in fixing the bug. A guess of mine might be invalid characters in the ARN messing up a parser somewhere, but I'm really not sure.

[mvisonneau]

Wow! This is just brilliant, thank you so much for sharing these findings @BlenderDude!

I confirm that adding the username property to the role mapping fixes it for me as well!

[costimuraru]

Amazing, @BlenderDude! This worked like a charm!! We had multiple roleArns in the aws-auth and except for the one with username: system:node:{{EC2PrivateDNSName}}, the others did not have a username specified. After adding the field to each one, the metrics-server started to work. The HorizontalPodAutoscaling also started to work:
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
Thanks a bunch!
We also have a ticket opened with AWS for this issue, I will let them know about this workaround so they can investigate what's going on behind the scenes. Perhaps something like the thing you mentioned with a parsing error.

[errordeveloper]

I can confirm that setting username: cluster-admin worked, I still have confirmed why this doesn't occur in all of the cluster that are setup in the same way, and whether the value of username actually matters, I just set it to the same name as the effective clusterrole is in this case.

[jiphex]

I don't know if other comments in this thread are relevant specifically to this issue, as my issue is reproducible on RKE on an entirely bare-metal cluster, specifically not on AWS. If I can apply the same config you've setup in your AWS config to this metrics-server deployment then it may be related.

[errordeveloper]

@jiphex yeah, the fix we are discussing is about aws-auth config map and it's 100% EKS-specific. It's possible that RKE has some kind of an auth component that works in way that is similar to EKS, and also in effect what metrics-servers sees is a username missing from client certificates.

[errordeveloper]

One thing for is that maintainers should look into making this error message more specific.

/cc @DirectXMan12

[serathius]

Looks like problem was resolved.

@errordeveloper please submit a separate issue about making error message more specific.

[jiphex]

FWIW I don't believe the issue I raised has been resolved, the other comments refer specifically to an EKS issue, which may be related to the original issue I reported which has nothing to do with EKS.

[errordeveloper]

Yes, at very least someone should update the docs or do something to that
extent. Right now this is "an issue with a known unexplained workaround",
not a solved issue by any means.

On Fri, 7 Feb 2020, 1:13 pm James Hannah, [e-mail] wrote:

FWIW I don't believe the issue I raised has been resolved, the other
comments refer specifically to an EKS issue, which may be related to the
original issue I reported which has nothing to do with EKS.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/metrics-server/issues/319?email_source=notifications&email_token=AAB5MS5MADSXJCFTDDOS3XDRBVNATA5CNFSM4I4Y2FJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELC3WAQ#issuecomment-583383810,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAB5MSYWRZAVXPWZH4FQHK3RBVNATANCNFSM4I4Y2FJA
.

[serathius]

Sorry for not noticing that issue was hijacked by EKS @jiphex. Metrics Server depends on https://github.com/kubernetes/apiserver to serve its API and configure certificates. I would encourage you to ask there.

[david-kirby]

Amazing, @BlenderDude! This worked like a charm!! We had multiple roleArns in the aws-auth and except for the one with username: system:node:{{EC2PrivateDNSName}}, the others did not have a username specified. After adding the field to each one, the metrics-server started to work. The HorizontalPodAutoscaling also started to work:
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
Thanks a bunch!
We also have a ticket opened with AWS for this issue, I will let them know about this workaround so they can investigate what's going on behind the scenes. Perhaps something like the thing you mentioned with a parsing error.

@costimuraru Did you ever hear back on from AWS on why this is? I too was experiencing the x509 and simply added a "username": "cluster-admin" for each entry under the mapRoles path and now metrics-server is running correctly