Metrics-server: x509: certificate signed by unknown authority

[max201920]

In Metric server pod getting following error.

[restful] 2018/10/01 06:35:24 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1001 06:35:24.584055 1 serve.go:96] Serving securely on [::]:443
E1001 06:36:24.615237 1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:kube: unable to fetch metrics from Kubelet kube (kube): Get https://kube:10250/stats/summary/: x509: certificate signed by unknown authority

E1001 06:37:24.543263 1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:kube: unable to fetch metrics from Kubelet kube (kube): Get https://kube:10250/stats/summary/: x509: certificate signed by unknown authority

[DirectXMan12]

It looks like you're trying to validate the kubelet serving certs without having them be signed by the main Kubernetes CA. If you're on metrics-server from before v0.3.0, this can be confirmed with the insecure=true source option (otherwise, you can pass a similar command line flag to v0.3.0+),

[max201920]

I deployed new kubernetes server 1.12 using kubeadm and then using kubectl create -f deploy/1.8+/ I created metric server , no custom changes were done in either kubelet or metric server of kubernetes API.

[DirectXMan12]

Please try using the --kubelet-insecure-tls flag for metrics-server, and see if that helps. Some of the installation tools don't set up certificates properly yet.

[albrr]

How and where do I use those flags? I have the same problem but can't find any information on how to use this

[k0nstantinv]

@albrr your metrics-server Deployment should be modified like this:
containers: - name: metrics-server image: k8s.gcr.io/metrics-server-amd64:v0.3.1 command: - /metrics-server - --kubelet-insecure-tls

[albrr]

thanks @k0nstantinv :)

[towolf]

@DirectXMan12 following recommendations I am employing a Kubelet CA separate from the main CA.

But now I get this all the time:

E1028 15:43:03.202456       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:kvm336: unable to fetch metrics from Kubelet kvm336 (10.109.0.3): Get https://10.109.0.3:10250/stats/summary/: x509: certificate signed by unknown authority,  ... ... ...]

The Apiserver itself can be configured to verify using the Kubelet CA.

--kubelet-certificate-authority=/etc/kubernetes/pki/kubelet-ca.pem

I suppose for metrics-server this would be possible too?

[rayanebel]

+1

[DirectXMan12]

Yes, we need a feature to support a separate kubelet CA.

@brancz this is becoming a decently common request

/kind feature
/priority important-soon

[rayanebel]

While waiting this new feature. It’s possible to mount the certificate inside the pod using a volume or something else ?

[brancz]

@rayanebel I don't think so as the same cert that is currently used by default for the kubelet is also used (and there it is valid) for the apiserver.

If anyone wants to give this a go, please feel free to open a PR :slightly_smiling_face: .

[metalmatze]

Just for completeness: We also need the same thing in the Prometheus Adapter.

[brancz]

@metalmatze feel free to open an issue there.

[Aiqbal1234]

Has any progress been made on this?
by setting the --kubelet-insecure-tls flag i can get autoscaling to work. But i still get the logs related to CA and not all metrics are available.

Do/should i need to regenerate the kubelet certificates (we use AWS and Kubeadm)? I cant see why i would need to do that, our pods are being deployed to the nodes... So the kubelet certificates should exist i believe.

btw i already have apiserver-kubelet-client.key and apiserver-kubelet-client.crt on master

[DirectXMan12]

there's hasn't been any PR for it yet, IIRC. It's on the list of things to do soon, though.

[desmatz]

I did the following (kubeadm ha cluster):
in accordance with:
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

1. Create a Certificate Signing Request

cat <<EOF | cfssl genkey - | cfssljson -bare kubelet-server
{
  "hosts": [
    "node-name-1",
    "node-name-2",
    ""...",
    "node-ip-1",
    "node-ip-2",
    "..."
  ],
  "CN": "kubelet-server",
  "key": {
    "algo": "ecdsa",
    "size": 256
  }
}
EOF

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: kubelet-server
spec:
  groups:
  - system:nodes
  - system:authenticated
  request: $(cat kubelet-server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

1. kubectl describe csr kubelet-server
2. kubectl certificate approve kubelet-server
3. kubectl get csr kubelet-server -o jsonpath='{.status.certificate}' | base64 --decode > kubelet-server.pem
4. Copy to all nodes in /var/lib/kubelet/pki/
5. Add to kubelet config.yaml tls cert & key
   see: https://godoc.org/k8s.io/kubernetes/pkg/kubelet/apis/config#KubeletConfiguration
   echo "tlsPrivateKeyFile: /var/lib/kubelet/pki/kubelet-server-key.pem" >> /var/lib/kubelet/config.yaml
   echo "tlsCertFile: /var/lib/kubelet/pki/kubelet-server.pem" >> /var/lib/kubelet/config.yaml
6. restart kubelet

systemctl daemon-reload && \
systemctl restart kubelet && \
systemctl status kubelet

1. view metrics-server logs. If ok
2. kubectl get hpa

[sarneaud]

Another kubeadm user here. I solved the problem by doing the following before running kubeadm init:

1. kubeadm --config config.yaml init phase certs all
2. Creating an X509 key in /var/lib/kubelet/pki/kubelet.key
3. Creating a cert signed by /etc/kubernetes/pki/ca.key, putting it in /var/lib/kubelet/pki/kubelet.crt

It's basically the same as @desmatz solution, but using the CA key directly instead of using the cert signing API.

[Tapa4ik]

@albrr your metrics-server Deployment should be modified like this:

      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
        command:
        - /metrics-server
        - --kubelet-insecure-tls

thank you so much!
Костя ты красавчик ;))

[ioanc]

security is a hot topic these days.
offering a possible solution by disabling security, using --kubelet-insecure-tls is not the right way to move forward.

@desmatz and @sarneaud solutions are the proper way to fix the issue.
Thank you guys !!!

[paphillon]

@ioanc From what I understood, these solutions intend to be planned in advance for use of metrics server, existing clusters with a separate CA for kubelet and or not using kubeadm do need support to specify kubelet CA public key as suggested by @DirectXMan12

[charlesguoooo]

when you use kubeadm deploy metrics-server, you can add serverTLSBootstrap: true to /var/lib/kubelet/config.yml ,it works for me. remember restart kubelet!

[Baykonur]

@DirectXMan12

It looks like you're trying to validate the kubelet serving certs without having them be signed by the main Kubernetes CA. If you're on metrics-server from before v0.3.0, this can be confirmed with the insecure=true source option (otherwise, you can pass a similar command line flag to v0.3.0+),

I have the same error:
manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:cored01: unable to fetch metrics from Kubelet cored01 (cored01): Get https://cored01:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:cored02: unable to fetch metrics from Kubelet cored02 (cored02): Get https://cored02:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:cored03: unable to fetch metrics from Kubelet cored03 (cored03): Get https://cored03:10250/stats/summary/: x509: certificate signed by unknown authority]

And the kubelet client certificate is signed by the main CA (kube-apiserver, or are we talking about another one?)
cored02 kubernetes # openssl verify -CAfile /etc/kubernetes/pki/ca.crt /var/lib/kubelet/pki/kubelet-client-current.pem /var/lib/kubelet/pki/kubelet-client-current.pem: OK

If I pass the --kubelet-insecure-tls, all works fine.

I used kubeadm to bootstrap the cluster (single master, 2 workers), no custom setup whatsoever, kubernetes version is 1.13.4.

on the master:
````
cored01 pki # ps -ef | grep kube-apiserver
root 1329 1310 2 Mar08 ? 02:33:09 kube-apiserver --authorization-mode=Node,RBAC --advertise-address=10.14.10.116 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=172.17.224.0/19 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

cored01 pki # openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Mar 5 06:56:50 2019 GMT
Not After : Mar 2 06:56:50 2029 GMT
Subject: CN=kubernetes
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:d9:8f:9d:04:de:50:ee:73:24:28:50:1d:98:e7:
72:e0:5f:78:44:be:47:b3:f8:e6:fc:8e:08:f2:fd:
da:33:a3:2e:c2:7b:d1:1e:00:db:13:e6:24:68:3b:
be:8d:ea:31:ff:37:ee:59:68:03:6f:08:73:94:cc:
60:69:e1:0b:67:08:25:2b:f3:b9:f4:6e:de:2d:86:
27:3a:9c:7d:06:84:af:f1:09:e5:5b:fd:7f:4f:4d:
50:c3:6b:d5:91:48:f5:ad:59:56:ff:d3:0c:3f:c0:
fa:ef:29:5f:da:21:8d:5b:44:98:45:e6:7d:78:0b:
62:ba:ed:1c:0b:71:cd:86:e4:20:e7:b3:ab:72:a9:
64:93:c4:0f:32:aa:01:61:70:f7:8e:4f:c1:ce:63:
1d:a4:2c:36:6b:af:16:6e:16:f3:8f:43:55:80:6c:
c3:6e:d0:94:65:92:bd:f1:29:e3:1a:be:60:42:64:
a9:84:45:b7:6f:00:b6:8c:fb:ae:7a:90:28:08:d4:
6a:58:fb:4e:63:79:0f:01:a1:79:61:ea:04:8d:5c:
f1:59:fa:a6:6e:f4:93:06:9f:3c:0f:5c:26:53:f5:
7e:f6:9a:2b:f7:e9:d8:43:66:b1:09:0f:5c:62:c0:
51:0b:7c:cd:6b:3b:07:d1:0e:fb:4c:05:55:ab:8f:
db:7f
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
5b:ef:38:c4:5a:c8:95:a3:71:e4:3b:30:e4:33:79:42:6d:fc:
02:f6:dd:d5:0e:e2:d2:83:aa:6b:3a:3b:74:04:c4:06:61:41:
94:ad:4f:88:03:0a:09:b8:59:ad:30:b2:38:0b:4b:11:00:64:
62:17:50:2f:ce:05:49:c2:99:3a:fb:11:d3:79:9b:2b:35:4f:
5d:67:73:2d:a1:f7:64:47:a5:35:1c:ea:4f:cf:d0:be:79:9b:
9c:a4:32:fd:8f:70:25:98:a8:b9:f0:d2:f7:7e:f9:2b:06:b3:
6f:ab:ab:f7:f3:63:12:eb:2f:c9:13:b0:f1:b5:4f:aa:56:9a:
3a:3f:2d:51:4a:68:0f:39:bc:2a:ee:91:a0:7e:ec:ea:cd:b9:
81:d7:4c:90:af:48:0a:ec:e7:84:8a:3d:84:71:31:40:98:f1:
4a:71:bc:29:0f:2c:58:da:c6:bd:4e:c2:fb:9a:bc:41:3a:0a:
55:df:13:98:27:52:e7:7f:5b:5f:a2:12:51:fc:08:6c:64:54:
13:6e:b4:7b:c6:d8:1c:aa:00:a8:55:5b:da:67:e5:8e:d9:a4:
79:6c:e9:35:dc:63:cf:25:48:47:2a:ee:ac:fe:5e:58:2b:89:
ba:0a:3f:78:3e:a8:e9:7a:9b:d4:5c:1b:af:a3:29:d9:c1:d9:
49:b2:d5:ee
````

on the worker:
cored02 kubernetes # openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 0 (0x0) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=kubernetes Validity Not Before: Mar 5 06:56:50 2019 GMT Not After : Mar 2 06:56:50 2029 GMT Subject: CN=kubernetes Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:d9:8f:9d:04:de:50:ee:73:24:28:50:1d:98:e7: 72:e0:5f:78:44:be:47:b3:f8:e6:fc:8e:08:f2:fd: da:33:a3:2e:c2:7b:d1:1e:00:db:13:e6:24:68:3b: be:8d:ea:31:ff:37:ee:59:68:03:6f:08:73:94:cc: 60:69:e1:0b:67:08:25:2b:f3:b9:f4:6e:de:2d:86: 27:3a:9c:7d:06:84:af:f1:09:e5:5b:fd:7f:4f:4d: 50:c3:6b:d5:91:48:f5:ad:59:56:ff:d3:0c:3f:c0: fa:ef:29:5f:da:21:8d:5b:44:98:45:e6:7d:78:0b: 62:ba:ed:1c:0b:71:cd:86:e4:20:e7:b3:ab:72:a9: 64:93:c4:0f:32:aa:01:61:70:f7:8e:4f:c1:ce:63: 1d:a4:2c:36:6b:af:16:6e:16:f3:8f:43:55:80:6c: c3:6e:d0:94:65:92:bd:f1:29:e3:1a:be:60:42:64: a9:84:45:b7:6f:00:b6:8c:fb:ae:7a:90:28:08:d4: 6a:58:fb:4e:63:79:0f:01:a1:79:61:ea:04:8d:5c: f1:59:fa:a6:6e:f4:93:06:9f:3c:0f:5c:26:53:f5: 7e:f6:9a:2b:f7:e9:d8:43:66:b1:09:0f:5c:62:c0: 51:0b:7c:cd:6b:3b:07:d1:0e:fb:4c:05:55:ab:8f: db:7f Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment, Certificate Sign X509v3 Basic Constraints: critical CA:TRUE Signature Algorithm: sha256WithRSAEncryption 5b:ef:38:c4:5a:c8:95:a3:71:e4:3b:30:e4:33:79:42:6d:fc: 02:f6:dd:d5:0e:e2:d2:83:aa:6b:3a:3b:74:04:c4:06:61:41: 94:ad:4f:88:03:0a:09:b8:59:ad:30:b2:38:0b:4b:11:00:64: 62:17:50:2f:ce:05:49:c2:99:3a:fb:11:d3:79:9b:2b:35:4f: 5d:67:73:2d:a1:f7:64:47:a5:35:1c:ea:4f:cf:d0:be:79:9b: 9c:a4:32:fd:8f:70:25:98:a8:b9:f0:d2:f7:7e:f9:2b:06:b3: 6f:ab:ab:f7:f3:63:12:eb:2f:c9:13:b0:f1:b5:4f:aa:56:9a: 3a:3f:2d:51:4a:68:0f:39:bc:2a:ee:91:a0:7e:ec:ea:cd:b9: 81:d7:4c:90:af:48:0a:ec:e7:84:8a:3d:84:71:31:40:98:f1: 4a:71:bc:29:0f:2c:58:da:c6:bd:4e:c2:fb:9a:bc:41:3a:0a: 55:df:13:98:27:52:e7:7f:5b:5f:a2:12:51:fc:08:6c:64:54: 13:6e:b4:7b:c6:d8:1c:aa:00:a8:55:5b:da:67:e5:8e:d9:a4: 79:6c:e9:35:dc:63:cf:25:48:47:2a:ee:ac:fe:5e:58:2b:89: ba:0a:3f:78:3e:a8:e9:7a:9b:d4:5c:1b:af:a3:29:d9:c1:d9: 49:b2:d5:ee
metrics-server deployment.yaml:
... containers: - name: metrics-server image: k8s.gcr.io/metrics-server-amd64:v0.3.1 imagePullPolicy: Always command: - /metrics-server - --logtostderr - --kubelet-preferred-address-types=Hostname ...
If you require I can increase the log level.

I would appreciate if you can take your time to explain what the issue here is and how I can solve it.

[sarneaud]

@charlesguoooo

when you use kubeadm deploy metrics-server, you can add serverTLSBootstrap: true to /var/lib/kubelet/config.yml ,it works for me. remember restart kubelet!

That works for me too.

I added this to my kubeadm.conf:

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
serverTLSBootstrap: true

After starting the master or joining any nodes, the CSRs need to be approved. kubectl get csr gets a list. The pending ones for the nodes need to be approved with kubectl certificate approve $CSR_NAME. After a few minutes, certs are all good.

[Baykonur]

@sarneaud , thanks a lot. Your last comment saved me. @DirectXMan12 , you can ignore my comment.

What missing was kubelet-server-[datetime].pem under /var/lib/kubelet/pki in the worker node (also in master obviously), apparently metrics-server needs that certificate and it has to be signed by the /etc/kubernetes/pki/ca.crt which is in the master node, if you want to make things secure by switching off --kubelet-insecure-tls.

What I do not get is why metrics-server cannot use /var/lib/kubelet/pki/kubelet-client-[datetime].pem which is also signed by the same ca and it has the same cn as the kubelet-server.pem, except this one is used by kubelet to talk to kube-apiserver.

[Baykonur]

To answer my own question

What I do not get is why metrics-server cannot use /var/lib/kubelet/pki/kubelet-client-[datetime].pem which is also signed by the same ca and it has the same cn as the kubelet-server.pem, except this one is used by kubelet to talk to kube-apiserver.

The answer is here:

The client certificate provided by TLS bootstrapping is signed, by default, for client auth only, and thus cannot be used as serving certificates, or server auth.

However even though RotateKubeletServerCertificate is enabled by default, those CSRs will need to be approved (as described in this comment) once the existing server certificate expires --validity is 1 year.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[secret104278]

I found this issue is probably caused by https://github.com/kubernetes/kubeadm/issues/1223

My solution is using api proxy by changing the lines in https://github.com/kubernetes-incubator/metrics-server/blob/afbc8a030383caaa0888ea1cdedf7c079f18c048/pkg/sources/summary/client.go#L93-L94 to

Host:   "kubernetes.default.svc:443",
Path:   fmt.Sprintf("/api/v1/nodes/%s/proxy/stats/summary/", host),

rebuild the metrics-server and change https://github.com/kubernetes-incubator/metrics-server/blob/afbc8a030383caaa0888ea1cdedf7c079f18c048/deploy/1.8%2B/resource-reader.yaml#L9-L12 to

  resources:
  - pods
  - nodes
  - nodes/proxy
  - nodes/stats

Hope this will help someones who struggling to use --kubelet-insecure-tls like me

[paphillon]

@secret104278 - This is how I have solved it. I have different CA for kubelet certs in my setup.

• CA public key of kubelet CA authority. With this we don't need to set flag --kubelet-insecure-tls

kubectl create secret generic ca-kubelet-cert --from-file=./ca-kubelet.pem -n kube-system
Metrics server parameter
````

• --kubelet-certificate-authority=/etc/kubelet/ca/ca-kubelet.pem
  ````

[Devying]

@secret104278 - This is how I have solved it. I have different CA for kubelet certs in my setup.

• CA public key of kubelet CA authority. With this we don't need to set flag --kubelet-insecure-tls

kubectl create secret generic ca-kubelet-cert --from-file=./ca-kubelet.pem -n kube-system 

Metrics server parameter

- --kubelet-certificate-authority=/etc/kubelet/ca/ca-kubelet.pem

I Got this

W0624 11:37:52.647355       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0624 11:37:52.647424       1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x128 pc=0x16908a5]

goroutine 1 [running]:
sigs.k8s.io/metrics-server/cmd/metrics-server/app/options.Options.kubeletConfig(0xc0005361c8, 0xc000196d80, 0xc000075800, 0xc00030cf7c, 0x0, 0x0, 0x0, 0xdf8475800, 0x0, 0x280a, ...)
    /root/metrics-server/cmd/metrics-server/app/options/options.go:175 +0x105
sigs.k8s.io/metrics-server/cmd/metrics-server/app/options.Options.MetricsServerConfig(0xc0005361c8, 0xc000196d80, 0xc000075800, 0xc00030cf7c, 0x0, 0x0, 0x0, 0xdf8475800, 0x0, 0x280a, ...)
    /root/metrics-server/cmd/metrics-server/app/options/options.go:118 +0x11c
sigs.k8s.io/metrics-server/cmd/metrics-server/app.runCommand(0xc0000ce360, 0xc00007e1e0, 0x0, 0x0)
    /root/metrics-server/cmd/metrics-server/app/start.go:52 +0xa2
sigs.k8s.io/metrics-server/cmd/metrics-server/app.NewMetricsServerCommand.func1(0xc00050e780, 0xc0004b64b0, 0x0, 0x3, 0x0, 0x0)
    /root/metrics-server/cmd/metrics-server/app/start.go:37 +0x33
github.com/spf13/cobra.(*Command).execute(0xc00050e780, 0xc00003a0d0, 0x3, 0x3, 0xc00050e780, 0xc00003a0d0)
    /home/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0xc00050e780, 0xc000074180, 0x0, 0x0)
    /home/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
    /home/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
    /root/metrics-server/cmd/metrics-server/metrics-server.go:38 +0xb6