Metrics-server: RBAC Deny when requesting metrics

Created on 8 Feb 2018 · 15Comments · Source: kubernetes-sigs/metrics-server

Hello,

I have a 3 node cluster on Virtual Machines (Kubernetes Version 1.9.0)

I added the following flags to the Kube API Server to enable aggregation

--requestheader-client-ca-file=/var/lib/kubernetes/ca.pem \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --proxy-client-cert-file=/var/lib/kubernetes/kube-apiserver.pem \
  --enable-aggregator-routing=true \
  --proxy-client-key-file=/var/lib/kubernetes/kube-apiserver-key.pem

Since this is a non production set up, i am using the same CA and Certs which i use for Kube API Server.

Then i deployed the manifests located at https://github.com/kubernetes-incubator/metrics-server/tree/master/deploy

When running

On Metrics Server Pod

 Rahul@rahul-mbp  ~/dev/2018-sandbox/k8s-on-vagrant/metrics-server   master  kubectl logs metrics-server-bb9ffc6b8-n8pt5 -n=kube-system
I0208 02:38:27.998984       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0208 02:38:27.999186       1 heapster.go:72] Metrics Server version v0.2.1
I0208 02:38:27.999374       1 configs.go:61] Using Kubernetes client with master "https://10.32.0.1:443" and version
I0208 02:38:27.999429       1 configs.go:62] Using kubelet port 10255
I0208 02:38:28.000372       1 heapster.go:128] Starting with Metric Sink
I0208 02:38:28.107149       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0208 02:38:28.384409       1 heapster.go:101] Starting Heapster API server...
[restful] 2018/02/08 02:38:28 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/02/08 02:38:28 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0208 02:38:28.385309       1 serve.go:85] Serving securely on 0.0.0.0:443
E0208 02:49:13.068371       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=37, ErrCode=NO_ERROR, debug=""
E0208 02:49:13.068739       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=37, ErrCode=NO_ERROR, debug=""
E0208 02:49:13.068953       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=37, ErrCode=NO_ERROR, debug=""
E0208 02:49:13.069191       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=37, ErrCode=NO_ERROR, debug=""
E0208 02:49:13.069387       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=37, ErrCode=NO_ERROR, debug=""
E0208 02:49:13.069673       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1603&timeoutSeconds=323&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:49:13.069716       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1603&timeoutSeconds=592&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:49:13.069835       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1603&timeoutSeconds=482&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:49:13.069878       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to watch *v1.Namespace: Get https://10.32.0.1:443/api/v1/namespaces?resourceVersion=1392&timeoutSeconds=347&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:49:13.069913       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to watch *v1.Pod: Get https://10.32.0.1:443/api/v1/pods?resourceVersion=1469&timeoutSeconds=478&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:49:16.917255       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0208 02:49:16.928352       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list pods at the cluster scope
E0208 02:49:16.928398       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0208 02:49:16.928433       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0208 02:49:16.928456       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0208 02:50:36.827934       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:50:36.831242       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:50:36.833029       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:50:36.833845       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:50:36.834837       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:50:36.835548       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1621&timeoutSeconds=426&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:50:36.835749       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1621&timeoutSeconds=536&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:50:36.835862       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to watch *v1.Pod: Get https://10.32.0.1:443/api/v1/pods?resourceVersion=1603&timeoutSeconds=384&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:50:36.835994       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1621&timeoutSeconds=376&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:50:36.836131       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to watch *v1.Namespace: Get https://10.32.0.1:443/api/v1/namespaces?resourceVersion=1603&timeoutSeconds=459&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:50:40.546884       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0208 02:50:40.547699       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0208 02:50:40.547731       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0208 02:50:40.547759       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0208 02:52:11.018945       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=27, ErrCode=NO_ERROR, debug=""
E0208 02:52:11.019294       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=27, ErrCode=NO_ERROR, debug=""
E0208 02:52:11.019906       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=27, ErrCode=NO_ERROR, debug=""
E0208 02:52:11.020206       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=27, ErrCode=NO_ERROR, debug=""
E0208 02:52:11.020512       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=27, ErrCode=NO_ERROR, debug=""
E0208 02:52:11.020821       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1646&timeoutSeconds=522&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:11.020873       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1646&timeoutSeconds=579&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:11.020909       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to watch *v1.Namespace: Get https://10.32.0.1:443/api/v1/namespaces?resourceVersion=1621&timeoutSeconds=372&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:11.024620       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to watch *v1.Pod: Get https://10.32.0.1:443/api/v1/pods?resourceVersion=1621&timeoutSeconds=540&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:11.024702       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1646&timeoutSeconds=393&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:14.954742       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0208 02:52:14.955042       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list pods at the cluster scope
E0208 02:52:14.955075       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0208 02:52:14.955181       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope
E0208 02:52:14.965734       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope
E0208 02:52:19.873651       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:52:19.874168       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:52:19.874504       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:52:19.876888       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:52:19.877350       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=29, ErrCode=NO_ERROR, debug=""
E0208 02:52:19.877753       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to watch *v1.Pod: Get https://10.32.0.1:443/api/v1/pods?resourceVersion=1646&timeoutSeconds=447&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:19.877809       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1646&timeoutSeconds=327&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:19.877844       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1646&timeoutSeconds=576&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:19.877880       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to watch *v1.Namespace: Get https://10.32.0.1:443/api/v1/namespaces?resourceVersion=1646&timeoutSeconds=593&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:19.877920       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1646&timeoutSeconds=578&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0208 02:52:23.954419       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope: [clusterrole.rbac.authorization.k8s.io "system:auth-delegator" not found, clusterrole.rbac.authorization.k8s.io "system:metrics-server" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]
E0208 02:52:23.954480       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list pods at the cluster scope: [clusterrole.rbac.authorization.k8s.io "system:auth-delegator" not found, clusterrole.rbac.authorization.k8s.io "system:metrics-server" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]
E0208 02:52:23.958895       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope: [clusterrole.rbac.authorization.k8s.io "system:metrics-server" not found, clusterrole.rbac.authorization.k8s.io "system:auth-delegator" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]
E0208 02:52:23.960312       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/processors/namespace_based_enricher.go:85: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list namespaces at the cluster scope: [clusterrole.rbac.authorization.k8s.io "system:auth-delegator" not found, clusterrole.rbac.authorization.k8s.io "system:metrics-server" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]
E0208 02:52:23.968730       1 reflector.go:205] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:metrics-server" cannot list nodes at the cluster scope: [clusterrole.rbac.authorization.k8s.io "system:auth-delegator" not found, clusterrole.rbac.authorization.k8s.io "system:metrics-server" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]
W0208 02:55:26.283956       1 x509.go:168] x509: subject with cn=kubernetes is not in the allowed list: [aggregator]
W0208 02:55:35.160885       1 x509.go:168] x509: subject with cn=kubernetes is not in the allowed list: [aggregator]
W0208 02:55:44.656033       1 x509.go:168] x509: subject with cn=kubernetes is not in the allowed list: [aggregator]
W0208 02:56:26.260001       1 x509.go:168] x509: subject with cn=kubernetes is not in the allowed list: [aggregator]
W0208 02:57:26.320622       1 x509.go:168] x509: subject with cn=kubernetes is not in the allowed list: [aggregator]
W0208 02:58:26.315669       1 x509.go:168] x509: subject with cn=kubernetes is not in the allowed list: [aggregator]

On Kube API Server:

Feb 08 03:49:09 k8s-master kube-apiserver[11124]: I0208 03:49:09.868936   11124 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:metrics-server" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "list" resource "pods" cluster-wide
Feb 08 03:49:09 k8s-master kube-apiserver[11124]: I0208 03:49:09.868984   11124 wrap.go:42] GET /api/v1/pods?resourceVersion=0: (26.023079ms) 403 [[metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format] 172.178.205.102:49548]
Feb 08 03:49:09 k8s-master kube-apiserver[11124]: I0208 03:49:09.869161   11124 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:metrics-server" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "list" resource "namespaces" cluster-wide
Feb 08 03:49:09 k8s-master kube-apiserver[11124]: I0208 03:49:09.868892   11124 wrap.go:42] GET /api/v1/nodes?resourceVersion=0: (4.397189ms) 403 [[metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format] 172.178.205.102:49548]
Feb 08 03:49:09 k8s-master kube-apiserver[11124]: I0208 03:49:09.869215   11124 wrap.go:42] GET /api/v1/namespaces?resourceVersion=0: (25.664566ms) 403 [[metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format] 172.178.205.102:49548]
Feb 08 03:49:09 k8s-master kube-apiserver[11124]: I0208 03:49:09.873930   11124 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:metrics-server" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "list" resource "nodes" cluster-wide
Feb 08 03:49:09 k8s-master kube-apiserver[11124]: I0208 03:49:09.873996   11124 wrap.go:42] GET /api/v1/nodes?resourceVersion=0: (29.451749ms) 403 [[metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format] 172.178.205.102:49548]
Feb 08 03:49:10 k8s-master kube-apiserver[11124]: I0208 03:49:10.874144   11124 wrap.go:42] GET /api/v1/namespaces?resourceVersion=0: (758.615µs) 200 [[metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format] 172.178.205.102:49548]
Feb 08 03:49:10 k8s-master kube-apiserver[11124]: I0208 03:49:10.875424   11124 wrap.go:42] GET /api/v1/nodes?resourceVersion=0: (606.465µs) 200 [[metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format] 172.178.205.102:49548]
Feb 08 03:49:10 k8s-master kube-apiserver[11124]: I0208 03:49:10.876138   11124 wrap.go:42] GET /api/v1/nodes?resourceVersion=0: (252.071µs) 200 [[metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format] 172.178.205.102:49548]
Feb 08 03:49:10 k8s-master kube-apiserver[11124]: I0208 03:49:10.877562   11124 wrap.go:42] GET /api/v1/nodes?resourceVersion=0: (230.239µs) 200 [[metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format] 172.178.205.102:49548]
Feb 08 03:49:10 k8s-master kube-apiserver[11124]: I0208 03:49:10.878229   11124 wrap.go:42] GET /api/v1/pods?resourceVersion=0: (1.375966ms) 200 [[metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format] 172.178.205.102:49548]

kubectl just responds with
Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "kubernetes" cannot list nodes.metrics.k8s.io at the cluster scope.

I don't know where the user "kubernetes" is picked up from , my admin has a CN named admin, the certificate is signed by an issuer with CN(kubernetes)

I thought the deployment manifests of metrics server would address its RBAC requirements, what other permissions does the metric server need?

Source

rahulmishra

Most helpful comment

I also get error about permissions:

Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "system:anonymous" cannot list nodes.metrics.k8s.io at the cluster scope.

I use admin certificate generated by kubespray:

openssl x509 -in client.pem -text -noout
...
Subject: CN=kube-admin-my-app, O=system:masters

We can temporary fix it by allowing system:anonymous to access metrics:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: view-metrics
rules:
- apiGroups:
    - metrics.k8s.io
  resources:
    - pods
    - nodes
  verbs:
    - get
    - list
    - watch

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: view-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view-metrics
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: system:anonymous

Paxa on 4 Jul 2018

👍19

All 15 comments

Looks like your RBAC setup is messed up -- look at the messages like clusterrole.rbac.authorization.k8s.io "system:auth-delegator" not found. Those should be part of the default Kubernetes RBAC setup, but you might have removed them or they might not have been created for some reason. Please make certain your RBAC is functioning normally in your cluster.

DirectXMan12 on 9 Feb 2018

It seems I need to dig deeper because I just bootstrapped a fresh cluster , the cluster role exists but it still doesn’t work

Will share my experience here as it unfolds

rahulmishra on 9 Feb 2018

So, i think i have narrowed it down to two sets of issues

Issue 1:
kubectl cannot list node/pod metrics, for some reason when i enable the Aggregation Server , the Kube API Server sees the user as anonymous, when i disable Aggregation Server, the user belongs to system:masters group. I believe this is happening because of me not fully understanding the 3 CA's as it relates to API builder. Working on that

 ✘ Rahul@rahul-mbp  ~/dev/2018-sandbox/k8s-on-vagrant/metrics-server   master  kubectl -v=12 get --raw "apis/metrics.k8s.io/v1beta1/nodes"
I0214 14:46:57.188672   50830 loader.go:357] Config loaded from file /Users/Rahul/.kube/config
I0214 14:46:57.191050   50830 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.9.0 (darwin/amd64) kubernetes/925c127" https://172.178.205.100:6443/apis/metrics.k8s.io/v1beta1/nodes
I0214 14:46:57.205985   50830 round_trippers.go:436] GET https://172.178.205.100:6443/apis/metrics.k8s.io/v1beta1/nodes 403 Forbidden in 14 milliseconds
I0214 14:46:57.206020   50830 round_trippers.go:442] Response Headers:
I0214 14:46:57.206032   50830 round_trippers.go:445]     X-Content-Type-Options: nosniff
I0214 14:46:57.206042   50830 round_trippers.go:445]     Content-Length: 281
I0214 14:46:57.206051   50830 round_trippers.go:445]     Content-Type: application/json
I0214 14:46:57.206061   50830 round_trippers.go:445]     Date: Wed, 14 Feb 2018 19:46:57 GMT
I0214 14:46:57.206136   50830 request.go:873] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes.metrics.k8s.io is forbidden: User \"system:anonymous\" cannot list nodes.metrics.k8s.io at the cluster scope.","reason":"Forbidden","details":{"group":"metrics.k8s.io","kind":"nodes"},"code":403}
I0214 14:46:57.206816   50830 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "nodes.metrics.k8s.io is forbidden: User \"system:anonymous\" cannot list nodes.metrics.k8s.io at the cluster scope.",
  "reason": "Forbidden",
  "details": {
    "group": "metrics.k8s.io",
    "kind": "nodes"
  },
  "code": 403
}]
F0214 14:46:57.206848   50830 helpers.go:119] Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "system:anonymous" cannot list nodes.metrics.k8s.io at the cluster scope.

Issue 2
The metrics server cannot list details from the API Server

E0214 19:21:40.720953       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1106&timeoutSeconds=365&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0214 19:21:40.721014       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1106&timeoutSeconds=503&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0214 19:21:40.721081       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/heapster.go:254: Failed to watch *v1.Pod: Get https://10.32.0.1:443/api/v1/pods?resourceVersion=996&timeoutSeconds=387&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused
E0214 19:21:40.723098       1 reflector.go:315] github.com/kubernetes-incubator/metrics-server/metrics/util/util.go:52: Failed to watch *v1.Node: Get https://10.32.0.1:443/api/v1/nodes?resourceVersion=1106&timeoutSeconds=440&watch=true: dial tcp 10.32.0.1:443: getsockopt: connection refused

This seems odd, because i can make calls to the API Server using the Pod Service Account Token just fine.

0214 19:16:09.272349       1 heapster.go:128] Starting with Metric Sink
I0214 19:16:09.423985       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0214 19:16:09.568479       1 heapster.go:101] Starting Heapster API server...

Here are the relevant roles and bindings which are reported as not found in original post

kubectl describe clusterrole system:metrics-server
Name:         system:metrics-server
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"system:metrics-server","namespace":""},"rules":[...
PolicyRule:
  Resources               Non-Resource URLs  Resource Names  Verbs
  ---------               -----------------  --------------  -----
  deployments.extensions  []                 []              [get list watch]
  namespaces              []                 []              [get list watch]
  nodes                   []                 []              [get list watch]
  pods                    []                 []              [get list watch]

kubectl describe clusterrolebinding system:metrics-server
Name:         system:metrics-server
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"system:metrics-server","namespace":""},"r...
Role:
  Kind:  ClusterRole
  Name:  system:metrics-server
Subjects:
  Kind            Name            Namespace
  ----            ----            ---------
  ServiceAccount  metrics-server  kube-system

@piosz @DirectXMan12 Any pointers will be very appreciated

rahulmishra on 14 Feb 2018

So, I found the solution to my issue.

First issue got resolved when i configured the following flags correctly.

--requestheader-client-ca-file=
--proxy-client-cert-file=
--proxy-client-key-file=

My Second issue was unique to my setup and occured because i was deploying the metrics pod "before" i set up the pod routes on master. My master nodes does not schedule pods and does not know how to get to pods.

Hopefully it helps someone else as well.

rahulmishra on 16 Feb 2018

🎉1 👍1

I am facing the same issue after setting below certificate and configuration for metrics server

--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem

--requestheader-extra-headers-prefix=X-Remote-Extra-

--requestheader-group-headers=X-Remote-Group

--requestheader-username-headers=X-Remote-User

--enable-aggregator-routing=true
other changes required for the same
proxy client certificate configuration is
--proxy-client-cert-file=/etc/kubernetes/ssl/apiserver.pem

--proxy-client-key-file=/etc/kubernetes/ssl/apiserver-key.pem

yashubh on 20 Mar 2018

complete logs after deployment of metrics server (kube8 version 1.9)

I0319 17:06:34.038557 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0319 17:06:34.038627 1 heapster.go:72] Metrics Server version v0.2.0
I0319 17:06:34.038811 1 configs.go:61] Using Kubernetes client with master "https://10.241.0.1:443" and version
I0319 17:06:34.038828 1 configs.go:62] Using kubelet port 10255
I0319 17:06:34.039693 1 heapster.go:128] Starting with Metric Sink
I0319 17:06:34.587942 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0319 17:06:34.781024 1 heapster.go:101] Starting Heapster API server...
[restful] 2018/03/19 17:06:34 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/03/19 17:06:34 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0319 17:06:34.782137 1 serve.go:85] Serving securely on 0.0.0.0:443
W0319 17:06:39.048609 1 x509.go:168] x509: subject with cn=kube-apiserver is not in the allowed list: [aggregator]
W0319 17:06:54.657665 1 x509.go:168] x509: subject with cn=kube-apiserver is not in the allowed list: [aggregator]
W0319 17:07:24.688893 1 x509.go:168] x509: subject with cn=kube-apiserver is not in the allowed list: [aggregator]
W0319 17:07:39.071042 1 x509.go:168] x509: subject with cn=kube-apiserver is not in the allowed list: [aggregator]
W0319 17:07:51.683204 1 x509.go:168] x509: subject with cn=kube-apiserver is not in the allowed list: [aggregator]
W0319 17:07:54.726782 1 x509.go:168] x509: subject with cn=kube-apiserver is not in the allowed list: [aggregator]
W0319 17:08:24.763838 1 x509.go:168] x509: subject with cn=kube-apiserver is not in the allowed list: [aggregator]

@rahulmishra any help appreciated

yashubh on 20 Mar 2018

temporary solution for above issue is set below mentioned flag-

--authorization-mode=AlwaysAllow
this flag will skip any authorization in kube8 setup.

yashubh on 22 Mar 2018

👍1

Your certificates have the wrong CN. You've either set up your certificates incorrectly, or passed the wrong certificate somewhere -- x509: subject with cn=kube-apiserver is not in the allowed list: [aggregator]

DirectXMan12 on 23 Mar 2018

@DirectXMan12

by default there is no requestheader-client-ca-file in kube8 version 1.9 setup , is there any specific flag given in metrics server to generate the same ca file? setup installation done via kargo installer , any idea which certificate i have to pass in requestheader-client-ca-file.

yashubh on 23 Mar 2018

@yashubh Sorry for the delay. You need to generate the requestheader-client-ca file and use the CA to sign the aggregator's cert.

You also need to include a flag in your API server configuration
--requestheader-allowed-names=aggregator,<> \

Atleast, thats the way i have made it work

rahulmishra on 26 Mar 2018

I also get error about permissions:

Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "system:anonymous" cannot list nodes.metrics.k8s.io at the cluster scope.

I use admin certificate generated by kubespray:

openssl x509 -in client.pem -text -noout
...
Subject: CN=kube-admin-my-app, O=system:masters

We can temporary fix it by allowing system:anonymous to access metrics:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: view-metrics
rules:
- apiGroups:
    - metrics.k8s.io
  resources:
    - pods
    - nodes
  verbs:
    - get
    - list
    - watch

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: view-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view-metrics
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: system:anonymous

Paxa on 4 Jul 2018

👍19

@Paxa Thanks, solved my problem!

raymondzhaoy on 26 Dec 2018

Same problem and inspired by @rahulmishra I solved it
vim metrics-server-csr.json

{
     "CN": "system:metrics-server",
     "hosts": [],
     "key": {
       "algo": "rsa",
       "size": 2048
     },
"names": [ {
         "C": "CN",
         "ST": "Peking",
         "L": "Peking",
         "O": "k8s",
         "OU": "system"
} ]

}

cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=ca-config.json -profile=kubernetes metrics-server-csr.json | cfssljson -bare metrics-server

Add config to kubelet

--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem --requestheader-allowed-names=aggregator,metrics-server --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/etc/kubernetes/ssl/metrics-server.pem --proxy-client-key-file=/etc/kubernetes/ssl/metrics-server-key.pem

xuchenCN on 10 Jan 2019

I also get error about permissions:

Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "system:anonymous" cannot list nodes.metrics.k8s.io at the cluster scope.

I use admin certificate generated by kubespray:

openssl x509 -in client.pem -text -noout
...
Subject: CN=kube-admin-my-app, O=system:masters

We can temporary fix it by allowing system:anonymous to access metrics:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: view-metrics
rules:
- apiGroups:
    - metrics.k8s.io
  resources:
    - pods
    - nodes
  verbs:
    - get
    - list
    - watch

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: view-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view-metrics
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: system:anonymous

Great!

xunknown on 3 Apr 2019

So, I found the solution to my issue.

First issue got resolved when i configured the following flags correctly.

--requestheader-client-ca-file=
--proxy-client-cert-file=
--proxy-client-key-file=

My Second issue was unique to my setup and occured because i was deploying the metrics pod "before" i set up the pod routes on master. My master nodes does not schedule pods and does not know how to get to pods.

Hopefully it helps someone else as well.

Solved my problem: I am missing the configuration: --proxy-client-cert-file and --proxy-client-key-file

$ kubectl top node
    Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "metrics.k8s.io" at the cluster scope