Metrics-server: unable to fetch pod metrics for pod on Kubernetes 1.12 metrics server v0.3.1

[joshshn]

Hi, I am facing issue running metrics server on my kubernetes cluster.
My K8s Cluster information:

Ubuntu 16.04.4 LTS
Linux kernel 4.10.0-27
Docker CE version : 1.13.1
kubernetes version : 1.12.0 , created cluster using kubeadm (Bare-metal Cluster)
network CNI : cilium
metrics-server version : 0.3.1
1 master and 2 nodes

Metrics server was installed using the documentation provided using the command : kubectl create -f ./1.8+/
However metrics are not generated for the pods. Here are the logs from the pod. Are there any issues in configuring the cluster/metrics-server?

admin1@Master:~/new/cmmDockerKubernetes/Ansible/metrics-server/deploy$ kubectl logs -n kube-system metrics-server-65f476c68f-rs67h
I1018 08:13:03.812308       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2018/10/18 08:13:04 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/10/18 08:13:04 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1018 08:13:04.606416       1 serve.go:96] Serving securely on [::]:443
I1018 08:13:04.606733       1 reflector.go:202] Starting reflector *v1.Node (0s) from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I1018 08:13:04.606699       1 reflector.go:202] Starting reflector *v1.Pod (0s) from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I1018 08:13:04.606765       1 reflector.go:240] Listing and watching *v1.Node from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I1018 08:13:04.606774       1 reflector.go:240] Listing and watching *v1.Pod from github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130
I1018 08:13:04.616600       1 authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched"
I1018 08:13:04.616729       1 wrap.go:42] GET /: (2.790633ms) 403 [[Go-http-client/2.0] 10.38.42.101:33012]
I1018 08:13:04.625951       1 authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched"
I1018 08:13:04.626056       1 wrap.go:42] GET /: (233.639µs) 403 [[Go-http-client/2.0] 10.38.42.101:33012]
I1018 08:13:05.963883       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (463.526µs) 200 [[kube-apiserver/v1.12.0 (linux/amd64) kubernetes/0ed3388] 10.38.42.101:33040]
I1018 08:13:06.276230       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (1.909224ms) 200 [[kube-controller-manager/v1.12.0 (linux/amd64) kubernetes/0ed3388/system:serviceaccount:kube-system:resourcequota-controller] 10.38.42.101:33040]
E1018 08:13:06.457727       1 reststorage.go:144] unable to fetch pod metrics for pod default/ippod0-9b6cdfc9-5ttgh: no metrics known for pod
E1018 08:13:06.457754       1 reststorage.go:144] unable to fetch pod metrics for pod default/dbpod0-6c7555b846-xwhs4: no metrics known for pod
E1018 08:13:06.457762       1 reststorage.go:144] unable to fetch pod metrics for pod default/dbpod1-76c87c466-zqndm: no metrics known for pod
E1018 08:13:06.457770       1 reststorage.go:144] unable to fetch pod metrics for pod default/cppod-86579b9db7-6kjtw: no metrics known for pod

[joshshn]

From the logs it is observed as : authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched"
Issue was resolved by adding RBAC for the path "/"

[KielChan]

@joshshn have u came up with this: https://github.com/kubernetes-incubator/metrics-server/issues/161,
i have noooooo idea now

[joshshn]

@KielChan : No i have not come up with #161 . I did not have to create any extra rolebindings in order to get it running as suggested.

[tomastzn]

@joshshn Can elaborate on how you fixed the issue: Issue was resolved by adding RBAC for the path "/"?

[joshshn]

There was an existing a cluster role with the following definition, for which i created a cluster role binding. Will need to reduce the scope of the cluster role for security purposes in the future, but this solves the issue for me to continue.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: open-cluster-role
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:

kubectl create clusterrolebinding openrolebinding --clusterrole=open-cluster-role --user=system:anonymous --group=system:unauthenticated

[KielChan]

finally, i resolved this problem by fix node network, cos' node network used cni plugin no flannel or any others known, node cannot access serviceIP by this cni plugin.
later, i bind the pod to master, use traditional flannel to fix it.
cluster role or any other RBAC are not my root causes.

[akshaya-git]

finally, i resolved this problem by fix node network, cos' node network used cni plugin no flannel or any others known, node cannot access serviceIP by this cni plugin.
later, i bind the pod to master, use traditional flannel to fix it.
cluster role or any other RBAC are not my root causes.

KielChan, can you share detailed steps of how do you do that?

[kc004]

@KielChan Please provide us details to solve this issue.

[KielChan]

I have review metrics-server source code, and I found that metrics-server in pod should connect kube-apiserver by two method, in-cluster method which means pod for metrics-server can access kube-apiserver by ping 10.96.0.1(the cluster IP for kubernetes service), or metrics-server can access kube-apiserver by providing kubeconfig file, also we should provide kubeconfig file for authoz and authn.If your pod in flannel network or which else can access 10.96.0.1, it must be in-cluster case, or your metrics-server setup with kubeconfig file, it can access any kube-apiserver IP or LB VIP for kube-apiserver.