metrics-server CrashLoopBackOff on v1.11.1

[GregSilverman]

Using 4-nodes on RHEL 7.

Did the following:

1. Open port 10250 on all nodes
2. Applied #97 to address no source found issue
3. Ran kubectl create -f deploy/1.8+/
4. Ran kubectl get pods -n=kube-system

and got CrashLoopBackOff on metrics-server

NAME                                          READY     STATUS             RESTARTS   AGE
coredns-78fcdf6894-4q7ct                      1/1       Running            10         7d
coredns-78fcdf6894-7tj52                      1/1       Running            10         7d
etcd-thalia0.ahc.umn.edu                      1/1       Running            0          7d
kube-apiserver-thalia0.ahc.umn.edu            1/1       Running            0          7d
kube-controller-manager-thalia0.ahc.umn.edu   1/1       Running            0          7d
kube-flannel-ds-amd64-78hbk                   1/1       Running            0          7d
kube-flannel-ds-amd64-gdttr                   1/1       Running            0          7d
kube-flannel-ds-amd64-rzhm2                   1/1       Running            0          7d
kube-flannel-ds-amd64-xc2n7                   1/1       Running            0          7d
kube-proxy-b86kn                              1/1       Running            0          7d
kube-proxy-g27sk                              1/1       Running            0          7d
kube-proxy-rtgtp                              1/1       Running            0          7d
kube-proxy-x2pp7                              1/1       Running            0          7d
kube-scheduler-thalia0.ahc.umn.edu            1/1       Running            0          7d
kubernetes-dashboard-7b7cb74c5c-wgt8f         1/1       Running            0          6d
metrics-server-85ff8f7b84-2x5th               0/1       CrashLoopBackOff   8          23m

1. Ran kubectl -n kube-system logs $(kubectl get pods --namespace=kube-system -l k8s-app=metrics-server -o name)

and got output:

I0828 19:26:41.686932       1 heapster.go:71] /metrics-server --source=kubernetes:https://kubernetes.default
I0828 19:26:41.687023       1 heapster.go:72] Metrics Server version v0.2.1
I0828 19:26:41.687360       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version
I0828 19:26:41.687388       1 configs.go:62] Using kubelet port 10255
E0828 19:27:01.692571       1 kubelet.go:331] Failed to load nodes: Get https://kubernetes.default/api/v1/nodes: dial tcp: lookup kubernetes.default on 10.96.0.10:53: read udp 10.244.2.4:34644->10.96.0.10:53: read: no route to host
I0828 19:27:01.692700       1 heapster.go:128] Starting with Metric Sink
I0828 19:27:02.500852       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W0828 19:27:04.381151       1 authentication.go:222] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
F0828 19:27:04.381187       1 heapster.go:97] Could not create the API server: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.96.0.1:443: getsockopt: no route to host

Not quite sure how to proceed with this error. I found a similar issue (#80) wrt 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA', but this does not seem relevant to my situation. The only similarity is I don't know what the values for YOUR_NS:YOUR_SA are.

[DirectXMan12]

Your cluster networking is messed up -- the metrics-server pod isn't able to connect to the kube API server. Make sure that your overlay network (if you have one) isn't blocking connections between pods and the API server.

[GregSilverman]

Hi, I thought something along these lines was happening given the last error. We are running flannel as the overlay network, so I am not sure where it would be blocking traffic on the k8s network though, since everything else appears to be working fine.

[GregSilverman]

Issue ended up being that flannel CNI was not working. Switched to calico.

[KielChan]

it's not clear, what do you mean messed up?
the prompt showed above messed me up too