Metrics-server: http: TLS handshake error from 172.30.117.64:25970: EOF

[XiaoMuYi]

$ kubectl logs -f metrics-server-589cc698c4-fbw5t -n kube-system
I0930 02:55:01.584798 1 logs.go:49] http: TLS handshake error from 172.30.117.64:25830: EOF
I0930 02:55:04.354311 1 logs.go:49] http: TLS handshake error from 172.30.117.64:25904: EOF
I0930 02:55:04.890066 1 logs.go:49] http: TLS handshake error from 172.30.117.64:25914: EOF
I0930 02:55:06.195554 1 logs.go:49] http: TLS handshake error from 172.30.51.128:14770: EOF

$ cat metrics-server-deployment.yaml
......
image: hexun/metrics-server-amd64:v0.3.0
imagePullPolicy: Always
volumeMounts:
- name: tmp-dir
mountPath: /tmp
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP

At present, it is possible to obtain resources by using kubectl top node / pod, but the log frequently appears "http: TLS handshake error from 172.30.117.64:52590:EOF", this problem can be solved?

[gjmzj]

in your cluster, is kubelet running with flags '--anonymous-auth=false' or not?

[XiaoMuYi]

@gjmzj yes,I have

[gjmzj]

my question is 'how can the metrics-server get authorized by the kubelet server '

when i upgrade the 'metrics-server' to 0.3.0, i get error log of 'metrics-server' POD:

E0930 02:45:31.619297       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:10.xx.yy.41: unable to fetch metrics from Kubelet 10.xx.yy.41 (10.xx.yy.41): request failed - "401 Unauthorized", response: "Unauthorized"

[XiaoMuYi]

@gjmzj

git clone https://github.com/kubernetes-incubator/metrics-server.git
cd ./metrics-server/deploy/1.8+

Is it through such a deployment?

[gjmzj]

through manifests in deploy/1.8+
yes, also i did a little changes:

      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      - name: ssl-dir
        secret:
          secretName: metrics-server-secrets
          defaultMode: 0400
      containers:
      - name: metrics-server
        #image: k8s.gcr.io/metrics-server-amd64:v0.3.0
        image: mirrorgooglecontainers/metrics-server-amd64:v0.3.1
        imagePullPolicy: IfNotPresent
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        - --logtostderr=true
        - --tls-cert-file=/etc/ssl/ms-cert
        - --tls-private-key-file=/etc/ssl/ms-key
        - --v=2
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
        - name: ssl-dir
          mountPath: /etc/ssl

i created 'metrics-server-secrets' before deployment

kubectl create secret generic -n kube-system metrics-server-secrets \
            --from-file=ca=ca.pem \
            --from-file=ms-key=metrics-server-key.pem \
            --from-file=ms-cert=metrics-server.pem"

[DirectXMan12]

@gjmzj metrics-server attempts to authorize itself using token authentication. Please ensure that you're running your kubelets with webhook token authentication turned on.

[DirectXMan12]

@XiaoMuYi looks like something's connecting and then immediately disconnecting. If everything is working, I wouldn't worry about it.

[alanh0vx]

I have similar situation

http: TLS handshake error from 192.168.133.64:51926:EOF

192.168.133.64 is the internal ip address of api-server

from api-server log

E1214 02:06:35.625504       1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1214 02:07:05.573958       1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.109.179.236:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1214 02:07:05.658841       1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

kubelet version is 1.12.3
metrics-server 0.3.1

i have another clusters with the same version and configuration, metrics-server works just fine

[tvildo]

metrics-server v0.3.1
calico v3.4
kubernetes v1.13.1
I was using calico CNI. TLS message is big packet and MTU in calico was wrong so I changed it according https://docs.projectcalico.org/v3.4/usage/configuration/mtu
I changed MTU from 1550 to 1450
and everything is fixed. It was not metrics-server issue in my case 👍

[alanh0vx]

thanks, turned out it's MTU issue

from calico config, it was 1500 while the interface is 1450

finally kubectl edit configmap calico-config -n kube-system and change the MTU value from 1500 to 1430

[zhangxingdeppon]

metrics-server v0.3.1
calico v3.4
kubernetes v1.13.1
I was using calico CNI. TLS message is big packet and MTU in calico was wrong so I changed it according https://docs.projectcalico.org/v3.4/usage/configuration/mtu
I changed MTU from 1550 to 1450
and everything is fixed. It was not metrics-server issue in my case 👍

@tvildo hi, buddy, why does the MTU configuration cause the problem? can you explain it for me? , I met the same problem, I want to kown the reason so that I can operation it , thanks

[Juludut]

Hi there,
Writing this here if it may help anyone else hitting this problem.

We had similar problems and it was also a MTU issue : while upgrading the underlying virtualization infrastructure of the kubernetes nodes, the nodes lost their MTU (8950) for a too little MTU (1450), which caused TLS packets to be too large - thus these messages (and a lot of other erratic errors).
Putting back the right MTU on masters/workers nodes solved the issue.