Metrics-server: not creating stats for kubernetes 1.15.1

[staticdev]

I am using master branch and when I try kubectl top node I get:
error: metrics not available yet. And when I try kubectl top pods I get: W0721 20:01:31.786615 21232 top_pod.go:266] Metrics not available for pod default/pod-deployment-57b99df6b4-khh84, age: 27h31m59.78660593s error: Metrics not available for pod default/pod-deployment-57b99df6b4-khh84, age: 27h31m59.78660593s

[shundezhang]

I am having issues running metrics server 0.3.3 on k8s 1.15.0 too. I am getting a lot of tls errors in metrics server log:
I0722 01:32:08.512698 1 log.go:172] http: TLS handshake error from 192.168.22.192:43146: EOF I0722 01:32:08.512847 1 log.go:172] http: TLS handshake error from 192.168.22.192:43150: EOF I0722 01:32:08.512911 1 log.go:172] http: TLS handshake error from 192.168.22.192:43148: EOF I0722 01:32:08.512946 1 log.go:172] http: TLS handshake error from 192.168.22.192:43152: EOF I0722 01:32:08.513372 1 log.go:172] http: TLS handshake error from 192.168.22.192:43154: EOF
The source IP 192.168.22.192 is the internal IP of one of the masters. In the api server log of a master I got:
E0722 01:43:55.398927 1 available_controller.go:407] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.28.56:443: Get https://10.101.28.56:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0722 01:44:00.399770 1 available_controller.go:407] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.28.56:443: Get https://10.101.28.56:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
seems there is a problem for the api server to talk to metrics server.
Just wondering if metrics server supports 1.15?

[metri]

Hi folks! Helped for me add options --enable-aggregator-routing=true in manifest kube-apiserver.

More details here https://github.com/kubernetes/kubernetes/issues/56430#issuecomment-361708175

[staticdev]

@metri I added to the deployment in metrics-server/deploy/1.8+/metrics-server-deployment.yaml

command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP

It now works.. does your change have the same effects?

[beanssoft]

/lifecycle stale

[shundezhang]

@staticdev how do you configure kube-apiserver?

[metri]

@staticdev
Yes. I made the same changes as you, but it did not help me (for some reason it seemed to me that you also made them, although you did not specify it). Then began to understand already more deeply, in kube-apiserver why doesn't work. As a result, I found a similar problem and the answer there.

PS: Sorry for my english.

[staticdev]

@shundezhang I am using default kubeadm configuration with calico 3.8 also on default.

[staticdev]

@metri I had to kubectl delete metrics-server/deploy/1.8+ and then kubectl create metrics-server/deploy/1.8+ after the changes. After some seconds you should be able to see results from kubectl top node (make sure metrics-server pods are running with kubectl get pods --all-namespaces).

[greg9702]

@staticdev Can you show your entire metrics-server-deployment.yaml?
I am using 1.15.0, tried everything and can not make it working.
Still got "error: metrics not available yet"

Edit.
Had to wait couple minutes and it works.
Thanks @staticdev for your advices.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[pierluigilenoci]

Have you tried to use this: https://github.com/kubernetes-incubator/metrics-server/issues/278#issuecomment-515370587

[staticdev]

@PierluigiLenociAkelius I used (as in the comments above):

command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[HYChou0515]

To be more explicit from @staticdev 's suggestion, I modified metrics-server/deploy/1.8+/metrics-server-deployment.yaml and solve this issue

...
apiVersion: extensions/v1beta1
...
spec:
  template:
    spec:
      containers:
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
...

[MIBc]

@shundezhang do you have solved the problem ?

[shundezhang]

@MIBc yes, it was an MTU issue. Had to adjust that in calico config.