Metrics-server: [EKS] metrics-server running but not getting any metrics

[cdenneen]

Spun up brand new demo EKS cluster.
Installed helm

Basically was following this guide (https://eksworkshop.com/scaling/deploy_hpa/)
Installed metrics-server and have the following status:

eksdemo/cloudformation - [master●] » k get apiservice v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  creationTimestamp: 2018-11-02T14:17:16Z
  labels:
    app: metrics-server
    chart: metrics-server-2.0.3
    heritage: Tiller
    release: metrics-server
  name: v1beta1.metrics.k8s.io
  resourceVersion: "78753"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: 0030a2d7-deaa-11e8-945e-120aad73bdd4
spec:
  caBundle: null
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: metrics
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: 2018-11-02T14:56:19Z
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

eksdemo/cloudformation - [master●] » kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[]}



md5-1a110b4fc9951071ef7dc2ae2fb28a64



eksdemo/cloudformation - [master●] » k -n metrics logs metrics-server-857bd4f45f-fp6f4
I1102 14:56:19.642622       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W1102 14:56:20.244978       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W1102 14:56:20.255204       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
[restful] 2018/11/02 14:56:20 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/11/02 14:56:20 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1102 14:56:20.306623       1 serve.go:96] Serving securely on [::]:443
E1102 14:56:28.601891       1 reststorage.go:144] unable to fetch pod metrics for pod default/php-apache-8699449574-4g5sd: no metrics known for pod
E1102 14:56:58.601742       1 reststorage.go:144] unable to fetch pod metrics for pod default/php-apache-8699449574-4g5sd: no metrics known for pod
E1102 14:57:28.601831       1 reststorage.go:144] unable to fetch pod metrics for pod default/php-apache-8699449574-4g5sd: no metrics known for pod
E1102 14:57:50.297812       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-240-42-42.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-42-42.ec2.internal (ip-10-240-42-42): Get https://ip-10-240-42-42:10250/stats/summary/: dial tcp: i/o timeout, unable to fully scrape metrics from source kubelet_summary:ip-10-240-52-17.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-52-17.ec2.internal (ip-10-240-52-17): Get https://ip-10-240-52-17:10250/stats/summary/: dial tcp: i/o timeout, unable to fully scrape metrics from source kubelet_summary:ip-10-240-62-245.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-62-245.ec2.internal (ip-10-240-62-245): Get https://ip-10-240-62-245:10250/stats/summary/: dial tcp: i/o timeout]
E1102 14:57:58.601410       1 reststorage.go:144] unable to fetch pod metrics for pod default/php-apache-8699449574-4g5sd: no metrics known for pod
E1102 14:58:20.290502       1 manager.go:102] unable to fully collect metrics: [unable to extract connection information for node "ip-10-240-62-62.ec2.internal": node ip-10-240-62-62.ec2.internal is not ready, unable to fully scrape metrics from source kubelet_summary:ip-10-240-42-42.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-42-42.ec2.internal (ip-10-240-42-42): Get https://ip-10-240-42-42:10250/stats/summary/: dial tcp: lookup ip-10-240-42-42 on 172.20.0.10:53: server misbehaving, unable to fully scrape metrics from source kubelet_summary:ip-10-240-52-17.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-52-17.ec2.internal (ip-10-240-52-17): Get https://ip-10-240-52-17:10250/stats/summary/: dial tcp: lookup ip-10-240-52-17 on 172.20.0.10:53: server misbehaving]
E1102 14:58:28.602294       1 reststorage.go:144] unable to fetch pod metrics for pod default/php-apache-8699449574-4g5sd: no metrics known for pod
E1102 14:58:58.602463       1 reststorage.go:144] unable to fetch pod metrics for pod default/php-apache-8699449574-4g5sd: no metrics known for pod
E1102 14:59:20.301718       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-240-52-17.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-52-17.ec2.internal (ip-10-240-52-17): Get https://ip-10-240-52-17:10250/stats/summary/: dial tcp: lookup ip-10-240-52-17 on 172.20.0.10:53: server misbehaving, unable to fully scrape metrics from source kubelet_summary:ip-10-240-42-42.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-42-42.ec2.internal (ip-10-240-42-42): Get https://ip-10-240-42-42:10250/stats/summary/: dial tcp: lookup ip-10-240-42-42 on 172.20.0.10:53: server misbehaving, unable to fully scrape metrics from source kubelet_summary:ip-10-240-62-62.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-62-62.ec2.internal (ip-10-240-62-62): Get https://ip-10-240-62-62:10250/stats/summary/: dial tcp: lookup ip-10-240-62-62 on 172.20.0.10:53: server misbehaving]
E1102 14:59:28.602524       1 reststorage.go:144] unable to fetch pod metrics for pod default/php-apache-8699449574-4g5sd: no metrics known for pod
E1102 14:59:55.145440       1 reststorage.go:129] unable to fetch node metrics for node "ip-10-240-42-42.ec2.internal": no metrics known for node
E1102 14:59:55.145462       1 reststorage.go:129] unable to fetch node metrics for node "ip-10-240-62-62.ec2.internal": no metrics known for node
E1102 14:59:55.145467       1 reststorage.go:129] unable to fetch node metrics for node "ip-10-240-52-17.ec2.internal": no metrics known for node
E1102 14:59:58.602483       1 reststorage.go:144] unable to fetch pod metrics for pod default/php-apache-8699449574-4g5sd: no metrics known for pod
E1102 15:00:13.054031       1 reststorage.go:129] unable to fetch node metrics for node "ip-10-240-62-62.ec2.internal": no metrics known for node
E1102 15:00:13.054066       1 reststorage.go:129] unable to fetch node metrics for node "ip-10-240-52-17.ec2.internal": no metrics known for node
E1102 15:00:13.054072       1 reststorage.go:129] unable to fetch node metrics for node "ip-10-240-42-42.ec2.internal": no metrics known for node
E1102 15:00:20.296886       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-240-52-17.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-52-17.ec2.internal (ip-10-240-52-17): Get https://ip-10-240-52-17:10250/stats/summary/: dial tcp: lookup ip-10-240-52-17 on 172.20.0.10:53: server misbehaving, unable to fully scrape metrics from source kubelet_summary:ip-10-240-42-42.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-42-42.ec2.internal (ip-10-240-42-42): Get https://ip-10-240-42-42:10250/stats/summary/: dial tcp: lookup ip-10-240-42-42 on 172.20.0.10:53: server misbehaving, unable to fully scrape metrics from source kubelet_summary:ip-10-240-62-62.ec2.internal: unable to fetch metrics from Kubelet ip-10-240-62-62.ec2.internal (ip-10-240-62-62): Get https://ip-10-240-62-62:10250/stats/summary/: dial tcp: lookup ip-10-240-62-62 on 172.20.0.10:53: server misbehaving]

[cbluth]

your server is trying to dial this hostname ip-10-240-52-17, and it cant find the dns record. this may fix your issue: https://github.com/kubernetes-incubator/metrics-server/issues/165#issuecomment-435272021

[cdenneen]

Thanks. This is the IP and DNS name it appears to be getting from ec2.internal DHCP.
Any reason this doesn’t use inter cluster communication and resolve on cluster.local?

[DirectXMan12]

You can configure the priority of addresses via similar mechanism to the kube apiserver. It prioritizes certain address types by default out of the listed kubelet addresses (in the Node object). Check the flag kubelet-preferred-address-types (which functions identically to the same flag on the API server) for more information.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.