Metrics-server: no metrics known for pod

Created on 3 Apr 2019  路  22Comments  路  Source: kubernetes-sigs/metrics-server

I have deployed NGINX (v1.15.9) PODs, but metrics-server cannot retrieve CPU statistics for these. I get the following error:

reststorage.go:144] unable to fetch pod metrics for pod nginx/nginx-1-75cdbcf4d8-6vzms: no metrics known for pod

even though I get statistics from other PODs on the same node. What is missing?

picture liakouras
👍16

Most helpful comment

Just to add another data point for anyone struggling with this.

With 1.12, I finally got around the no metrics known for pod with the following combo of changes (started with the stable helm charts and modified from there):

  1. Add the following to metrics-server-deployment.yaml:

    command:
    - /metrics-server
    - --v=2
    - --kubelet-preferred-address-types=InternalIP
    - --kubelet-insecure-tls

  2. Set hostNetwork.enabled=true in values.yaml

  3. Ensuring the pods that need HPA have both CPU requests and limits defined:

    resources:
    limits:
    cpu: "1000m"
    requests:
    cpu: "1000m"

picture michael-pratt on 14 Oct 2019
👍16 🚀2 🎉1 😄1

All 22 comments

Same here

philipakash picture philipakash on 3 Apr 2019

same here,

I am using EKS, kubernetes version 1.11.

kubectl top nodes and kubectl top pod works perfectly fine. But kubectl get hpa shows <unknown> cpu usage and I am seeing the no metrics known for pod in metrics-servers log.

sunao-uehara picture sunao-uehara on 9 Apr 2019
👍8

@sunao-uehara I fixed my problem by adding 

        command:
        - /metrics-server
        - --v=2
        - --kubelet-preferred-address-types=InternalIP

This is required to make the metrics server work in AWS EKS .

philipakash picture philipakash on 9 Apr 2019
👍19 👎4

I have the same problem -(weird behavior)

AWS  EKS 1.12.6 
Metrics server v0.3.2

```bash
kubectl top nodes --> works
kubectl top po ---> works
kctl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq . --> works
kctl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq . ---> works

But metrics server keeps complaining. Any ideas what is going on?


```bash
bash-4.4$ kubectl logs -f metrics-server-656d4cd9ff-5q254 -n kube-system
I0602 09:24:56.909292       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W0602 09:24:57.320632       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0602 09:24:57.330976       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
I0602 09:24:57.346827       1 manager.go:95] Scraping metrics from 0 sources
I0602 09:24:57.346849       1 manager.go:150] ScrapeMetrics: time: 1.834碌s, nodes: 0, pods: 0
[restful] 2019/06/02 09:24:57 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2019/06/02 09:24:57 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0602 09:24:57.375777       1 serve.go:96] Serving securely on [::]:443
E0602 09:25:03.556935       1 reststorage.go:148] unable to fetch pod metrics for pod tm-uk/xxxxx: no metrics known for pod
I0602 09:25:57.347019       1 manager.go:95] Scraping metrics from 8 sources
I0602 09:25:57.348270       1 manager.go:120] Querying source: kubelet_summary:ip-10-247-236-144.eu-west-1.compute.internal
I0602 09:25:57.353254       1 manager.go:120] Querying source: kubelet_summary:ip-10-247-222-203.eu-west-1.compute.internal
I0602 09:25:57.354192       1 manager.go:120] Querying source: kubelet_summary:ip-10-247-233-2.eu-west-1.compute.internal
javapapo picture javapapo on 2 Jun 2019

I did increase the log level (v=4 ) and I see something interesting

I0602 09:33:03.937019       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1/namespaces/tm-uk/pods?labelSelector=app%3Diccp-backend%2Cenv%3Ddev: (608.776碌s) 200 [[kube-controller-manager/v1.12.6 (linux/amd64) kubernetes/d69f1bf/horizontal-pod-autoscaler] 10.xxxx.xxxx.xxxx:39224]
I0602 09:33:06.007139       1 authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched"
I0602 09:33:06.007246       1 wrap.go:42] GET /: (2.154984ms) 403 [[Go-http-client/2.0] 10.247.253.176:34250]
I0602 09:33:06.070646       1 authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched"
I0602 09:33:06.070745       1 wrap.go:42] GET /: (190.466碌s) 403 [[Go-http-client/2.0] 10.247.253.176:34250]
I0602 09:33:07.521568       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (330.622碌s) 200 [[kube-apiserver/v1.12.6 (linux/amd64) kubernetes/d69f1bf] 10.247.253.176:34280]
I0602 09:33:09.059437       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (292.236碌s) 200 [[kube-controller-manager/v1.12.6 (linux/amd64) kubernetes/d69f1bf/resourcequota-controller] 10.xxx.xxx.xxx:39224]
I0602 09:33:18.039808       1 authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched"
I0602 09:33:18.039897       1 wrap.go:42] GET /: (10.106366ms) 403 [[Go-http-client/2.0] 10.247.253.176:34250]
I0602 09:33:18.944882       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1/namespaces/tm-uk/pods?labelSelector=app%3Diccp-xxxx%2Cenv%3Ddev: (654.207碌s) 200 [[kube-controller-manager/v1.12.6 (linux/amd64) kubernetes/d69f1bf/horizontal-pod-autoscaler
javapapo picture javapapo on 2 Jun 2019
👍2

@sunao-uehara I fixed my problem by adding

        command:
        - /metrics-server
        - --v=2
        - --kubelet-preferred-address-types=InternalIP

This is required to make the metrics server work in AWS EKS .

This did the trick for me too, thanks!

Depends on the setup, you may also need --kubelet-insecure-tls=true for metrics-server to properly scrape the kubelet. Just in case it helps somebody else :)

rata picture rata on 21 Jun 2019
👍4 👎1

I did increase the log level (v=4 ) and I see something interesting

I0602 09:33:03.937019       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1/namespaces/tm-uk/pods?labelSelector=app%3Diccp-backend%2Cenv%3Ddev: (608.776碌s) 200 [[kube-controller-manager/v1.12.6 (linux/amd64) kubernetes/d69f1bf/horizontal-pod-autoscaler] 10.xxxx.xxxx.xxxx:39224]
I0602 09:33:06.007139       1 authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched"
I0602 09:33:06.007246       1 wrap.go:42] GET /: (2.154984ms) 403 [[Go-http-client/2.0] 10.247.253.176:34250]
I0602 09:33:06.070646       1 authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched"
I0602 09:33:06.070745       1 wrap.go:42] GET /: (190.466碌s) 403 [[Go-http-client/2.0] 10.247.253.176:34250]
I0602 09:33:07.521568       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (330.622碌s) 200 [[kube-apiserver/v1.12.6 (linux/amd64) kubernetes/d69f1bf] 10.247.253.176:34280]
I0602 09:33:09.059437       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (292.236碌s) 200 [[kube-controller-manager/v1.12.6 (linux/amd64) kubernetes/d69f1bf/resourcequota-controller] 10.xxx.xxx.xxx:39224]
I0602 09:33:18.039808       1 authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched"
I0602 09:33:18.039897       1 wrap.go:42] GET /: (10.106366ms) 403 [[Go-http-client/2.0] 10.247.253.176:34250]
I0602 09:33:18.944882       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1/namespaces/tm-uk/pods?labelSelector=app%3Diccp-xxxx%2Cenv%3Ddev: (654.207碌s) 200 [[kube-controller-manager/v1.12.6 (linux/amd64) kubernetes/d69f1bf/horizontal-pod-autoscaler

@javapapo Even I'm facing with same issue after updating the EKS cluster version. Did u solved this ? if yes how did you do that ?

VinayVanama picture VinayVanama on 26 Jun 2019

Same here, - --kubelet-preferred-address-types=InternalIP
and - --kubelet-preferred-address-types=InternalDNS do not help.

GrigorievNick picture GrigorievNick on 15 Aug 2019
👍5

+1

jaydipdave picture jaydipdave on 15 Aug 2019

Same issue here.
--kubelet-preferred-address-types=InternalIP and --kubelet-insecure-tls=true are set and still not working.
Sometimes metrics-server throws this error: unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary

Running EKS 1.13
metrics-server 0.3.3

mmingorance-dh picture mmingorance-dh on 15 Aug 2019

kubectl top pods
Result:

W0827 14:26:49.076592   81772 top_pod.go:266] Metrics not available for pod default/busybox, age: 332h53m37.076586606s
error: Metrics not available for pod default/busybox, age: 332h53m37.076586606s

kubectl logs <metrics-server-pod-name>
Result:
E0827 13:37:10.001523 1 reststorage.go:147] unable to fetch pod metrics for pod default/busybox: no metrics known for pod

I face the same issue when I run kubectl top pods command which worked earlier without an issue. Initially, I had about 2-5 pods and now it has around 15-20 pods. top pods command doesnt work now but top nodes command works correctly.
I tried with increasing replicas and editing the metrics-server-deployment.yaml file and adding parameters below to fix the error. but it also doesn't work for me.
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP

OsandaDeemantha picture OsandaDeemantha on 27 Aug 2019

Have you tried to use this: https://github.com/kubernetes-incubator/metrics-server/issues/278#issuecomment-515370587

pierluigilenoci picture pierluigilenoci on 29 Aug 2019
👎7

I've tried https://github.com/kubernetes-incubator/metrics-server/issues/157#issuecomment-529867498 and it works for me now. But I have no idea why, and it is driving me crazy

AlexRRR picture AlexRRR on 10 Sep 2019

Just to add another data point for anyone struggling with this.

With 1.12, I finally got around the no metrics known for pod with the following combo of changes (started with the stable helm charts and modified from there):

  1. Add the following to metrics-server-deployment.yaml:

    command:
    - /metrics-server
    - --v=2
    - --kubelet-preferred-address-types=InternalIP
    - --kubelet-insecure-tls

  2. Set hostNetwork.enabled=true in values.yaml

  3. Ensuring the pods that need HPA have both CPU requests and limits defined:

    resources:
    limits:
    cpu: "1000m"
    requests:
    cpu: "1000m"

michael-pratt picture michael-pratt on 14 Oct 2019
👍16 🚀2 🎉1 😄1

Log no metrics known for pod doesn't indicate the problem. Details: https://github.com/kubernetes-sigs/metrics-server/issues/349

serathius picture serathius on 19 Nov 2019

Just to add another data point for anyone struggling with this.

With 1.12, I finally got around the no metrics known for pod with the following combo of changes (started with the stable helm charts and modified from there):

  1. Add the following to metrics-server-deployment.yaml:
    command: <ul> <li>/metrics-server</li> <li>--v=2</li> <li>--kubelet-preferred-address-types=InternalIP</li> <li>--kubelet-insecure-tls<br />

  2. Set hostNetwork.enabled=true in values.yaml
  3. Ensuring the pods that need HPA have both CPU requests and limits defined:
    resources: limits: cpu: "1000m" requests: cpu: "1000m"

@michael-pratt

What definition are you setting hostNetwork.enabled=true in? In the metrics-server Deployment or Service?

Nevermind, I see that it's used in the Deployment yaml for metrics-server.

kyle-ax picture kyle-ax on 20 Nov 2019

@sunao-uehara I fixed my problem by adding

        command:
        - /metrics-server
        - --v=2
        - --kubelet-preferred-address-types=InternalIP

This is required to make the metrics server work in AWS EKS .

hey, where i should put this command? inside the metric-server deployment yaml?

yishaihl picture yishaihl on 29 Jun 2020

Yes

rata picture rata on 29 Jun 2020

Just to add another data point for anyone struggling with this.

With 1.12, I finally got around the no metrics known for pod with the following combo of changes (started with the stable helm charts and modified from there):

  1. Add the following to metrics-server-deployment.yaml:
    command: <ul> <li>/metrics-server</li> <li>--v=2</li> <li>--kubelet-preferred-address-types=InternalIP</li> <li>--kubelet-insecure-tls<br />

  2. Set hostNetwork.enabled=true in values.yaml
  3. Ensuring the pods that need HPA have both CPU requests and limits defined:
    resources: limits: cpu: "1000m" requests: cpu: "1000m"
    Edit deployment yaml file
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: metrics-server
    namespace: kube-system
    labels:
    k8s-app: metrics-server
    spec:
    selector:
    matchLabels:
    k8s-app: metrics-server
    template:
    metadata:
    name: metrics-server
    labels:
    k8s-app: metrics-server
    spec:
    serviceAccountName: metrics-server
    volumes:
    # mount in tmp so we can safely use from-scratch images and/or read-only containers

    • name: tmp-dir

      emptyDir: {}

      containers:

    • name: metrics-server

      image: registry.cn-shanghai.aliyuncs.com/k8sgcrio_containers/metrics-server:v0.3.7

      imagePullPolicy: IfNotPresent

      command:



      • /metrics-server


      • --v=2


      • --kubelet-preferred-address-types=InternalIP


      • --kubelet-insecure-tls


        args:


      • --cert-dir=/tmp


      • --secure-port=4443


        ports:





        • name: main-port



          containerPort: 4443



          protocol: TCP



          securityContext:



          readOnlyRootFilesystem: true



          runAsNonRoot: true



          runAsUser: 1000



          volumeMounts:



        • name: tmp-dir



          mountPath: /tmp



          nodeSelector:



          kubernetes.io/os: linux






[root@client00 hpa-test]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 10 1 13m

zouchengli picture zouchengli on 2 Sep 2020

I have this error every time I deploy the metrics-server on DigitalOcean. Adding the following in the Deployment spec fix the thing.

````
command:

  • /metrics-server
  • --logtostderr
  • --kubelet-insecure-tls=true
  • --kubelet-preferred-address-types=InternalIP
  • --v=2
    ````
lucj picture lucj on 15 Sep 2020

Hi,

For me - --v=2 did the trick

Isakgicu picture Isakgicu on 18 Oct 2020
👍2

Using EKS version 1.18 I was able to fix this by just adding - --v=2 into the metrics-server container's argument, nothing else. 

      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --v=2
azepedaas picture azepedaas on 1 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

serathius picture serathius  路  9Comments
joshshn picture joshshn  路  9Comments
Sh4d1 picture Sh4d1  路  6Comments
godomainz picture godomainz  路  6Comments
Constantin07 picture Constantin07  路  3Comments