Metrics-server: I got a message "no metrics known for pod"

[Strangedeveloper]

I deployed metric-server.
but my "kubectl logs metric-server-pod" gave me this message.

I0813 02:06:46.985836 1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key) I0813 02:06:47.348587 1 secure_serving.go:116] Serving securely on [::]:443 E0813 02:07:00.339450 1 reststorage.go:128] unable to fetch node metrics for node "client-master": no metrics known for node

my kubernetes version :
kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:37:41Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

sure, I added below options on my metric-server-deployment.yaml
command: - /metrics-server - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP

I use internal network and I installed kubernetes offline.
how can I fix this problem?
please, help me..

[eamrnas]

I'm having the same issue on the same K8s version 1.15.0. I used /deploy/1.8+ .
is there any solution for this problem?

[xilu0]

me too.

[OsandaDeemantha]

kubectl top pods
Result:

W0827 14:26:49.076592   81772 top_pod.go:266] Metrics not available for pod default/busybox, age: 332h53m37.076586606s
error: Metrics not available for pod default/busybox, age: 332h53m37.076586606s

kubectl logs <metrics-server-pod-name>
Result:
E0827 13:37:10.001523 1 reststorage.go:147] unable to fetch pod metrics for pod default/busybox: no metrics known for pod

I face the same issue when I run kubectl top pods command which worked earlier without an issue. Initially, I had about 2-5 pods and now it has around 15-20 pods. top pods command doesnt work now but top nodes command works correctly.
I tried with increasing replicas and editing the metrics-server-deployment.yaml file and adding parameters below to fix the error. but it also doesn't work for me.
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP

[pierluigilenoci]

Have you tried to use this: https://github.com/kubernetes-incubator/metrics-server/issues/278#issuecomment-515370587

[serathius]

@Strangedeveloper Metrics Server is expected to report "no metrics known for pod" until cache will be populated. Cache can be empty on freshly deployed metrics-server or can miss values for newly deployed pods.

[allenhaozi]

got same error on k8s 1.8
metrics-server log nothing unnatural
but
kubectl top pod

show the following error
W0122 22:33:20.491035 25678 top_pod.go:192] Metrics not available for pod xxx/apigateway-apigateway-kbhwf, age: 79h26m43.491020032s
error: Metrics not available for pod xxx/apigateway-apigateway-kbhwf, age: 79h26m43.491020032s

[onedr0p]

Any reason all these issues were closed? we're still seeing issues...

[serathius]

Original author was not providing any updates and "no metrics known for node" log was explained. If you have a similar problem feel free to open new Issue.

[onedr0p]

That's hardly a reason to close all these issues, yes, they are duplicates and they should be closed. But the amount of people having these issues in each one doesn't justify closing all of them.

By your logic, me opening a duplicate issue with the same information, you would close it in a few months with the same reason.

[nerusnayleinad]

For those, who have tried everything, and it is not working yet, check the apiVersion of the object the HPA is hooked to. This is what I mean.

...
  scaleTargetRef:
    apiVersion: apps/v1           <-be sure deployment (in this case) `apiVersion` and hpa `spec.scaletargetref.apiversion` match
    kind: Deployment
    name: nginx
...

If not, it is going to return no metrics known for pod.

And if you want to also solve --kubelet-insecure-tls flag issue, you can get the CA cert of the cluster, create a secret from it, and add it to the metrics server pod. Then to refer to it when starting the server. This is what I've got:

$ kubectl get deploy metrics-server -n kube-system -oyaml | grep -A 7 args
      - args:
        - /metrics-server
        - --v=4
        - --kubelet-preferred-address-types=InternalIP
        - --cert-dir=/tmp
        - --secure-port=4443
        - --client-ca-file=/usr/local/secret/ca.crt  
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6

and it works:

$ kubectl top no
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
v1-17-master   224m         11%    1400Mi          36%       
v1-17-worker   279m         13%    1787Mi          46%       
$ kubectl get hpa -n flux-demo
NAME        REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpa-nginx   Deployment/nginx   0%/40%    1         1         1          3h34m
$ kubectl top po -n flux
NAME                         CPU(cores)   MEMORY(bytes)   
flux-85f4cd5d56-lcsgz        202m         21Mi            
memcached-86bdf9f56b-nwzwb   3m           27Mi  

[mhewedy]

client-ca-file=/usr/local/secret/ca.crt

I did edited the deployment and followed your instructions but not worked for me:

    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --client-ca-file=/usr/local/secret/ca.crt
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        name: metrics-server
        ports:
        - containerPort: 4443
          name: main-port
          protocol: TCP
        resources: {}
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
        - mountPath: /usr/local/secret/
          name: ca
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/arch: amd64
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: metrics-server
      serviceAccountName: metrics-server
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: tmp-dir
      - name: ca
        secret:
          defaultMode: 420
          items:
          - key: ca.crt
            path: ca.crt
          secretName: default-token-t79hl

and secret:

k -n kube-system get secrets --no-headers | grep default
default-token-t79hl                              kubernetes.io/service-account-token   3     24h

Then to verify:

k -n kube-system get pods --show-labels -l k8s-app=metrics-server
NAME                              READY   STATUS    RESTARTS   AGE   LABELS
metrics-server-748dfdc556-v4dwl   1/1     Running   0          17s   k8s-app=metrics-server,pod-template-hash=748dfdc556

But still k top pods and k top nodes not working

[nerusnayleinad]

But you gotta start the metrics server. You are missing /metrics-server in the beginning. Otherwise, you are just passing random arguments to the principal endpoint of your container.

[bu3ny]

Following settings worked for me:

  containers:
  - name: metrics-server
    image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
    imagePullPolicy: IfNotPresent
    args:
      - --cert-dir=/tmp
      - --secure-port=4443
      - --kubelet-insecure-tls
      - --kubelet-preferred-address-types=InternalIP
      - /metrics-server
      - --v=2

K8 version: 1.19.3
Metrics Server: 0.3.7

Just download https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

edit the file as per above directives and it should work.

[osaffer]

I downloaded 0.4.0. I modified components yaml file like you, I cannot fetch metrics from pods, from nodes it works.