Metrics-server: no metrics known for pod. kubectl top works

[bianchi2]

Is it notmal that metrics server logs no metrics known for pod for ALL pods in the cluster on start, though it seems to be working normally since kubectl top returns results for pods and nodes just fine.

Using k8s.gcr.io/metrics-server-amd64:v0.3.3 with --kubelet-insecure-tls --kubelet-preferred-address-types=InternalIP args

[jihed]

I have the same issue on k8s 1.15.2.

[Woap]

Same problem with metrics-server-3.2.0 deployed with Helm
K8S 1.15.0

And with those arguments

/metrics-server
--kubelet-insecure-tls
--kubelet-preferred-address-types=InternalIP

Metrics-servers logs :

I0911 14:01:46.782718       1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0911 14:01:47.315002       1 secure_serving.go:116] Serving securely on [::]:8443
E0911 14:01:47.937715       1 reststorage.go:135] unable to fetch node metrics for node "***": no metrics known for node
E0911 14:01:47.937740       1 reststorage.go:135] unable to fetch node metrics for node "***": no metrics known for node
E0911 14:01:47.937747       1 reststorage.go:135] unable to fetch node metrics for node "***": no metrics known for node
E0911 14:01:47.941365       1 reststorage.go:160] unable to fetch pod metrics for pod dev/***: no metrics known for pod

[KieranP]

Same issue.

Versions:

% kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

% minikube version
minikube version: v1.3.1
commit: ca60a424ce69a4d79f502650199ca2b52f29e631

% helm version
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

Install:

% helm upgrade -i metrics-server stable/metrics-server --namespace kube-system --set args="{--logtostderr,--kubelet-insecure-tls,--kubelet-preferred-address-types=InternalIP\,ExternalIP\,Hostname}"

% helm ls
NAME                REVISION    UPDATED                     STATUS      CHART                   APP VERSION NAMESPACE
metrics-server      1           Sun Sep 15 21:19:08 2019    DEPLOYED    metrics-server-2.8.5    0.3.4       kube-system

Deployment:

% kubectl get deploy
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
users-service-deployment      1/1     1            1           4h40m

% kubectl get pods
NAME                                          READY   STATUS    RESTARTS   AGE
users-service-deployment-796874f86f-2t845     2/2     Running   0          9m15s

Autoscaler:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: users-service-autoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: users-service-deployment
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

Result:

% kubectl top pod
NAME                                          CPU(cores)   MEMORY(bytes)
users-service-deployment-796874f86f-2t845     6m           50Mi

% kubectl get hpa
NAME                          REFERENCE                                TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
users-service-autoscaler      Deployment/users-service-deployment      <unknown>/50%   1         10        1          5m1s

% kubectl log metrics-server-7564b87f5d-57knn -n kube-system
I0915 09:19:10.278955       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0915 09:19:10.716436       1 secure_serving.go:116] Serving securely on [::]:8443
E0915 09:20:57.608741       1 reststorage.go:160] unable to fetch pod metrics for pod default/users-service-deployment-796874f86f-2t845: no metrics known for pod

kube top works fine, hpa can't get the current CPU usage, and thus never scales.

[Woap]

Same problem here #300 #299

[morenice]

Check resources(requests-cpu) configuration in Deployment file.

https://aws.amazon.com/premiumsupport/knowledge-center/eks-metrics-server-pod-autoscaler/?nc1=h_ls

[serathius]

Metrics server logging no metrics known for pod is expected behavior (maybe changing error to info or log name).
This is due metrics server scraping metrics periodically (default 1 minute), which can lead to situations where pod is running for 30s but metrics-server didn't had chance to scrape it.

@KieranP Your describing different issue where HPA cannot reach metrics-server, please look into https://github.com/kubernetes-sigs/metrics-server/issues/317