Dashboard: Kubernetes Dashboard does not show CPU Usage and Memory Usage graph if namespace has Cron Jobs

[tchellomello]

Environment

I've deployed the kubernetes dashboard as documented at https://github.com/kubernetes/dashboard/releases/tag/v2.0.0-beta2 and got it working as expected.

The metrics-scaper is working as expected and I can see the graph displayed below:

However, if a namespace has a Cron Job defined, the top 2 graphs (CPU and Memory Usage) are not displayed due to the error below:

[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:54 Getting list of all cron jobs in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:54 [2019-07-30T23:10:54Z] Incoming HTTP/2.0 GET /api/v1/daemonset/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:54 [2019-07-30T23:10:54Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:54 [2019-07-30T23:10:54Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/deployment/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all deployments in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/job/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all jobs in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/pod/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all pods in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/replicaset/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all replica sets in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/replicationcontroller/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all replication controllers in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/statefulset/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all pet sets in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/ingress/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/service/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of all services in the cluster 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/configmap/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/persistentvolumeclaim/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list config maps in the namespace kubeapps 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list persistent volumes claims 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Incoming HTTP/2.0 GET /api/v1/secret/kubeapps?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 10.42.0.4:35784:  
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting list of secrets in &{[kubeapps]} namespace 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Getting pod metrics 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 Skipping metrics because of error: Received invalid number of resources from sidecar. Expected 10 received 0 
[kubernetes-dashboard-76b969b44b-8bvv8] 2019/07/30 23:10:55 [2019-07-30T23:10:55Z] Outcoming response to 10.42.0.4:35784 with 200 status code 

Then the dashboard is displayed like this:

Installation method: kubectl apply
Kubernetes version: v1.14.4
Dashboard version: v2.0.0-beta2+0.ge9d82f7a
Operating system: CentOS 7
Node.js version ('node --version' output):
Go version ('go version' output):

Steps to reproduce

1. Deploy the dashboard

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta2/aio/deploy/recommended.yaml

1. Deploy a test pod and the graph will be displayed
2. Add a cronjob to the namespace and the graph will not be displayed.

Observed result

CPU and Memory Usage graphs are not displayed when a cronjob is listed

Expected result

CPU and Memory Usage graphs should be displayed when a cronjob is listed

Comments

[tchellomello]

Just for the documentation purposes, it is still happening on v2.0.0-beta3+0.g677aff93

[maciaszczykm]

@tchellomello We are aware, we have a couple of issues with metrics that are not fixed yet. We will work on them.

[davidtseng]

I also see the CPU/memory graphs not show up if there is a crashlooping pod in the namespace.

[maciaszczykm]

The issue is not directly caused by Cron Jobs. [kubernetes-dashboard-76b969b44b-8bvv8] [restful] 2019/07/30 23:10:54 log.go:33: There was an error during transformation to sidecar selector: Resource "cronjob" is not a native sidecar resource type or is not supported does not appear anymore.

There is an issue with pods created by cron jobs and with crashlooping pods.

[maciaszczykm]

@jeefy I think we have a problem with metrics scraper here. When the pod is completed (i.e. job's child) or in a crashed state, we are dealing with 2019/09/05 15:22:04 Skipping metrics because of error: Received invalid number of resources from the sidecar. Expected 3 received 2. Could you take a look on this as you are more familiar with the code both on the dashboard and scraper side?

[jeefy]

I'll take a look at this soon. Thanks!

[clhlc]

+1

[lefranco6910]

I just got the same issue with 2.0.0beta4 today. I found this thread by googling the issue. I thought it might be good to share. Thanks for looking at this issue.

[maciaszczykm]

Any update, @jeefy?

[SMelnykov]

Dashboard v2.0.0-beta4+0.gf232c589 - Not see any graphs on Cluster -> Nodes page.
See graphs on the name space (All namespaces) -> Overview page.
kubectl top node --> See performance information per nodes
Will be good see memory in percentage also not in Mi

[strainovic]

+1 same issue with beta5

[floreks]

We are aware of that. That's why it is not closed.

[teeram]

Thanks for working on this guys. I'm having this problem too.

[sompholyai]

Do you have solution ?

[kaikuchn]

The bug has been fixed with https://github.com/kubernetes/dashboard/pull/4506
Using 2.0.0-rc2 works. If you still don't get metrics maybe your issue is something else.