Kube-prometheus: Not able see the data of Pods/Deployment/Statefulset on the pre-configured grafana.

[hamid2013]

What did you do?
I have just installed the kube-prometheus on my cluster.

What did you expect to see?
Pre-Configured Dashboard for the Pods/Deployment/Statefulset/etc

What did you see instead? Under which circumstances?
I can see node related graph, but when it comes to pods related workload not able to see anything. Its just giving NA.

Environment
Pre-Prod

• Prometheus Operator version:

  insert image tag or Git SHA here

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit
:"1dc5c66f5dd61da08412a74221ecc79208c2165b", GitTreeState:"clean", BuildDate:"201
7-07-14T02:00:46Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitComm
it:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2
018-08-20T09:56:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  insert how you created your cluster: acs-engine.

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:

insert Prometheus Operator logs relevant to the issue here

• Node Exported Logs
  time="2018-09-26T11:32:58Z" level=info msg=" - zfs" source="node_exporter.go:97" time="2018-09-26T11:32:58Z" level=info msg="Listening on :9100" source="node_exporter.go:111" time="2018-09-26T11:34:37Z" level=error msg="ERROR: infiniband collector failed after 0.000680s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:34:37Z" level=error msg="ERROR: infiniband collector failed after 0.001399s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:34:43Z" level=error msg="ERROR: infiniband collector failed after 0.000661s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:34:43Z" level=error msg="ERROR: infiniband collector failed after 0.000999s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:35:13Z" level=error msg="ERROR: infiniband collector failed after 0.021550s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:35:13Z" level=error msg="ERROR: infiniband collector failed after 0.000814s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:35:43Z" level=error msg="ERROR: infiniband collector failed after 0.020590s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:35:43Z" level=error msg="ERROR: infiniband collector failed after 0.001209s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:36:13Z" level=error msg="ERROR: infiniband collector failed after 0.022243s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:36:13Z" level=error msg="ERROR: infiniband collector failed after 0.041703s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:36:43Z" level=error msg="ERROR: infiniband collector failed after 0.000838s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:36:43Z" level=error msg="ERROR: infiniband collector failed after 0.001064s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:37:13Z" level=error msg="ERROR: infiniband collector failed after 0.001676s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132" time="2018-09-26T11:37:13Z" level=error msg="ERROR: infiniband collector failed after 0.001240s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:132"

• Kube state Metrics Logs
  I0926 11:33:07.800467 1 main.go:186] Starting metrics server: 0.0.0.0:8080 E0926 11:33:07.902006 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list secrets at the cluster scope E0926 11:33:07.903804 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list namespaces at the cluster scope E0926 11:33:07.999175 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list configmaps at the cluster scope E0926 11:33:07.999687 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v2beta1.HorizontalPodAutoscaler: horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list horizontalpodautoscalers.autoscaling at the cluster scope E0926 11:33:08.000150 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list endpoints at the cluster scope E0926 11:33:08.000194 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list persistentvolumes at the cluster scope E0926 11:33:08.904261 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list secrets at the cluster scope E0926 11:33:08.905088 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list namespaces at the cluster scope E0926 11:33:09.101081 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list configmaps at the cluster scope E0926 11:33:09.101081 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v2beta1.HorizontalPodAutoscaler: horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list horizontalpodautoscalers.autoscaling at the cluster scope E0926 11:33:09.101286 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list endpoints at the cluster scope E0926 11:33:09.203038 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list persistentvolumes at the cluster scope E0926 11:33:09.999401 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list secrets at the cluster scope E0926 11:33:10.000321 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list namespaces at the cluster scope E0926 11:33:10.103061 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list configmaps at the cluster scope E0926 11:33:10.200734 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v2beta1.HorizontalPodAutoscaler: horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list horizontalpodautoscalers.autoscaling at the cluster scope E0926 11:33:10.201435 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list endpoints at the cluster scope E0926 11:33:10.300657 1 reflector.go:205] k8s.io/kube-state-metrics/pkg/collectors/collectors.go:91: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:monitoring:kube-state-metrics" cannot list persistentvolumes at the cluster scope

[brancz]

How did you install the stack?

[hamid2013]

I have used this one year before on k8s version 1.7.2. And cloned that repo that time, so that i can use it later. So the same repo i have used and not cloned the latest one as there are many changes happened.
So on that version we have hack/cluster-monitoring/deploy, so i have used that script to do that.
But in this case, we have the k8s version as 1.10.7

[brancz]

Yes there are lots of changes that are necessary as there were breaking changes in Kubernetes, so you will have to use the latest version of the stack.

[hamid2013]

Even in the new repo, the pod data is not coming through.

Checked the log for the kube-state-metrics, didn't find anything wrong.

[schizo99]

I deployed with the latest images today and I get the same errors as above.
Running kubernetes 1.11.3 in Azure AKS.

time="2018-10-26T07:30:36Z" level=info msg="Listening on 0.0.0.0:9100" source="node_exporter.go:76"
time="2018-10-26T07:30:50Z" level=error msg="ERROR: infiniband collector failed after 0.000536s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:123"
time="2018-10-26T07:31:04Z" level=error msg="ERROR: infiniband collector failed after 0.000686s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:123"

[mendrugory]

I deployed using coreos helm charts in azure aks with kubernetes 1.9.11

time="2018-10-26T12:57:00Z" level=info msg="Listening on 0.0.0.0:9100" source="node_exporter.go:76"
time="2018-10-26T12:57:15Z" level=error msg="ERROR: infiniband collector failed after 0.000851s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:123"
time="2018-10-26T12:57:30Z" level=error msg="ERROR: infiniband collector failed after 0.025460s: strconv.ParseUint: parsing \"N/A (no PMA)\": invalid syntax" source="collector.go:123"

[schizo99]

Seems to be solved in the latest version of node-exporter. Tested with node-exporter v0.17.0-rc.0 and I then don't get the errors. So I guess we'll have to wait for the release and for the coreos helm charts to be updated.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[marcinma]

Happens still when installing through kube-prometheus/manifests both AKS 1.13.10 and 1.14.6

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[brancz]

Closing due to inactivity.