Helm-charts: [prometheus-kube-stack] Prometheus-prometheus and prometheus-alertmanager pods are not getting generated

[Jenishk56]

Describe the bug
Prometheus-prometheus and prometheus-alertmanger pods are not getting generated even using the default values of the chart. However, i can see promtheus-prometheus service and ingress (I choose to have one) been generated and not having any serving pod as there is none and the same for Alertmanager as well.

Version of Helm and Kubernetes:

Helm Version:

$ helm version
v3.2.4

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:56:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-gke.401", GitCommit:"eb94c181eea5290e9da1238db02cfef263542f5f", GitTreeState:"clean", BuildDate:"2020-09-09T00:57:35Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

Which chart: prometheus-kube-stack
Which version of the chart: 10.1.0

What happened:
Only prometheus-operator, prometheus-node-exporter and prometheus-grafana pods are generated.
prometheus-prometheus and prometheus-alertmanager pods are missing.
prometheus-grafana-7669d9b76-4w45f 2/2 Running 0 23h prometheus-kube-state-metrics-85758dcfc9-w6c6b 1/1 Running 0 29h prometheus-operator-66f77f6d85-bg7p6 1/1 Running 0 3h26m prometheus-prometheus-node-exporter-27btc 1/1 Running 0 29h prometheus-prometheus-node-exporter-47gzb 1/1 Running 0 24h prometheus-prometheus-node-exporter-mm9gj 1/1 Running 0 29h

What you expected to happen:
To have prometheus-prometheus and prometheus-alertmanager pods as well to make it working.

How to reproduce it (as minimally and precisely as possible):
Just installed the helm chart with mostly default values in GKE private.
<~--
This could be something like:

Any help would be helpful to resolve this.

[marcus-sa]

Same is happening here

[paranoidsp]

Had this happen to me too, and prometheus-operator did not seem to be logging anything to stdout. Had to restart operator, after which it seemed to do its job fine.

[marcus-sa]

So after some debugging I found out the error causing it was a Stackdriver sidecar which couldn't be started due to the volume mounts couldn't be found.
https://cloud.google.com/stackdriver/docs/solutions/gke/prometheus

Just putting it here in case it might help some of you.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.