Helm-charts: Both Prometheus and AlertManager fail starting up on kube-prometheus-stack-10.3.1

[davidxianggao]

Describe the bug

Helm3 deployed the following chart but keep getting the pod restarting on prometheus-kube-prometheus-stack-prometheus-0

kube-prometheus-stack pang 1 2020-10-28 18:57:18.905088 -0400 EDT deployed kube-prometheus-stack-10.3.1 0.42.1

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled default-scheduler Successfully assigned pang/prometheus-kube-prometheus-stack-prometheus-0 to 111.222.333.444
Normal Pulled 0s kubelet, 111.222.333.444 Container image "quay.io/prometheus/prometheus:v2.21.0" already present on machine
Warning Failed 0s kubelet, 111.222.333.444d Error: cannot find volume "config-out" to mount into container "prometheus"
Normal Pulled 0s kubelet, 111.222.333.444 Container image "rancher/coreos-prometheus-config-reloader:v0.38.1" already present on machine
Warning Failed 0s kubelet, 111.222.333.444 Error: cannot find volume "config" to mount into container "prometheus-config-reloader"
Normal Pulled 0s kubelet, 111.222.333.444 Container image "rancher/coreos-configmap-reload:v0.0.1" already present on machine
Warning Failed 0s kubelet, aaa.bbb.ccc.ddd Error: cannot find volume "prometheus-kube-prometheus-stack-prometheus-rulefiles-0" to mount into container "rules-configmap-reloader"

Also, the similar restart problem happens on the pod of alertmanager-kube-prometheus-stack-alertmanager-0 too

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled default-scheduler Successfully assigned pang/alertmanager-kube-prometheus-stack-alertmanager-0 to 111.222.333.444
Normal Pulled 3s kubelet, 111.222.333.444 Container image "quay.io/prometheus/alertmanager:v0.21.0" already present on machine
Warning Failed 3s kubelet, 111.222.333.444 Error: cannot find volume "config-volume" to mount into container "alertmanager"
Normal Pulled 3s kubelet, 111.222.333.444 Container image "rancher/coreos-configmap-reload:v0.0.1" already present on machine
Warning Failed 3s kubelet, 111.222.333.444 Error: cannot find volume "config-volume" to mount into container "config-reloader"
Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.6", GitCommit:"d32e40e20d167e103faf894261614c5b45c44198", GitTreeState:"clean", BuildDate:"2020-05-20T13:08:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
kube-prometheus-stack

Which version of the chart:
CHART APP VERSION
kube-prometheus-stack-10.3.1 0.42.1

What happened:
I've deployed the Prometheus Kubernets stack as following command, and failed on two pods of prometheus-kube-prometheus-stack-prometheus-0 and alertmanager-kube-prometheus-stack-alertmanager-0. Both keeps restarting.

$ helm install kube-prometheus-stack --namespace pang prometheus-community/kube-prometheus-stack

What you expected to happen:
all pods/workloads should be running as expected.

How to reproduce it (as minimally and precisely as possible):
helm delete the release and helm install again with the following command
$ helm install kube-prometheus-stack --namespace pang prometheus-community/kube-prometheus-stack

There is no any customization like using "--value" or "--set" in Helm command

Anything else we need to know:

[alfsch]

I have the same behavior with kube-prometheus-stack-10.3.4 on k8s 1.7.4

[BlackBsd]

Same problem here. Grafana seems to come online, but the alert manager and prometheus services seem to not come up.

[marcus-sa]

@BlackBsd all services are up and running for me, only the prometheus pod isn't bring created.
Been waiting for this to be fixed for almost a month 🤦‍♂️

[BlackBsd]

@marcus-sa I ended up doing a helm repo update to chart 11.0 and now all things seem to be working.

[alfsch]

I also tried an update to 11.0 but then I ran into problems with argo cd 1.7.7 :-(

[marcus-sa]

@marcus-sa I ended up doing a helm repo update to chart 11.0 and now all things seem to be working.

What version of Kubernetes are you running?
I'm using v1.17.12-gke.1504

[alfsch]

My Problem was Rancher K8s specific. By default they install prometheus-operator watching all namespaces for their monitoring solution. We as project need to install our own prometheus-operator and now we had to opperators battling to start and stop alertmanager and prometheus instances....

Problem solved by configuring each operator only to look into the right namespaces (eg.--prometheus-instance-namespaces=cattle-prometheus --alertmanager-instance-namespaces=cattle-prometheus)

[skolenkin]

I have the same issue:

kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9", GitCommit:"4fb7ed12476d57b8437ada90b4f93b17ffaeed99", GitTreeState:"clean", BuildDate:"2020-07-15T16:18:16Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.13-gke.2001", GitCommit:"00c919adfe4adf308bcd7c02838f2a1b60482f02", GitTreeState:"clean", BuildDate:"2020-11-06T18:24:02Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

helm -n prometheus list
NAME        NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                           APP VERSION
prometheus  prometheus  1           2020-11-27 12:07:28.649482 +0200 EET    deployed    kube-prometheus-stack-12.2.3    0.43.2

kubectl -n prometheus get events
36s         Normal    Scheduled                pod/prometheus-prometheus-kube-prometheus-prometheus-0             Successfully assigned prometheus/prometheus-prometheus-kube-prometheus-prometheus-0 to gke-opa2-default-pool-08c311b5-wjzf
33s         Warning   FailedMount              pod/prometheus-prometheus-kube-prometheus-prometheus-0             Unable to attach or mount volumes: unmounted volumes=[config-out tls-assets prometheus-prometheus-kube-prometheus-prometheus-db prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 prometheus-kube-prometheus-prometheus-token-vv4d7 config], unattached volumes=[config-out tls-assets prometheus-prometheus-kube-prometheus-prometheus-db prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 prometheus-kube-prometheus-prometheus-token-vv4d7 config]: timed out waiting for the condition
26s         Normal    Scheduled                pod/prometheus-prometheus-kube-prometheus-prometheus-0             Successfully assigned prometheus/prometheus-prometheus-kube-prometheus-prometheus-0 to gke-opa2-default-pool-08c311b5-zttk
23s         Warning   FailedMount              pod/prometheus-prometheus-kube-prometheus-prometheus-0             Unable to attach or mount volumes: unmounted volumes=[prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 prometheus-kube-prometheus-prometheus-token-vv4d7 config config-out tls-assets prometheus-prometheus-kube-prometheus-prometheus-db], unattached volumes=[prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 prometheus-kube-prometheus-prometheus-token-vv4d7 config config-out tls-assets prometheus-prometheus-kube-prometheus-prometheus-db]: timed out waiting for the condition

kubectl -n prometheus get secrets |grep prometheus-prometheus-kube-prometheus-prometheus
prometheus-prometheus-kube-prometheus-prometheus                  Opaque                                1      35m
prometheus-prometheus-kube-prometheus-prometheus-tls-assets       Opaque                                1      35m

kubectl -n prometheus get cm |grep prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0   26     1s

I tried to add options --prometheus-instance-namespaces=prometheus and --alertmanager-instance-namespaces=prometheus but It didn't help me.

    spec:
      containers:
      - args:
        - --kubelet-service=kube-system/prometheus-kube-prometheus-kubelet
        - --logtostderr=true
        - --localhost=127.0.0.1
        - --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.43.2
        - --config-reloader-cpu=100m
        - --config-reloader-memory=25Mi
        - --web.enable-tls=true
        - --web.cert-file=cert/cert
        - --web.key-file=cert/key
        - --web.listen-address=:8443
        - --web.tls-min-version=VersionTLS13
        - --prometheus-instance-namespaces=prometheus
        - --alertmanager-instance-namespaces=prometheus
        image: quay.io/prometheus-operator/prometheus-operator:v0.43.2
        imagePullPolicy: IfNotPresent
        name: kube-prometheus-stack

 kubectl -n prometheus get pods
NAME                                                     READY   STATUS        RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   0/2     Terminating   0          6s
prometheus-grafana-76fbdc64df-db9ht                      2/2     Running       0          47m
prometheus-kube-prometheus-operator-9cf7dc8bb-gj72x      1/1     Running       0          92s
prometheus-kube-state-metrics-6df5d44568-f5n9h           1/1     Running       0          47m
prometheus-prometheus-kube-prometheus-prometheus-0       0/3     Terminating   0          6s
prometheus-prometheus-node-exporter-4j9wk                1/1     Running       0          21m
prometheus-prometheus-node-exporter-5q2fh                1/1     Running       0          21m
prometheus-prometheus-node-exporter-bx8ml                1/1     Running       0          21m
prometheus-prometheus-node-exporter-d7zn7                1/1     Running       0          21m
prometheus-prometheus-node-exporter-gg6k9                1/1     Running       0          21m
prometheus-prometheus-node-exporter-hln6c                1/1     Running       0          21m
prometheus-prometheus-node-exporter-nmnn2                1/1     Running       0          21m

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.