Charts: Helm on Kubernetes install stable/prometheus prometheus-server can start just 1 pods

[ZhangBohan1987]

Is this a request for help?:

Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
This is BUG REPORT

Version of Helm and Kubernetes:
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Which chart:
stable/prometheus 6.8.0

What happened:
I set server.replicaCount to 3, but the deployment of "prometheus-server" just start 1 pod
other 2 pod to be fail, I check the log, it show information as bellow:
level=error ts=2018-09-28T09:58:30.26410111Z caller=main.go:597 err="Opening storage failed lock DB directory: resource temporarily unavailable"

What you expected to happen:
The deployment of "prometheus-server" shoud be start 3 pods

How to reproduce it (as minimally and precisely as possible):
Execute this commond install chart of "stable/prometheus"
helm install stable/prometheus --name prometheus --namespace kube-system \ --set alertmanager.image.tag=v0.15.0,\ alertmanager.baseURL=/,\ alertmanager.persistentVolume.storageClass=prometheus-managed-nfs-storage,\ configmapReload.image.tag=v0.2.2,\ kubeStateMetrics.image.tag=v1.3.1,\ nodeExporter.image.tag=v0.16.0,\ pushgateway.image.tag=v0.5.2,\ server.image.tag=v2.3.1,\ server.terminationGracePeriodSeconds=360,\ server.replicaCount=3,\ server.persistentVolume.storageClass=prometheus-managed-nfs-storage

Anything else we need to know:

[lcavajani]

It is not possible to deploy Prometheus in "HA" by changing the number of Replicas.

If you try to do that with the current helm chart, Kubernetes will create a deployment for 3 pods which share the same volume. The result of this is that the first pod locks de Prometheus and hence the other one can't use it (so they are useless).

There is currently a PR to replace the deployment with a statefulset which solve this situation.

[giacomoguiulfo]

Just to add to what @lcavajani said, this is the open PR: #7116

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[giacomoguiulfo]

Not stale