Helm-charts: [prometheus-kube-stack] Grafana is not persistent

[ofiryy]

Describe the bug
I installed the prometheus-community/kube-prometheus-stack chart.
and then I defined panels and alerts on grafana.
when I delete the grafana pod - all the data is deleted from grafana - there is no persistency.
I wanted to use this solution: https://github.com/prometheus-operator/prometheus-operator/issues/2558#issuecomment-565119967
but to my surprise - no pv or pvc was created by the prometheus-kube-stack chart.

how can I make my Grafana persistent ?

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.0.3", GitCommit:"ac925eb7279f4a6955df663a0128044a8a6b7593", GitTreeState:"clean", GoVersion:"go1.13.6"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-eks-2ba888", GitCommit:"2ba888155c7f8093a1bc06e3336333fbdb27b3da", GitTreeState:"clean", BuildDate:"2020-07-17T18:48:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Which chart: prometheus-kube-stack

Which version of the chart: 12.3.0

How to reproduce it (as minimally and precisely as possible): Install prometheus-kube-stack and define a panel in grafana, then delete the grafana pod

[survivant]

There is a setting for prometheus and alertmanager

storage: volumeClaimTemplate: spec: accessModes: [ "ReadWriteOnce" ] storageClassName: sc-mirror resources: requests: storage: 300Mi

I think we should have the same for grafana ?

[totoroot]

This is not a bug as data persistence is not enabled by default. You can either claim a PersistentVolume in your custom values.yaml file like @survivant suggested or export your dashboards as JSON definition files and create a ConfigMap with the JSON-formatted data for each custom dashboard. This way with each new release of the stack via helm, the modifications within Grafana do not persist but your exported dashboards get redeployed with everything else.

[blademainer]

@ofiryy I update the yaml: values.yaml add:

grafana:
  persistence:
    enabled: true

to fix the grafana persistent problem.

[survivant]

@blademainer but we still can't choose our storage class

[BertelBB]

@survivant prometheus-community/kube-prometheus-stack chart uses the grafana/grafana chart as a dependency. So any values you can pass to grafana/grafana you can pass to the grafana object in this chart. Or am I misunderstanding the issue being raised?

This works for me

grafana:
  enabled: true

  persistence:
    enabled: true
    type: pvc
    storageClassName: default
    accessModes:
    - ReadWriteOnce
    size: 4Gi
    finalizers:
    - kubernetes.io/pvc-protection

[survivant]

@BertelBB thank you. I don't know what I did wrong at that time.. but it works fine. Now, I need to find a workaround for https://github.com/prometheus-community/helm-charts/issues/437

[mkoziel2000]

I'm not sure the workflow of expecting all the grafana settings to get zapped on the next stopping of a pod has got the best interests of the enterprise in mind. I get the argument of exporting the charts as JSON and storing into configMaps to make them deployment agnostic, but there are other settings not related to charts that we don't want to have disappear when a pod crashes either (such as user login information, settings around alerting, and so forth). So, unless there is a best practice for storing all of that into configmaps as well (and a good user UI for how to do that, which doesn't require kubectl and a Kubernetes admin), it seems shortsighted to think that Grafana can live in an enterprise environment as an application that doesn't require persistence. It seems the opposite would be true.

I too am wringing out the kinks of my Prometheus install and ran into this exact same problem of grafana not supporting persistence out of the box. It was rather alarming to learn that after I began building out dashboards, I lost that work when I tested out the failover scenarios of the pod going down. I did not see a persistence piece in the grafana part of the values.yaml and didn't know that this would turn grafana into an app with a temporary persistence layer.

In hind sight, I should have done my pod failover test first before beginning to "persist" data in grafana to learn about this annoying default. I do wish that the helm chart can be upgraded to have a section under grafana that allows the ability to define the persistence layer...even if its commented out.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.