Kube-prometheus: Persist Grafana Storage

[saraAlizadeh]

What is missing?
I need to add a PV to grafana deployment.
I found no sample in .jsonnet format. like the thing you've provided for prometheus here

Why do we need it?
every time my grafana pod gets recreated, everything is missing. by everything, I mean custom dashboards, users, data sources, folders, permissions and so on .
I need them to be persistent.

Environment

• kube-prometheus version:

0.4

Anything else we need to know?:
my kubernetes version: 1.16

[brancz]

We intentionally setup Grafana in a stateless way, where all of Grafana is provisioned via configuration files managed via version control. That's the opinionated way how we choose to deploy Grafana and we don't intend to change that :)

[JoseArgueta]

Hi @brancz we have similar situation, is there any documentation to read about it?

[songrijie]

@saraAlizadeh We've implemented persist storage in our project in order to save accounts and privileges. Here is the way...

grafana-storage.libsonnet:

// In original volumes definition,  "grafana-storage" goes to emptydir.
// storageWithPVClaim replace "grafana-storage" in grafana deployment. 
local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet'; 

local pvc = k.core.v1.persistentVolumeClaim; 
local grafanaPvClaimName = 'grafana-storage';
local grafanaVolName = 'grafana-storage';
local vol = k.apps.v1beta1.deployment.mixin.spec.template.spec.volumesType;

// Convert to map for easier overloading, assumes all array elements are maps having "name" field
local toNamedMap(array) = { [x.name]: x for x in array };

// Convert back to array
local toNamedArray(map) = [{ name: x } + map[x] for x in std.objectFields(map)];

local grafanaStorageWithPVClaim(storageClassName) = {
    grafana+:: {
        deployment+: {
            spec+: {  
                template+: {
                    spec+: {
                        volumes: 
                        toNamedArray(toNamedMap(super.volumes) + toNamedMap([ 
                            vol.fromPersistentVolumeClaim(grafanaVolName, grafanaPvClaimName),
                        ]))
                    },
                },
            },
        },
        pvc: 
            pvc.new() +  
            pvc.mixin.spec.withAccessModes('ReadWriteOnce') +
            pvc.mixin.spec.resources.withRequests({ storage: '5Gi' }) +
            pvc.mixin.spec.withStorageClassName(storageClassName) +
            pvc.mixin.metadata.withNamespace($._config.namespace) + 
            pvc.mixin.metadata.withName(grafanaPvClaimName)

    },
};


{
  grafanaStorageWithPVClaim:: grafanaStorageWithPVClaim,
}

in entry jsonnet:

...
local grafanaPv = import 'wk-grafana-storage.libsonnet';

// default storageClass for monitoring.
local defaultPvcStorageName = 'wbox-log';

...
local kp = 
...
// grafana persistent storage
grafanaPv.grafanaStorageWithPVClaim(defaultPvcStorageName);
...

The solution introduces grafana-pvc.yaml, and it's up to you to keep it or not.

Not sure if there's better solution... hope it helps.

BTW, I agreed with @brancz that Grafana can go to stateless way, and we are moving to it soon as Grafana has connected to gitlab ce with oauth.

[TamasNeumer]

@brancz

Hi and thanks for the collaboration in the prometheus-operator!

I understand that the stateless approach would be the "best practice", however how should I proceed with alerts and notification channels then?

• Dashboards: The way I see I can export dashboards to configmaps, and these will be reloaded on pod restart.
• OAuth / OIDC config: The way I see it's also configured in a static way - pod restart won't effect it, users will be able to log in just as before.
• What happens if users create notification channels and create alerts for their dashboards? I guess these are gone with a pod restart, and won't be recreated once the new pod spins up. Is it possible to export these to ConfigMaps as well? Or would you rather suggest using the HA Prometheus-Alert for alerting and Grafana is merely a "visualization tool"?

Thanks!

[paulfantom]

Or would you rather suggest using the HA Prometheus-Alert for alerting and Grafana is merely a "visualization tool"?

Yes, this is our suggestion. Especially that kube-prometheus stack already deploys it.