Kube-prometheus: Usage of prometheus-pvc

[huangqun]

It seems that by default prometheus will be created with an EmptyDir for the 'prometheus-k8s-db' volume, but that's not really a great idea for production servers that wants to keep the data even pod is relocated.

Could you suggest how we can make use of prometheus-pvc.jsonnet while generating the manifests for a kubeadm based cluster?

Thanks!

[brancz]

We used to have a storage guide before we pulled kube prometheus out of the Prometheus operator repo. We should bring that back.

cc @metalmatze

[wilmardo]

There's this example:

https://github.com/coreos/kube-prometheus/blob/master/examples/prometheus-pvc.jsonnet

[huangqun]

@wilmardo do you know if there's a generated yaml sample somewhere?

[wilmardo]

Not that I know of but this is what we use

local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet';
local pvc = k.core.v1.persistentVolumeClaim;

{
    prometheus+:: {
        prometheus+: {
          spec+: {
            retention: '30d',
            storage: {  // https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#storagespec
              volumeClaimTemplate:  // (same link as above where the 'pvc' variable is defined)
                pvc.new() +  // http://g.bryan.dev.hepti.center/core/v1/persistentVolumeClaim/#core.v1.persistentVolumeClaim.new

                pvc.mixin.spec.withAccessModes('ReadWriteOnce') +

                // https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#resourcerequirements-v1-core (defines 'requests'),
                // and https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota (defines 'requests.storage')
                pvc.mixin.spec.resources.withRequests({ storage: '1Mi' }) +

                // A StorageClass of the following name (which can be seen via `kubectl get storageclass` from a node in the given K8s cluster) must exist prior to kube-prometheus being deployed.
                pvc.mixin.spec.withStorageClassName('oceanstor-nfs'),
            },
          },
        },
      },
}

And it results in:

---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitor
roleConfig:
  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: prometheus-k8s-config
    namespace: monitor
  rules:
  - apiGroups:
    - extensions
    resourceNames:
    - prometheus
    resources:
    - podsecuritypolicies
    verbs:
    - use
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitor
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    beta.kubernetes.io/os: linux
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  retention: 30d
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  storage:
    volumeClaimTemplate:
      apiVersion: v1
      kind: PersistentVolumeClaim
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Mi
        storageClassName: oceanstor-nfs
  version: v2.9.2

[huangqun]

That helps, thanks!

[brancz]

@huangqun the whole point of kube-prometheus is that you generate everything yourself, the generated manifests are just for quick-start :)

[huangqun]

Yes @brancz that's the next step! :D

[roshan8]

@brancz @wilmardo we tried building /examples/prometheus-pvc.jsonnet
This generated right prometheus yaml file. However,/prometheus/setup seems to be empty

[thkukuk]

This generated right prometheus yaml file. However,/prometheus/setup seems to be empty

Look at the files generated by the prometheus-pvc.jsonnet example, it's 0prometheus-operator-* and not setup/prometheus-operator-*