Kube-prometheus: Docs: update documentation on filtering prometheus rules

[J-Are]

Hi, I would like to update "CPUThrottlingHigh" alert expression so I followed instructions in the Readme.md and I use this snippet of code in my example.jsonnet.
```local filter = {
prometheusAlerts+:: {
groups: std.map(
function(group)
if group.name == "kubernetes-resources" then
group {
rules: std.filter(function(rule)
rule.alert != "CPUThrottlingHigh",
group.rules
)
}
else
group,
super.groups
),
},
};

local update = {
prometheusAlerts+:: {
groups: std.map(
function(group)
if group.name == "kubernetes-resources" then
group {
rules: std.map(
function(rule)
if rule.alert == "CPUThrottlingHigh" then
rule {
expr: "sum(increase(container_cpu_cfs_throttled_periods_total{container!=\"\",namespace!~\".*slaves\" }[5m])) by (container, pod, namespace)/sum(increase(container_cpu_cfs_periods_total{}[5m])) by (container, pod, namespace)> ( 25 / 100 )"
}
else
rule,
group.rules
)
}
else
group,
super.groups
),
}
};

local kp = (import 'kube-prometheus/main.libsonnet') + filter + update +
// Uncomment the following imports to enable its patches
// (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
// (import 'kube-prometheus/addons/managed-cluster.libsonnet') +
// (import 'kube-prometheus/addons/node-ports.libsonnet') +
// (import 'kube-prometheus/addons/static-etcd.libsonnet') +
// (import 'kube-prometheus/addons/custom-metrics.libsonnet') +
// (import 'kube-prometheus/addons/external-metrics.libsonnet') +

{
values+:: {
common+: {
namespace: 'monitoring',
},
},
};

{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
{
for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ 'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule } +
{ 'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }`

Afterwared, I ran the command `./build.sh example.jsonnet` and the manifests generated by this script are stored in manifests folder. 
Unfortunately, my rule CPUThrottlingHigh did not change :( 

I used the **main** branch but the stranger thing is when I generate the manifests from **release-0.4 branch** with nearly the same code, it works.

This is the following snippet of code I use (example.jsonnet file always):

```local filter = {
   prometheusAlerts+:: {
     groups: std.map(
       function(group)
         if group.name == 'kubernetes-resources' then
           group {
             rules: std.filter(function(rule)
               rule.alert != "CPUThrottlingHigh",
               group.rules
             )
           }
         else
           group,
       super.groups
     ),
   },
};

local update = {
   prometheusAlerts+:: {
     groups: std.map(
       function(group)
         if group.name == 'kubernetes-resources' then
           group {
             rules: std.map(
               function(rule)
                 if rule.alert == "CPUThrottlingHigh" then
                   rule {
                     expr: "sum(increase(container_cpu_cfs_throttled_periods_total{container!=\"\",namespace!~\".*slaves\" }[5m])) by (container, pod, namespace)/sum(increase(container_cpu_cfs_periods_total{}[5m])) by (container, pod, namespace)> ( 25 / 100 )"
                   }
                 else
                   rule,
                 group.rules
             )
           }
         else
           group,
       super.groups
     ),
    }
};

local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') + update +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
  // (import 'kube-prometheus/addons/managed-cluster.libsonnet') +
  // (import 'kube-prometheus/addons/node-ports.libsonnet') +
  // (import 'kube-prometheus/addons/static-etcd.libsonnet') +
  // (import 'kube-prometheus/addons/custom-metrics.libsonnet') +
  // (import 'kube-prometheus/addons/external-metrics.libsonnet') +

{
     values+:: {
       common+: {
         namespace: 'monitoring',
       },
     },
};

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

Thank you in advance for your help
Regards,
Julien

[pschulten]

Having the same issue. The documentation is outdated. Openshift has a config that (I assume) works, but my jsonnet-fu fails me to adapt this to my setup :smile_cat:

https://github.com/openshift/cluster-monitoring-operator/blob/a6bc9824035ceb8dbfe7c53cf0c138bfb2ec5643/jsonnet/patch-rules.libsonnet

@paulfantom do you have an example for mere mortals?

[paulfantom]

@pschulten Some mixins can be customized just by adjusting their _config map. This is exposed now (release-0.8 and newer) in kube-prometheus, for example for kubernetes it would be in $.values.kubernetesControlPlane.mixin._config. This works for example for changing CPUThrottlingHigh threshold as seen in this example. If the config is not enough I would kindly ask you to open an issue in the mixin repository to notify maintainers that this should be customizable.

However, if you need to change things fast, you have to remember that mixins are not stored in global objects anymore (hence manipulating prometheusAlerts won't work. Depending on how many rules you need to patch, there are 2 methods:
1) One when you have a small number of alerts to patch and they are located in one group
2) One when you need to patch multiple alerts across all groups

The latter one is OpenShift case and if you really need to use it, then we as maintainers probably failed somewhere :( The former one is similar to what was present in previous releases so it may be easier to grasp. For example, If you need to patch PrometheusDuplicateTimestamps to use 1h for duration, you can do sth like:

prometheus+: {
  prometheusRule+: {
    spec+: {
     groups: std.map(
       function(group)
         if group.name == 'prometheus' then
           group {
             rules: std.map(
               function(rule)
                 if rule.alert == "PrometheusDuplicateTimestamps" then
                   rule {
                     for: '1h',
                   }
                 else
                   rule,
                 group.rules
             )
           }
         else
           group,
       super.groups
     ),
    }
    }
  }
}

As you can see the format is the same as it was before, the difference is that now it is not applied to $.prometheusAlerts but directly to a relevant prometheusRule.spec object.

[pschulten]

Thank you very much!

I created a helper function for removing unwanted alerts:

{
  filterAlerts(alerts):: {
    spec+: {
      groups: std.map(
        function(group)
          group {
            rules: std.filter(
              function(rule)
                if std.objectHas(rule, 'alert') then
                  std.count(alerts, rule.alert) == 0
                else true,
              group.rules
            ),
          }, super.groups
      ),
    },
  },

}

and use it like this:

{
  // ...
  kubernetesControlPlane+: {
    prometheusRule+: util.filterAlerts(['KubeClientCertificateExpiration']),
  },
  // ...
}

[J-Are]

@paulfantom and @pschulten Thank you for your help and your returns, I will test that for my use case :)

[paulfantom]

Could anyone create a PR with update to our docs with the snippet from @pschulten ?

[ArthurSens]

I would say that this function also be awesome to have in the jsonnet/kube-prometheus/lib directory! WDYT?

[paulfantom]

I would be cautious about adding code snippets to jsonnet/kube-prometheus/lib. Especially ones we don't use directly as those have a bad tendency to convert into stale code. However, if we use this snippet in https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/addons/managed-cluster.libsonnet#L9-L20 then I am 100% fine with adding it to lib/.

That said, docs update is a must and adding this to lib/ is a nice to have :)

This potentially could be done in one PR by doing the following

1. Add snippet to jsonnet/kube-prometheus/lib
2. Amend https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/addons/managed-cluster.libsonnet#L9-L20 to use the snippet
3. Use embedmd to include the snippet in docs + amend docs a bit