Kube-prometheus: Inhibition rules not inhibiting common info alerts

[brancz]

What happened?

The CPUThrottlingHigh alert fired in the Polar Signals production cluster and caused a notification, even though no warning or critical alert was firing.

Did you expect to see some different?

The CPUThrottlingHigh alert to be inhibited unless there are other alerts firing.

I would suggest to remove the inhibition rule to require the alertname to be equal, and just have the namespace to be required to be equal, but I'm happy about any other suggestions that would lead to the same result.

How to reproduce it (as minimally and precisely as possible):

This should be reproducible on any cluster that has the alert, node-exporter causes it to fire pretty often.

Environment

GKE 1.18

@prometheus-operator/kube-prometheus-reviewers

[paulfantom]

Seems like this would require to have an inhibition when no alerts are firing (inhibit CPUThrottlingHigh when everything is ok and do not inhibit it when an alert is firing), sadly inhibition is not meant to do this currently.

cc @simonpasquier for confirmation on above

[brancz]

This should now be possible with negative matching: https://github.com/prometheus/alertmanager/issues/1023

[ArthurSens]

But what is the request here? Does kube-prometheus provide an alertmanagerconfig that should work for everyone using this project?

I think that alertmanager configuration is something that doesn't have a one-for-all solution 🤔

[paulfantom]

We do provide a basic alertmanager configuration with some predefined inhibition rules. The idea is to expand this default one with negative matching to prevent the alert from firing when not necessary. This will give guidance to other users.

[ArthurSens]

We do provide a basic alertmanager configuration with some predefined inhibition rules. The idea is to expand this default one with negative matching to prevent the alert from firing when not necessary. This will give guidance to other users.

Ah, nice. That makes sense :)

but I'm reading the docs for the inhibition rules, I got the impression that inhibiting an alert if another(s) isn't firing is still not possible. It says:

A list of matchers for which one or more alerts have to exist for the inhibition to take effect.

# DEPRECATED: Use target_matchers below.
# Matchers that have to be fulfilled in the alerts to be muted.
target_match:
  [ <labelname>: <labelvalue>, ... ]
# DEPRECATED: Use target_matchers below.
target_match_re:
  [ <labelname>: <regex>, ... ]

# A list of matchers that have to be fulfilled by the target 
# alerts to be muted.
target_matchers:
  [ - <matcher> ... ]

# DEPRECATED: Use source_matchers below.
# Matchers for which one or more alerts have to exist for the
# inhibition to take effect.
source_match:
  [ <labelname>: <labelvalue>, ... ]
# DEPRECATED: Use source_matchers below.
source_match_re:
  [ <labelname>: <regex>, ... ]

# A list of matchers for which one or more alerts have 
# to exist for the inhibition to take effect.
source_matchers:
  [ - <matcher> ... ]

# Labels that must have an equal value in the source and target
# alert for the inhibition to take effect.
[ equal: '[' <labelname>, ... ']' ]

From what I understood, an alert must be firing for this one to be inhibited