Alertmanager: Blackhole receiver

[aecolley]

Create a receiver which drops all alerts. Currently this requires a workaround involving email or webhook receivers and a dummy server.

Rationale: alertmanager should fully support the page/ticket/log model for alert severity. Alerts of "log" severity should be displayed on a dashboard but otherwise generate no notifications. Unless there's a plan to exclude them from being sent by prometheus to alertmanager, they must be dropped .

[brian-brazil]

https://github.com/prometheus/prometheus/issues/1744

It is not desirable for performance reasons to have non-notifying alerts in the alertmanager.

[aecolley]

I agree. It's better not to send them at all. How?

[beorn7]

I don't know how to not send an alert at all, but it is perfectly possible to set up a blackhole receiver.
At SoundCloud, we use info level alerts where the (still to be improved) Alertmanager UI serves as an "alerts dashboard" (as suggested by my SRECon talk).

Just configure a receiver with a name only:

receivers:
- name: something-info
  # Deliberately left empty to not deliver anywhere.
- name: some-other-receiver
  email_configs:
  # ...

[beorn7]

I'll close this as blackhole receivers are in fact possible.
The case where you don't want to send an alert notification to AM in the first place is a Prometheus side issue covered in prometheus/prometheus#1744 .

[NightTsarina]

I was trying to do exactly this today, and I could not find how. Wouldn't it be good to add this in the documentation?

[roidelapluie]

Maybe. Note: You can also drop this at prometheus level (not send certain alerts at all):

alerting:
  alert_relabel_configs:
  - source_labels: [time_window]
    regex: never                                                                                                                          
    action: drop                                                                                                                          

[metost]

Maybe. Note: You can also drop this at prometheus level (not send certain alerts at all):

alerting:
  alert_relabel_configs:
  - source_labels: [time_window]
    regex: never                                                                                                                          
    action: drop                                                                                                                          

@roidelapluie This is pretty interesting, can you elaborate a bit more, for example if I want to drop specific alarm before it goes to alertmanager. Let's say If I want to do this for "alertname": watchdog?
Thanks in advance!