Alertmanager: Expose Alert severity metric

[pete-leese]

When building dashboards it would be useful if I could filter the severity from alert manager metrics.

Critical = 10
Warning = 5

Currently I do this using Prometheus datasource but unfortunately does not know what alerts are silenced therefore my dashboards are reporting inaccurate statistics.

If the alert severity was exposed in the alert manager metrics then I can configure grafana to look at the pdometheus alert manager datasource as opposed to Prometheus.

This feels like a bit of a gap / disconnect between Prometheus and alert manager.

Thanks.

Pete

[brancz]

Thanks for opening this issue! A problem I'm seeing about this is that the severity label, or in fact any label is just a convention, so it's difficult to expose metrics like this. I see the use for this though.

Maybe someone has an idea how to do this in a generic or completely different way to achieve the same result. Maybe @brian-brazil has a trick up his sleeve?

[brian-brazil]

This sort of thing doesn't belong on /metrics, as it's not about AM performance.

Even given that, doing it generically would be hard. It sounds in this case like the right solution is someone to write an AM data source for Grafana.

[brancz]

I recall @davkal was thinking of an Alertmanager datasource in Grafana, that seems like a reasonable/correct solution to me.

[roidelapluie]

https://grafana.com/plugins/camptocamp-prometheus-alertmanager-datasource/installation

[brancz]

@roidelapluie neat, thanks! I had not known about this plugin.

[roidelapluie]

Yes and you have the annotations too

[pete-leese]

Yes there is a data source for Alert Manager and I have tried using that but was not able to filter on severity :)

[brancz]

Well that just sounds like the plugin should be extended to support the full capability of the Alertmanager API :) (I have yet to try it out, so I don’t know what is and what’s not possible)

[pete-leese]

Looks like the plugin has recently been updated to bring back severity labels, but cannot see that I can pull back the number of silenced alerts - I have raised an issue with the plugin author.

[roidelapluie]

But .. on the other hand we have ALERTS in prometheus. Which is not about prom performance. Why can't we expose ALERTS in here, with 2 metrics:

alerts_processed
alerts_unprocessed

[brian-brazil]

ALERTS is not on Prometheus's /metrics. I'm not sure what you're proposing.

[roidelapluie]

it that inserted directly in the prometheus DB? oh that is a surprise for me.

[simonpasquier]

@VR6Pete it seems that using the AlertManager datasource for Grafana fulfills your requirements and there's no reason to expose alert counters on arbitrary labels.
If so can you close the issue?

[pete-leese]

Leave a comment