Alertmanager: have alertmanager provide log with sended alert ?

[jinhang]

I want to have alert history log when some alerts send successful. could you give me some advice?

[jinhang]

I want to have all alert's send email history.

[fabxc]

We do have an internal log with meta information about which notifications where sent. We want to expose it at some point via API.

It sounds though that you want to have a full log of the entire emails being sent?
That rather sounds like archival which your email server handles for you – you can just CC an email address where you store all alert emails forever if you want.

[jinhang]

yeah,I try to use api/v1/alerts/groups to have some information,but it is not what I want. thanks for your email server handles for you but it may be not to be available. It seems I want to have all notifications have been sended ,like logs. I think this information should in AlertManager Server,to have all sended notifications history records.

[jinhang]

like this:

[brancz]

If the email server is unavailable then the Alertmanager will retry until timeout (GroupInterval). At that point there will be logs saying the email could not be delivered due to unavailability, logs of retries, logs of retry timeouts, and meta information of the notification that was attempted to be sent via the API.

Can you elaborate how additional information (eg. message body) would improve the situation?

[jinhang]

oh no. Available is mean that I can not get some information about email sended

[jinhang]

I want to get alertmanager send notifaction history via email .

[jinhang]

like use alertmanager API api/v1/alerts/groups have like this:

{
    "status":"success",
    "data":[
        {
            "labels":{
                "alertname":""
            },
            "blocks":[
                {
                    "routeOpts":{
                        "receiver":"team-X-mails",
                        "groupBy":[
                            "alertname"
                        ],
                        "groupWait":3000000000,
                        "groupInterval":5000000000,
                        "repeatInterval":3000000000
                    },
                    "alerts":[
                        {
                            "labels":{
                                "alertname":"",
                                "instance":"",
                                "job":"",
                                "monitor":"codelab-monitor",
                                "severity":"fatal"
                            },
                            "annotations":{
                                "description":"",
                                "summary":""
                            },
                            "startsAt":"2016-11-18T15:48:09.803+08:00",
                            "endsAt":"0001-01-01T00:00:00Z",
                            "generatorURL":"",
                            "inhibited":false,
                            "silenced":18
                        }
                    ]
                }
            ]
        }
   }
}

But It is current alerts.

[jinhang]

I want to have the firing alerts history.

[jinhang]

@brancz

[brancz]

As far as I understand you want each email to contain the previously sent emails, to have a history of sent (related) emails when you receive one. I feel this is simply a restriction by using email, if you used other systems like pagerduty this is part of their service. Wdyt @fabxc ?

[jinhang]

no,I want to have that the firing state alerts send to Alertmanager server. I ask it alert history.

[brancz]

If I understand correctly, you want to be able to display the history of alerts received by the Alertmanager (accessible through the Alertmanager's web UI), which were sent by the Prometheus instance(s). The problem with that is that in as soon as you have a HA setup this will break, as the design of the alerting mechanisms of Prometheus/Alertmanager expect multiple (HA) Prometheus instances to fire all alerts against all Alertmanager instances, which would in turn mean, that a specific fired alert will occur up to the number of instances of Prometheus that are firing that alert. Additionally, in a production environment with multiple hundred targets this would consume very large amounts of memory and/or storage. Does that make sense?

[jinhang]

yes,you understand correctly,but I want to display the history of alerts received.even it consume very large amounts of memory ,I plan to user database manage this history of alerts, because this information can use in analysis of business.

[jinhang]

@brancz @fabxc hi，can I ask your a question? How to use our internal notification system with firing alert? should I add function into source code? (for example implements the Notifier interface.)

[fabxc]

For the case of your internal notification system as well as for your archival needs, you'll want to configure AM to send notifications to a webhook.
You can then relay this into your own notification system as well as into a system archiving all notifications.

https://prometheus.io/docs/alerting/configuration/#webhook-receiver-webhook_config

[jinhang]

if I want to use SMS to notify,The configuration file can be resolved?

[jinhang]

That is to say other notification system can also be configured to solve?

[brancz]

The contract between Prometheus and Alertmanager is that Prometheus keeps firing alerts until the alert evaluation does not trigger anymore and then once the resolve_timeout has passed sends a resolved notification if send_resolved is configured for that receiver.

[jinhang]

@brancz if I want to use SMS to notify,The configuration file can be resolved?

[brancz]

I'm sorry but I don't understand what you mean by "The configuration file can be resolved".

You can send notifications via the available receiver providers that support SMS. If none of the services are an option for you, then you can use a webhook receiver and write your own small application that sends the notification to the provider of your choice.

[jinhang]

@brancz
Thanks,Could you give me an example?

[brancz]

Unfortunately I don't have any insight into people doing this, it might be worth collecting those integrations in our docs.

For local testing purposes we have a small script which simply outputs the payload received by the alertmanager. This is the configuration used to send notifications againt that server.

[iksaif]

I was thinking about adding a "log" / "syslog" notification plugin, would that help ?

[poblahblahblah]

If I am understanding the request and response correctly, the requester is wanting to know if AM stores an alert history that they can query later on. We need something similar to this for SOX compliance. Are you suggesting that the path of least resistance is for us to just write an HTTP endpoint that will have all alerts sent to it so we can just query it whenever we need to?

[stuartnelson3]

There is currently no stored alert history that can be later queried.

Are you suggesting that the path of least resistance is for us to just write an HTTP endpoint that will have all alerts sent to it so we can just query it whenever we need to?

Yeah, that would be the fastest way to get a log of sent alerts.

[iksaif]

We use https://github.com/cloudflare/alertmanager2es

[LuboVarga]

What about to use ALERTS metrics directly from promerheus? Is it enough?

[Kamori]

Having a similar situation to this. ALERTS works fine for now, however, we will soon be adding alerts to alertmanager from sources not tracked by Prometheus. And they don't make sense to be collected by prometheus.

The alertmanager2es looks like the easiest solution atm. However, the goal is we want to track Alertname + arbitrarily defines labels on each alert, and use that data to trend out how frequently a type of alert occurs per customer.

A simple webhook accepting webhook seems like a reasonable solution, but it would be great if the alertmanager exporter has some of this built in, even if toggleable

[brancz]

You want a consistent database for this type of data, which is why Alertmanager is never going to correctly be able to expose this type of data. The webhook receiving and writing this into a consistent data store sounds like a good idea.

[pcarranza]

I've built https://gitlab.com/yakshaving.art/alertsnitch which saves alerts to a MySQL database which can then be used as a log.

It stores the whole alert payload.

This may work as a workaround, maybe?

[brancz]

That’s not a workaround, that’s a correct way to do it :) as I mentioned above, alertmanager will never be able to offer what was asked for here, so something like that is the right strategy.

[simonpasquier]

@pcarranza if you wish, you can add a reference to your project in the Prometheus documentation as I don't see a similar entry there.

[pcarranza]

Would it make sense to document and then close this issue?

I can take the documentation part myself.

[mxinden]

That sounds good to me. If there are not objections I would suggest a pull request at https://github.com/prometheus/docs/. Thanks @pcarranza.

[pcarranza]

@mxinden it's documented and published already.

[simonpasquier]

@pcarranza thanks! Closing it now.