Alertmanager: Feature request: "script receiver"

[ahajnal]

Dear Developers,
I don't know if it is possible to implement but I would be interested in having a sort of "script" receiver, which would execute a user-defined (bash) script on alert events, like:

receivers:

• name: script
  script_configs:
  
  
  
  • script: /home/user/myalert.sh
  
  

If possible, working in a way that alert.sh either get alert json string as argument, or it would be even greater that the script could be templated (with like some {{label=''}} stuff).

I ask it because with using webhook I have to deploy another component to receive alert POSTs
(nor I can configure webhook to send GET/PUT/... with the desired content) and do something special I need.

In this way alertmanager would be more flexible. As I remember Zabbix can also run scripts on triggers.

What do you think?

Best regards,
Akos Hajnal

[fabxc]

Hi Akos,

I get that the webhook is another hop, which is an inconvenience to setup.
Templating the script itself is not possible from a security perspective.

I'm open to discuss this and would like to hear more opinions and whether
more people feel a need for something similar.

Fabian

On Tue, May 17, 2016 at 11:16 AM ahajnal [email protected] wrote:

Dear Developers,
I don't know if it is possible to implement but I would be interested in
having a sort of "script" receiver, which would execute a user-defined
(bash) script on alert events, like:

receivers:

• name: script script_configs:
  
  
  
  • script: /home/user/myalert.sh
  
  

If possible, working in a way that alert.sh either get alert json string
as argument, or it would be even greater that the script could be templated
(with like some {{label=''}} stuff).

I ask it because with using webhook I have to deploy another component to
receive alert POSTs
(nor I can configure webhook to send GET/PUT/... with the desired content)
and do something special I need.

In this way alertmanager would be more flexible. As I remember Zabbix can
also run scripts on triggers.

What do you think?

Best regards,
Akos Hajnal

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/prometheus/alertmanager/issues/354

[brian-brazil]

I'm against scripts, too much of a reliability threat (how well do you deal with 100 forks+exec of a script per second?) and there's the security risks too. There's nothing stopping development of a webhook utility that runs scripts.

[ahajnal]

Hi Fabian,
To add some further details... In my special case, I would like not only to monitor something and get alerts from Prometheus-Alertmanager but to react actively on events - using as few components as possible. Script is something that can be quite active. What about a REST receiver, or a more generic receiver in which users can configure HTTP method/headers/payload? This avoids forks+exec and is similar to webhooks.
Akos

[discordianfish]

There's nothing stopping development of a webhook utility that runs scripts.

I've written exactly this. It takes a script as arg and listens on http for prometheus webhook. Will try to open source it and ping you.

[ahajnal]

Hi discordianfish,
That will be a definitely useful tool not just for alertmanager.
Akos

[discordianfish]

@ahajnal Just opened it up: https://github.com/imgix/prometheus-am-executor

[ahajnal]

Looks great!

[fabxc]

Closing as a we now even have a webhook implementation solving this.

[urashidmalik]

Not sure if you guys looked at monit.
https://mmonit.com/wiki/Monit/ConfigurationExamples

Which provides option for exec, pretty helpful.