Alertmanager: How to Alertmanager running shell-script

[ihtiking]

Server CentOS7 and intalled alertmanager in OpenShift 3.11.

When some alert is triggered, i want run shell-script.

Now, yml-file in Config Map alertmanager contains block receivers:

• name: alert1
  webhook_configs:
  
  
  
  • url: http://......
  
  

and he work fine.

In end point i want run command "curl -X POST "https://some-site/?var=text1""

Please, help.

Linux 3.10.0-862.14.4.el7.x86_64 x86_64

alertmanager, version 0.15.1 (branch: HEAD, revision: 8397de1830f154535a31150f9262da0072d8725d)
build user: root@prometheus-alertmanager-binary-3-build
build date: 20180720-09:32:24
go version: go1.10.3

prometheus, version 2.3.2 (branch: HEAD, revision: 71af5e29e815795e9dd14742ee7725682fa14b7b)
build user: root@prometheus-binary-6-build
build date: 20180720-09:19:12
go version: go1.10.3

[simonpasquier]

You should have a look at https://github.com/imgix/prometheus-am-executor or https://github.com/adnanh/webhook

[ihtiking]

Big thanks. I'm gotting this - https://github.com/adnanh/webhook. This is solution IMHO very simple.

My steps:

1. Create hooks.json:
   [
   {
   "id": "runscript",
   "execute-command": "/usr/bin/sendmessage.sh",
   "command-working-directory": "/usr/bin"
   }
   ]

1. Create shell-script sendmessage.sh:
   

   !/bin/bash

curl -I -X POST "https://"

1. Create Dockerfile:
   

   FROM almir/webhook - this is string i used when create compiled file "webhook" in first build docker.

FROM centos:7
COPY webhook /usr/bin/
COPY hooks.json /var/webhook/
COPY sendmessage.sh /usr/bin/
RUN chmod +x /usr/bin/sendmessage.sh
CMD ["/usr/bin/webhook", "-verbose", "-hooks=/var/webhook/hooks.json", "-hotreload"]

1. Create some folder and copy here webhook-binary file, hooks.json and shell-script sendmessage.sh.

2. Next, run command:
   docker build -t webhook-image . --tag=webhook-image && docker run --network="host" -d -p 9000:9000 --name=webhook webhook-image && docker start webhook

3. Next in browser open url http://:9000/hooks/runscript
   When open your site on port 9000, webhook is executed shell-script.

It's very great!!!

[simonpasquier]

I'm closing the issue then.

[mchugh19]

The proposed webhook service seems like a bit more work and additional points of complexity and failure than having alert manager support script running. Can this issue be reopened as a feature request?

[simonpasquier]

@mchugh19 unfortunately no:

1. Alertmanager's scope isn't about remediation but rather notifying 3rd party systems.
2. We don't accept new integrations currently and the webhook integration is the recommended approach.

[alwaysastudent]

It would be great if alert manager can support running shell scripts.

[sukhinin]

Running external scripts is the standard and flexible way to integrate with third-party systems. Definitely would like to see it supported.

[dgl]

https://github.com/adnanh/webhook is a generic webhook receiver written in Go, you can easily use this with alertmanager's webhooks in order to run shell commands.

[sukhinin]

@dgl I'm aware of it. That's one more service we have to run, monitor and support. It just makes system more complex and adds yet another moving part.

[MalloZup]

in case it help anyone, I do have written a blogpost for helping people using this pattern:

https://mallozup.github.io/posts/self-healing-systems-with-prometheus/