Alertmanager: Alertmanager integration with Discord

[rossigee]

Discord is the new Slack/Hipchat for many organisations, including for one of my clients.

This ticket is to cover adding Discord support, based on the Slack/Hipchat implementations.

[rossigee]

I've done most of the work here...

https://github.com/rossigee/alertmanager/tree/add-discord-support

It largely works, and when configured will send a brief message to Discord via the configured webhook URL, but the notification generation code is still lacking, so the main detail from the alerts aren't being passed through.

This is my first time using Go. It seems pretty straightforward so far, but It's taken me a few hours to get this far (mostly toolchain issues), and now I'm struggling with simple stuff, like debugging/developing the part that iterates the array of alerts to produce a set of embeds for the discord notification. If anyone with a better understanding of Go (and alertmanager) could help me figure out how best to generate the notification content, that would be massively appreciated.

Only other thing of note is that it's also failing one of the retry tests for HTTP response 429, but I haven't been able to figure out why.

[simonpasquier]

Unfortunately as stated in the documentation, no new receiver is being added to AlertManager for the time being. The recommended approach is to use a standalone service that will get notifications from the webhook receiver and push it to the third-party service. It seems that https://github.com/benjojo/alertmanager-discord already does this for Discord. If it suits your needs, it can be listed in the Prometheus documentation on the integrations page.

Again thanks for the effort you've put into this.

[rossigee]

Thanks for bringing that to my attention @simonpasquier.

Unfortunately benjojo's service doesn't suite me too well. As it stands, it has no run-time configurability, no build script/docs, no packaging etc. It's a bit too raw, and my lack of experience with the Go toolset means that I won't be in a position to finish cooking it any time soon. Until/unless it's polished up a bit, I can't see it being useful to anyone else unless they're also a Go hacker with some spare time on their hands.

For now, I've created a simple NodeRED flow which works well for us using the webhook notification mechanism.

To be honest, if every organisation that uses Discord and wants to use Alertmanager has to find/write and maintain yet more webservices on their infrastructure just so Alertmanager can send them alerts, that's going to add up to a lot of technical man hours that could be put to better use, plus a lot of people that probably wouldn't bother with alertmanager at all. Especially if users of Hipchat and Slack can achieve the same goal with a much simpler setup, but the door is closed for Discord users to be able to use it in the same way.

Is there a particular reason you don't want to accept any more mechanisms? It seems to me it might be because to implement a mechanism for every new IM platform that ever popped up would involve making the monolithic 'notify/impl.go' and the template file even more bloated/kludgy? If it's not already obvious, you should really probably consider breaking the different notification mechanisms etc in that file out into their own files, folders (or repos) to make it all more modular and generally easier to work with and extend.

[simonpasquier]

Is there a particular reason you don't want to accept any more mechanisms?

Every new receiver increases mechanically the maintenance burden. The history has shown that while people are willing to contribute new receivers, it is often difficult for them to follow up. And it isn't only about implementing the notification code but also extending the AlertManager's configuration as well as writing the documentation. If you look at the sheer number of IM platforms, it would be impossible to support all of them and a line has to be drawn somewhere. You check #701 and #1259 for previous discussions on the topic. Again I definitely understand and share your frustration.

[luceos]

@rossigee just wanted to point out that discord supports slack compatible webhooks. Any service that allows you to specify a (complete) slack webhook endpoint allows you to use discord too. In your discord channel create a webhook and retrieve the url; then just append /slack to the end.

This should work with the current prometheus integration as well.

[ScrumpyJack]

@luceos do you have a working an example?

[Just-Insane]

Has anyone gotten this working? I have used @luceos suggestion in the past, however, it does not seem to work with the slack_config in alertmanager.

Please refer to https://github.com/grafana/grafana/issues/7964 which notes that Discord's slack support does not work well. Trying the workaround provided above results in error 400 on Discord's end.

There is also a PR (https://github.com/grafana/grafana/pull/7926) with example code for Discord notifications.

[cocuba]

this image works fine with discord https://hub.docker.com/r/funkypenguin/alertmanager-discord , set env with the url and that's it (i'm using it in k8s)