Alertmanager: duplicate alert groups returned by api

Created on 3 May 2019 · 6Comments · Source: prometheus/alertmanager

What did you do?

view the alert list page via the ui

What did you expect to see?

non-duplicated alert groups

What did you see instead? Under which circumstances?

some alert groups are duplicated. i'm not sure why yet.

Environment

System information:

insert output of uname -srm here
Alertmanager version:

0.17.0

Of the 110 currently active alert groups at soundcloud, only 2 of the groups are duplicated (for a total of 4 groups being returned from the api, when only 2 of them should be). The groups appear to be perfectly duplicated.

componenui kinbug

Source

stuartnelson3

Most helpful comment

I also see duplicate groups. Version 0.18.

Screen Shot 2019-07-10 at 01 04 42

psalaberria002 on 10 Jul 2019

👍5

All 6 comments

This might be an artifact of that particular group's configuration. It only affects a single team. I'm thinking there might be some race condition in creating the groups internal to alertmanager when the routes are configured in a certain way. This would also explain why they have been seeing duplicate messages.

stuartnelson3 on 3 May 2019

We currently see a duplicated alert in the UI with 0.17.0 but not with amtool alert query. Clicking "+ Info" on either expands both alerts in the UI. Not sure if it is the same issue.

taisph on 24 Jun 2019

I also see duplicate groups. Version 0.18.

Screen Shot 2019-07-10 at 01 04 42

psalaberria002 on 10 Jul 2019

👍5

I can reproduce as well. It happens when I configure routes with the same group_by and continue: true. I'll dig into it...

simonpasquier on 11 Jul 2019

https://github.com/prometheus/alertmanager/pull/1964 has fixed the problem on the API v2 side but there's still a glitch in the UI: when several groups have the same grouping labels, clicking on the +/- button will expand/collapse all of them.

screencast