Alertmanager: grouping alerts

Created on 14 Feb 2017  路  10Comments  路  Source: prometheus/alertmanager

problem: can't disable grouping (to make it in opsgenie)
(two alerts are two diff cpus of same instance)
AM config:

route:
  receiver: 'dev_null'
################################
  routes:
  - receiver: 'opsgenie'
    group_wait: 0s
    group_interval: 0s
    continue: true

receivers:
 - name: dev_null

 - name: "opsgenie"
   opsgenie_configs:
   - api_key: XXX
     teams: support_team
     tags: '{{ range .Alerts }}{{ .Labels.region }} {{ end }},{{ range .Alerts }}{{ .Labels.Type }} {{ end }}, {{ range .Alerts}}{{ .Labels.severity }} {{ end }}'
     details: { 'instance': '{{ range .Alerts }}{{ .Labels.instance }} {{ end }}' }

expected to get all the alerts not grouped

what happens:
in opsgenie:
message:
Description:
(i-48fa1def) in region reported over 90% CPU utilization in the past 5 minutes. Monitored by job: . Public IP:
Alerts Firing:
Labels:

  • alertname = CpuUtilizationWarning
  • cpu = cpu3
  • instance = xxx
  • region = xxx
  • service =
  • severity = warning
  • value = 94%
    Annotations:
  • description = (xxx) in region reported over 90% CPU utilization in the past 5 minutes. Monitored by job: . Public IP:
    Source: http://xxx:9090/graph?g0.expr=floor%28100+-...
    Labels:
  • alertname = CpuUtilizationWarning
  • cpu = cpu0
  • instance = xxx
  • region = xxx
  • service =
  • severity = warning
  • value = 90%
    Annotations:
  • description = (xxx) in region reported over 90% CPU utilization in the past 5 minutes. Monitored by job: . Public IP:
    Source: http://xxx:9090/graph?g0.expr=floor%28100+-...

Most helpful comment

group_by: ['instance'] groups by each mchine

All 10 comments

I'm having a similar problem with Slack integration. There are no group_ properties in my routing tree.

In my case I tried to group by a dummy label (group_by: ['foobar']) and alerts still get grouped

Disclaimer: This is just a guess/opinion.
While it might be nice to have an option to say "Don't group", the implementation of "group_by" is to group by the value of the labels provided, not by the labels themselves. In other words, if you have a label that has multiple values that you don't want grouped together, put that label in the "group_by" array. For the OP, this would simply be group_by: ['cpu']

Closing here, feel free to reopen here in case this is an issue with Alertmanager or on Prometheus users Google groups in case of a usage question.

I think there is (the same in Google groups) still no answer: is it possible to disable grouping at all in Alertmanager?
We face the same issue and still have no success with solving it (

group_by: ['instance'] groups by each mchine

@semyonslepov @vivekthangathurai do you guys found a way to not group at all? I think even group_by: ['instance'] is still bad, there should be a disable_grouping: true option

You can use group_by: [...] to disable grouping.

@roidelapluie I tried that, all the open alerts are sent together to opsgenie.

@fernandocarletti It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

Was this page helpful?
0 / 5 - 0 ratings