Alertmanager: AlertManager not sending summary or description

[jrxFive]

I believe this is just something I haven't configured properly but all notifications triggered are only sending the following format:

[FIRING|RESOLVED:#] ALERT_NAME (LABELS VALUES...)

Notifications received:

[FIRING:1] KUBERNETES_STATUS (172.17.4.202 kubernetes-cluster 172.17.4.202 node)
[RESOLVED] KUBERNETES_STATUS (172.17.4.202 kubernetes-cluster 172.17.4.202 node)

Labels available:
instance="172.17.4.202" job="kubernetes-cluster" kubernetes_io_hostname="172.17.4.202" kubernetes_role="node"

Example Alerting Rule:

ALERT KUBERNETES_STATUS
  IF up{job="kubernetes-cluster"} == 0
  FOR 1m
  ANNOTATIONS {
    summary = "Kubernetes instance: {{$labels.instance}} not responding",
    description = "Kubernetes instance: {{$labels.instance}} not responding",
  }

Currently using 0.1.0-beta2, and the slack notifier. I am unsure if I need to specify more information in my receiver to structure the alert properly.

[brian-brazil]

The defaults for chat-type systems like slack are not to include the annotations, as that tends to be a lot of lines that gets in the way of ongoing conversations and operational response.

For example if your alert was called something like KubernetesInstanceDown then the summary/description add no further information and you can always drill down further from the link.

[jrxFive]

Ahh makes sense thanks for the quick reply!

[juliusv]

@brian-brazil Not sure if that was clear to @jrxFive from your reply, but you _can_ use custom templates that include the annotations.

[cvielma]

Hi @juliusv can you elaborate on that or point me into some documentation related to that? I'd like to know how to send the alert either using a template or not, but with the summary and description.

[juliusv]

@cvielma Unfortunately there's no full documentation on Alertmanager notification templating yet. There's a blog post to help you get started with the basics: https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/

For examples of what you can do, have a look at the default templates that you can override in your AM config file: https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl

[cvielma]

Thanks!

[Vannevelj]

@juliusv Could you give an example of how to use the annotations?

For example consider the following alert:

ALERT InstanceDown
IF up == 0
FOR 30s
ANNOTATIONS {
    summary = "Server {{ $labels.Server }} is down.",
    description = "{{ $labels.Server }} ({{ $labels.job }}) is down for more than 30 seconds."
}

receivers:
- name: general_receiver
  slack_configs:
  - api_url: https://hooks.slack.com/services/token
    channel: "#telemetry"
    title: "My summary"
    text: "My description"

What do I change in my receiver to get this data?

[asbjornenge]

@Vannevelj did you figure out this? I'm having the same issue. Been trying loads of different variable names without any luck :stuck_out_tongue_closed_eyes:

[ivankovnatsky]

Guys, have a look at this: https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates.

[ivankovnatsky]

I think as the software develops, there will be more examples to help people to get into.

[Vannevelj]

@asbjornenge I did, actually. I used a custom template which can access to variables. I'm at the gym now so can't link to it but search for me on Stack Overflow (Jeroen Vannevel) and look at my most recent question or wait about an hour until I get home ;-) Will update this soon

[asbjornenge]

@sevenfourk yes, I have read that article and it only addresses GroupLabels. I've also studied template.go which leads me to believe that .CommonAnnotations should include the annotations, but I end up with an empty string.

@Vannevelj excellent, thanks! :+1:

[Vannevelj]

@sevenfourk one issue with that blogpost is that it's very general and there appear to be magic variables everywhere. Certainly doesn't help when you don't know Go :D

[ivankovnatsky]

@Vannevelj, @asbjornenge, sorry guys, I was thinking you would go for comments, anyway:

I did like this, nothing fancy, for now it works for me:

title: '{{ .CommonAnnotations.summary }}'
text: '{{ .CommonAnnotations.description }}'

that picks my summary and description in prometheus' alert.rules

[asbjornenge]

@sevenfourk what version of the alertmanager are you using? I'm using v0.4.2, but I only get empty strings for those two... Maybe I need to upgrade prometheus...?

My alert struct is like this:

[{ 
  "labels": { 
    "alertname": "node_cpu",
    "instance": "10.247.14.196:9101",
    "monitor": "taghub" 
  },
  "annotations": { 
    "description": "10.247.14.196:9101 of job  has been using over 90% cpu for more than 2 minutes.",
    "summary": "Instance 10.247.14.196:9101 using too much cpu" 
  },
  "startsAt": "2016-09-21T10:14:07.805Z",
  "endsAt": "2016-09-21T10:14:22.805Z"
}]

[ivankovnatsky]

@asbjornenge it's 0.4.2. let me share my slack notifications block:

receivers:
- name: 'slack-notifications'
  slack_configs:
  - channel: '#1'
    send_resolved: true
    title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonAnnotations.summary }}'
    text:  '{{ .CommonAnnotations.description }}'

try to trigger alert after alertmanager restart.

This is basically status of the alert, either [FIRING:1] or [RESOLVED]:

[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]

[asbjornenge]

Super weird. Tried upgrading to the latest prometheus v1.1.3 and also latest alertmanager v0.4.2. Both (old and new prom) producing the same alert struct (same as above), but both .CommonAnnotations.description and .CommonAnnotations.summary produce empty strings in slack :confused:

If I set anything else in text or title, it works as expected.

[ivankovnatsky]

let's see what your alert.rules looks like.

[asbjornenge]

Aaaalright, got it working! Don't know what I did exactly (not very helpful, sorry), but might have been that the new containers I thought I was running was the old containers :stuck_out_tongue_closed_eyes: So my best tip, use the latest version of both :wink: :tada:

[Vannevelj]

For completeness' sake, this was my scenario and solution:

ALERT InstanceDown
IF up == 0
FOR 30s
ANNOTATIONS {
    summary = "Server {{ $labels.Server }} is down.",
    description = "{{ $labels.Server }} ({{ $labels.job }}) is down for more than 30 seconds."
}

receivers:
- name: general_receiver
  slack_configs:
  - api_url: https://hooks.slack.com/services/token
    channel: "#telemetry"
    title: "My summary"
    text: "My description"

{{ define "__slack_text" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.description}}{{ end }}                                                                                
{{ end }}

{{ define "__slack_title" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.summary}}{{ end }}                                                                                
{{ end }}

{{ define "slack.default.text" }}{{ template "__slack_text" . }}{{ end }}
{{ define "slack.default.title" }}{{ template "__slack_title" . }}{{ end }}

[jazzerjazzer]

If you are using confd, then you need to escape the curly braces:
Ex:
name: 'slack_chatbots'slack_configs: - send_resolved: true api_url: '<url of slack web hook>' channel: '#test' text: 'https://internal.myorg.net/wiki/alerts/{{"{{"}} .GroupLabels.alertname {{"}}"}}'

[mikkiweesenaar]

Good to know/realize is that the xyz-part of .Annotations.xyz is case sensitive :)

[wrossmann]

You can skip the template creation in @Vannevelj's suggestion with something like:

text: "{{ range .Alerts }}**[{{ .Status }}]** {{ .Annotations.summary }}\n{{ .Annotations.description }}\n\n{{ end }}"

Which will get you something like:

[FIRING:2] filesystem_threshold_exceeded (ext4 node)
[firing] server1.foo.bar:9100's filesystem usage is dangerously high
/data only has 0% free.

[resolved] server2.foo.bar:9100's filesystem usage is dangerously high
/data only has 3.247% free.

[firing] server3.foo.bar:9100's filesystem usage is dangerously high
/data only has 6.901% free.

More useful info found in my Googlings:

• Default template vars
• Alert struct definition

[vinaymorkar]

@Vannevelj, @asbjornenge, sorry guys, I was thinking you would go for comments, anyway:

I did like this, nothing fancy, for now it works for me:

title: '{{ .CommonAnnotations.summary }}'
text: '{{ .CommonAnnotations.description }}'

that picks my summary and description in prometheus' alert.rules

This isn't working for me

[vinaymorkar]

I am getting an empty summary and description, even it is present in the rules file.

• alert: awesome_down5
  expr: absent(container_memory_usage_bytes{name="my_awesome_container5"})
  for: 5s
  labels:
  severity: moderate
  annotations:
  summary: "Hello, your my_awesome_container5 is down"
  description: "lorem ipsum.. ."

- name: 'slack'
slack_configs:
- channel: '#server'
text: "description: {{ .CommonAnnotations }} {{ .CommonAnnotations.description }}\nsummary: {{ .CommonAnnotations.summary }}"
api_url: 'https://hooks.slack.com/services/*/*/*'
send_resolved: True