Alertmanager: display some metrics value in alert email.

[changsijay]

I have a rule about disk almost full as below which will send email when available disk size is less than 10%

alert DiskLow
  if node_filesystem_avail{instance="10.60.90.147:9100", mountpoint="/"}/node_filesystem_size{instance="10.60.90.147:9100", mountpoint="/"} < 0.1
  for 5m
  labels { severity = "email" }
  annotations {
    summary = "Node {{ $labels.instance }} disk low, free space < 10%",
    description = "{{ $labels.instance }} disk is almost full!!",
  }

Would it possible to display the size information in the email? say: current disk usage is 93G/100G

[brancz]

You can render the current value with {{ $value }}, however that is the remaining disk space in this case, so you might need some tweaking of your statement, but what you are trying to achieve is definitely already possible.

[changsijay]

Hi @brancz , sorry I am not very clear about how to do that.
In the doc

ALERT <alert name>
  IF <expression>
  [ FOR <duration> ]
  [ LABELS <label set> ]
  [ ANNOTATIONS <label set> ]

the {{ value }} is the labels: value pair in the if expression.
because our customers request to alert when the disk usage percentage > 90%.
I am not sure how I can get the value of

node_filesystem_avail{instance="10.60.90.147:9100", mountpoint="/"}

and

node_filesystem_size{instance="10.60.90.147:9100", mountpoint="/"}

separately in the rule.

[matthiasr]

You can use the query template function in your annotations. You'll have to try it out, but something like

foobar = '{{ printf `node_filesystem_avail{instance="%s", mountpoint="/"}` $labels.instance | query | first }} / {{ printf `node_filesystem_size{instance="%s", mountpoint="/"}` $labels.instance | query | first }}'

should do what you want.

[changsijay]

Thanks a lot, I don't know about query template function before. going to study now 👯

[cmcginty]

@matthiasr I'm new to prometheus. Is there a complete example of a template query and usage for reference?

[andrewhowdencom]

@cmcginty Generally speaking usability questions are redirected to the mailing list:

https://groups.google.com/forum/#!forum/prometheus-users

You'll find a bunch of help there. Beyond that, the most complete documentation I've found is here:

https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/

But the mailing list will likely have more.

[inkdude]

@matthiasr if i use $labels.mountpoint

how i should write query?
'{{ printf node_filesystem_avail{mountpoint="$labels.mountpoint"} $labels.instance | query | first }}

or how?