Alertmanager: AlertManager - .GeneratorURL doesn't expand correctly more complex (only?) alert's conditions

[it-praktyk]

I've found that .GeneratorURL doesn't expand correctly more complex (only?) alert's conditions

A part of template - the file https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl#L130-L138

<tr>
    <td class="content-block">
    <strong>Labels</strong><br />
    {{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}<br />{{ end }}
    {{ if gt (len .Annotations) 0 }}<strong>Annotations</strong><br />{{ end }}
    {{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}<br />{{ end }}
    <a href="{{ .GeneratorURL }}">Source</a><br />
    </td>
</tr>

for the alert definition

ALERT storage_90_percent
IF round((node_filesystem_free{job="nodes",mountpoint!="/var/lib/nfs/rpc_pipefs"} / node_filesystem_size{job="nodes"}) * 100, 1) <= 10 and on (device) node_filesystem_readonly != 1
FOR 10m
LABELS { severity = "critical", category = "storage"  }
ANNOTATIONS {
  summary= 'Free space on a storage is less than 10%',
  description= 'Free space on a storage is less 10%'
}

is expanded for

<tr style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">
    <td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; margin: 0; padding: 0 0 20px;" valign="top">
    <strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels</strong><br style="font-family: 'Helvetica Neue', H
elvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
    alertname = storage_90_percent<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />category = storage<br style="f
ont-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />device = euhpc-nas02.euhpc.arm.com:/ifs/arm/ref<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />fstype = nfs<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />instance = bc-d6-2-8.eu.iaas.arm.com:9100<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />job = nodes<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />monitor = prometheus-server<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />mountpoint = /arm/ref<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />severity = critical<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
    <strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
    description = Free space on a storage is less 10%<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />summary = Free space on a storage is less than 10%<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
    <a href="http://HOST_NAME:PORT_NUMBER/graph?g0.expr=round%28%28node_filesystem_free%7Bjob%3D%22nodes%22%2Cmountpoint%21%3D%22%2Fvar%2Flib%2Fnfs%2Frpc_pipefs%22%7D&#43;%2F&#43;node_filesystem_size%7Bjob%3D%22nodes%22%7D%29&#43;%2A&#43;100%2C&#43;1%29&#43;%3C%3D&#43;10&#43;and&#43;ON%28device%29&#43;node_filesystem_readonly&#43;%21%3D&#43;1&amp;g0.tab=0" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #348eda; text-decoration: underline; margin: 0;">Source</a><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />
    </td>
</tr>

It was previously posted at Prometheus Developers Google Group.

[brian-brazil]

What exactly do you believe is a bug here?

[mxinden]

@it-praktyk Can you elaborate further on your bug description?

When I decode:

http://HOST_NAME:PORT_NUMBER/graph?g0.expr=round%28%28node_filesystem_free%7Bjob%3D%22nodes%22%2Cmountpoint%21%3D%22%2Fvar%2Flib%2Fnfs%2Frpc_pipefs%22%7D+%2F+node_filesystem_size%7Bjob%3D%22nodes%22%7D%29+%2A+100%2C+1%29+%3C%3D+10+and+ON%28device%29+node_filesystem_readonly+%21%3D+1&g0.tab=0

I get the following query:

round(
  (
    node_filesystem_free{
      job="nodes",mountpoint!="/var/lib/nfs/rpc_pipefs"
    }
    +/+
    node_filesystem_size{
      job="nodes"
    }
  )+*+100,+1
)+
<=+
10+

and

+ON(
  device
)+node_filesystem_readonly+!=+1

[it-praktyk]

I expect that encoded URLs should be clickable means I can click and open a web page with a query results in a browser.

Due that encoded string contains hash(es) browser didn't open link correctly. Please read e.g. https://www.oho.com/blog/explained-60-seconds-hash-symbols-urls-and-seo

Correctly opened url has form (a query by analogy, not exactly the same):

"http://\+100%2C+1)+%3E+10)+and+(round((node_filesystem_free%7Bjob%3D%22nodes%22%2Cmountpoint!%3D%22%2Fvar%2Flib%2Fnfs%2Frpc_pipefs%22%7D+%2F+node_filesystem_size%7Bjob%3D%22nodes%22%7D+and+ON(device)+node_filesystem_readonly+!%3D+1)++100%2C+1)+%3C+25)&g0.tab=0"

I've receive correct link by click like a green arrow shows.

[base698]

My .GeneratorURL goes to a pod name, how do you change the name to it's domain name? Is this a prometheus config option?

[base698]

web.external-url in case anyone is looking for it :)

[simonpasquier]

I failed to reproduce the issue with the most recent versions of Prometheus and Alertmanager. I'm closing the issue, feel free to reopen if needed.