Alertmanager: [victorops] make message_type template-able

[caarlos0]

Sorry, I don't know how to express exactly what I meant in words, so, here is code.

We have several teams here, and alerts are routed to the responsible team. For example:

    - match_re:
        service: ^(foo|bar).*
      receiver: team1-info
      routes:
        - match:
             severity: page
           receiver: team1-page

Later, we define the receivers as:

- name: 'team1-page'
  victorops_configs:
  - routing_key: team1
    message_type: CRITICAL

- name: 'team1-info'
  victorops_configs:
  - routing_key: team1
    message_type: WARNING

The problem with this is that we need to replicate a lot of code (for example, matching services, instances, etc), which is not a breaker but it makes the config more difficult to read because of the repetition.

I wonder if there is a way of doing something like:

- name: team1
  victorops_configs:
  - routing_key: team1
    message_type: {{ $labels.severity }}

If that was possible, it would be easier to maintain this configs.

Any tips?

[stuartnelson3]

This is a usage question, which in the future should be directed to [email protected]

(False description removed)
The config file is read as a complete file, with no interpolation happening when parsing. We recommend using templating as provided by your config management software of choice (chef, ansible, etc.) to turn an alertmanager config template into an alertmanager config on your node.

[stuartnelson3]

If you look at the documentation, you can see how to accomplish this: https://prometheus.io/docs/alerting/notification_examples/

- name: team1
  victorops_configs:
  - routing_key: team1
    message_type: {{ .Labels.severity }}

This might do what you want.

[jperville]

Hello, having the same issue here with the following config:

    - name: 'victorops'
      victorops_configs:
        - api_key: 'XXXXXXXXXXXXXXXXXXXX'
          routing_key: 'myenvironment'
          message_type: '{{ .Labels.severity | toUpper }}'  # <-------
          state_message: 'Alert: {{ .CommonLabels.alertname }}. Summary:{{ .CommonAnnotations.summary }}. RawData: {{ .CommonLabels }}'

On VictorOps, all alerts have "message_type = CRITICAL", as if the value of the expanded message_type was not one of INFO, WARNING or CRITICAL and the default value (CRITICAL) was used.

[stuartnelson3]

Please move your usage question to [email protected]

[caarlos0]

@stuartnelson3 what @jperville seems to be a bug... doesn't it?

[caarlos0]

Maybe we could even always set state_message to upper, as all valid options in the victorops side are uppercase as well

[stuartnelson3]

https://github.com/prometheus/alertmanager/blob/master/notify/impl.go#L862-L868

It looks like the code doesn't support templating the message_type key. It also sets a firing alert to CRITICAL if it's not set to one of the accepted variables. So, what's happening is your template string isn't being used, and then it's being set to CRITICAL because what you have doesn't match an accepted message type.

I'm guessing that this was done to make sure alertmanager couldn't send an invalid message.

If you want, you can replace the code on line 862 to template out the message_type:

messageType  = tmpl(n.conf.MessageType)

The check below should remain to make sure the templating is resulting in an acceptable value.

Adding code in this notifier to upcase all strings would also be cool, if that's required by victorops.

[jperville]

Thanks @stuartnelson3 , I will page the victorops people to see if they can fix (even if this is not their code it is in their interest that the issue gets fixed).

[caarlos0]

@jperville I can work on a patch as well

[caarlos0]

@jperville @stuartnelson3 see #1038

[jperville]

Thank you @caarlos0 for the PR. For information, my contact at VictorOps told me that the message_type field is not case-sensitive, so the toUpper in the alertmanager template is not necessary, we can just the value of the severity label (as long as that value is valid for VictorOps API).

[caarlos0]

@jperville hmm, I tested it and it defaulted to CRITICAL when I passed a lowercased warning...

[jperville]

Right, that must be because of https://github.com/prometheus/alertmanager/blob/master/notify/impl.go#L866 ; what my VictorOps contact said is that their API will ignore the case of the message_type value (I forgot about the check in the notifier which forces the value to CRITICAL if it is not in a case-sensitive whitelist). So we have to uppercase.

[jperville]

The documentation in https://prometheus.io/docs/alerting/configuration/#%3Cvictorops_config%3E should be updated to say that the values must be uppercase and not snakecase as implied by the following comment:

# Describes the behavior of the alert (Critical, Acknowledgement, Info, Recovery).
[ message_type: <string> ]

I will prepare a PR immediately.

[caarlos0]

@jperville oh, totally missed that.

I had seen the docs and forgot about updating them. Thanks for the help here 🚀

[jperville]

@caarlos0 here we go: https://github.com/prometheus/docs/pull/863

[jperville]

Note that the official documentation of the VictorOps REST API does not document the WARNING value for the message_type attribute. I will tell them to update their documentation.