Alertmanager: HTML URL not loading properly in Alertmanager configuration from other template file.

[shubhamc183]

What did you do?
I am configuring alertmanager config to choose slack webhook depending upon some dynamic logic. But I cannot see that value getting placed in slack_api_url

What did you expect to see?
The normal load of Alertmanager

What did you see instead? Under which circumstances?
Errors

Environment
Ubuntu

• System information:
  Linux 5.3.0-40-generic x86_64

• Alertmanager version:
  alertmanager:v0.20.0

• Prometheus version:
  2.15.2

• Alertmanager configuration file:

global:
  resolve_timeout: 5m
  slack_api_url: '{{ template "slack.myorg.webhook" . }}'
route:
  group_by: ['alertname']
  ...
  ...
  ...
templates: ['/etc/alertmanager/config/*.tmpl']```

• slack-myorg-template.tmpl

# To identify from which cluster the alerts are comming
{{ define "slack.myorg.username" }}Minikube{{ end }}

# select webhook
{{ define "slack.myorg.webhook" }}https://hooks.slack.com/services/XXX/XX/XX{{ end }}

Here I'll write some if else condition to choose the channel accordingly and similarly, I have some logic to get slack channel also.

• Logs:

level=error ts=2020-03-12T10:41:06.004Z caller=coordinator.go:124 component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config/alertmanager.yaml err="unsupported scheme \"\" for URL"

The only issue if slack.myorg.webhook is not perfectly loading into slack_api_url by calling '{{ template "slack.myorg.webhook" . }}'.

• Other things I tried

1. Just mentioning {{ define "slack.myorg.webhook" }}https://hooks.slack.com/services/XXX/XX/XX{{ end }} doesn't cause any issue.
2. If I do username: '{{ template "slack.myorg.webhook" . }}' under slack configuration it's not throwing error.

[simonpasquier]

slack_api_url (as well as api_url from slack_config) don't support templating.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.

[dsmith73]

Since this requires us to put an API key in plain text, shouldn't it allow templating?

[FeargusOG]

slack_api_url (as well as api_url from slack_config) don't support templating.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.

Is there a particular reason why the api_url from slack_config can't be templated? Or has it not been implemented yet. This would be a useful feature imo.

[simonpasquier]

@FeargusOG see https://prometheus.io/docs/operating/security/#alertmanager
TL;DR: we don't want users to be able to configure secrets using templates because it could cause a security breach.

[FeargusOG]

@simonpasquier Thanks for getting back to me. Some secrets are templatable though, correct? Those that use the <tmpl_secret>? That doc you linked mentions Any secret fields which are templatable are intended for routing notifications. It seems odd to allow some secrets to be templatable but not others if that is the case.

Also, would api_url not be an example of intended for routing notifications?

[simonpasquier]

To be clear, only PagerDuty's routing_key and service_key support templating. I'm not that familiar with PagerDuty but IIUC different routing_key/service_key map to different monitored entities within your organization.
As for Slack, I'd expect that you'd use different channels (the channel field in Alertmanager supports templates) rather than different Slack instances.
@brian-brazil can probably articulate better.

[brian-brazil]

The templatable secret fields are templatable in spite of being secrets, as those particular fields can be used for routing.

Templating the slack URL would seem odd to me, it'd be like templating the smtp smarthost.

[FeargusOG]

The slack URL I am referring to is an Incoming WebHook (documented here) meaning that the api_url I am setting is specific to a single user, and a single channel, not a different slack instance. This is why I was hoping to template it.

[brian-brazil]

There may be a valid case for that, however I'd recommend setting up the type of webhook that isn't tied to just one channel as that makes all of this a lot easier to do.

[Khemdevi]

@FeargusOG I am trying to achieve the same, using a template for different slack channels, the code is below, it is possible to configure this way?

receivers: - name: "receiver_slack" slack_configs: - api_url: Imcming_webhok_slack_url channel: "#test" send_resolved: true text: '{{ template "slack.myorg.text" . }}' username: Prometheus