Alertmanager: Allow templating in the webhook URL

[prasenforu]

Alertmanager version: 0.5.1

My alert configure as follows ...

ALERT PoDHighCPUUsage
  IF sum(rate(container_cpu_usage_seconds_total{container_label_io_kubernetes_pod_namespace!=""}[2m])) BY (kubernetes_pod_name, kubernetes_io_hostname, container_label_io_kubernetes_pod_namespace) >= 0.2
  FOR 2m
  LABELS {severity="criticalup"}
  ANNOTATIONS {summary="CPU usage of POD {{ $labels.kubernetes_pod_name }} {{ $labels.container_label_io_kubernetes_pod_namespace }} utilization is  >= 0.4 cores (current value: {{ $value }}%)"}

webhooks setup as follows ..

- name: 'email-n-webhookup'
  email_configs:
  - to: 'mail-id'
    send_resolved: true

  webhook_configs:
  - url: http://<webhook_ip>:9000/hooks/ose-webhook?in1={{.Labels.container_label_io_kubernetes_pod_namespace}}&in2={{.Labels.kubernetes_pod_name}}&in3=plus

- name: 'email-n-webhookdn'
  email_configs:
  - to: 'mail-id'
    send_resolved: true

  webhook_configs:
  - url: http://<webhook_ip>:9000/hooks/ose-webhook?in1=tomi&in2=chat&in3=minus

In webhook server logs I am getting data as follows.

[webhook] 2017/04/03 10:03:17 2017-04-03T10:03:17Z | 200 |       194.078µs | 172.30.200.104:9000 | POST /hooks/ose-webhook
[webhook] 2017/04/03 10:03:17 executing /etc/webhook/ose.sh (/etc/webhook/ose.sh) with arguments ["/etc/webhook/ose.sh" "{{.labels.container_label_io_kubernetes_pod_namespace}}" "{{$labels.kubernetes_pod_name}}" "plus"] and environment [] using /etc/webhook as cwd

not as a data, its coming as a complete variable.

So I am not able to pass the value/data "labels.kubernetes_pod_name" & "labels.container_label_io_kubernetes_pod_namespace" in webhook_configs "email-n-webhookup"

Statically it worked as per "email-n-webhookdn"

I think its supported for Email as per https://www.robustperception.io/using-labels-to-direct-email-notifications/

not sure if I can use it in webhook as well ......

Please help ....

[juliusv]

The webhook URL is not a templateable field. Instead the webhook notifier sends all metadata in the request body, as outlined in https://prometheus.io/docs/alerting/configuration/#.

You can grab the labels from there in your webhook implementation.

Closing, as this is hopefully answered. If not, please ask questions on the users mailing list instead of opening an issue: https://prometheus.io/community/

[prasenforu]

Thanks.

Don't you think its a enhancement.

[juliusv]

I can reopen this as a feature request, but it started out as a question.

[brian-brazil]

This wouldn't make sense feature wise. The webhook is meant to work in exactly one way, with the other end specifically designed to accept what we send.

The other end already has all the information it needs.

[juliusv]

That's what I thought too. Confirmed then...

[prasenforu]

Like other config where url used that all are customised template.

Why not webhook? and can't be a static all time.

Let it be open for further discussion..

[juliusv]

@prasenforu The reasoning here goes:

1. None of the other notification mechanisms have a templateable URL field.
2. You get all the dynamic information that you need, just not in the URL of the request, but in the request body.
3. Two Prometheus core developers have agreed here that we don't think adding this is a good idea, so I think we'll keep this closed.

[prasenforu]

OK.
Thanks.

Let me explain why ?

I am using prometheus in containers platform (kubernetes) where things are changing very dynamically.

I do agree your point.

Can you please suggest any alternative solution where I can pass labels in that url.

Otherwise I have to write for each conrainer separate webhook which is not a good idea.

Basically I want to make global.

Anyway thanks for your continuous support.

[brian-brazil]

Can you please suggest any alternative solution where I can pass labels in that url.

Uhm, write a webhook that does this and sends out the requests in the format you need.

[prasenforu]

Finally able to setup webhook without templating webhook url.

Thanks all to rethink on this.

Bydefault webhook resolve option is true.

Is that mean when it resolve it will fire same webhook.

Is that understanding correct?

[juliusv]

Yes.

On Fri, Apr 7, 2017 at 3:34 PM, prasenforu [e-mail] wrote:

Finally able to setup webhook without templating webhook url.

Thanks all to rethink on this.

Bydefault webhook resolve option is true.

Is that mean when it resolve it will fire same webhook.

Is that understanding correct?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/alertmanager/issues/684#issuecomment-292537589,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAg1mIKX77ECyF5wnB96WNC1xACQxkpqks5rtjtzgaJpZM4MxXAt
.

[rajkiran616]

I have small question. Lets say we need to call API endpoint based on alert. If alert is about a particular docker service. How can we pass the service name as part of webhook url?

I am able to call the endpoint with hardcoded service name. http://url/api/coreservice/scaleup/. And configure the url in alertmanager.yml.

webhook_configs:

• url: http://url/api/{{servicename}}/scaleup/

How can we get he service name?

Can we use {{.Labels.container_label_com_docker_swarm_service_name}} ?

[brian-brazil]

@rajkiran616 Please don't put support requests on closed issues. The Prometheus users list is the best place for such questions.

[nehalraf]

@brian-brazil If we allow templating in url we can use opegenie to track heatbeats (via DeadmanSwitch) in case of alerts coming from multiple clusters. Ops genie will create seperate heartbeat for each cluster.

route:
routes:
- receiver: 'opsgenie-heartbeat'
match:
alertname: DeadMansSwitch

receivers:

• name: 'opsgenie-heartbeat'
  webhook_configs:
  
  
  
  • url: 'https://api.opsgenie.com/v2/heartbeats/{{ .GroupLabels.ClusterID }}/ping?apiKey=*'
    
    send_resolved: true
  
  

[sevagh]

We allow users to specify their slack, email, and pagerduty from the alert file itself:

$ cat example_alert.yml
groups:
- name: example
  rules:
  - alert: ExampleAlert
    expr: up{job="myjob"} == 0
    for: 5m
    labels:
      slack: '#my-slack-channel'
      email: '@my-email'
      pagerduty: 'my-pagerduty-service-id'
    annotations:
      description: 'foo'
      summary: 'foo'

These get expanded in the alertmanager config, e.g.

- name: generic_email
  email_configs:
  - to: "{{ .GroupLabels.email }}"

That way, team-specific configs stay out of the alertmanager config. I prefer to have teams put their own configs in their own files, and I (as a sysadmin/ops/whatever) maintain the alertmanager config.

The one exception I've run into so far is that webhooks aren't templateable, so I need to allow people to write to the alertmanager config.

This describes a usecase where I'd be happy if webhooks became templateable.

[abhishekjiitr]

@brian-brazil If we allow templating in url we can use opegenie to track heatbeats (via DeadmanSwitch) in case of alerts coming from multiple clusters. Ops genie will create seperate heartbeat for each cluster.

route:
routes:

• receiver: 'opsgenie-heartbeat'
  match:
  alertname: DeadMansSwitch

receivers:

* name: 'opsgenie-heartbeat'
  webhook_configs:

  * url: '[https://api.opsgenie.com/v2/heartbeats/{{](https://api.opsgenie.com/v2/heartbeats/%7B%7B) .GroupLabels.ClusterID }}/ping?apiKey=*******'
    send_resolved: true

+1, need templating in webhook URL for OpsGenie HeartBeat

[jangrewe]

+1, this would be the only way (i can think of) to make prometheus-msteams usable with MS Teams stupid decision to require a separate webhook for every. single. channel., by using a K8s namespace <-> webhook mapping config for it.

[Yzzzi]

I have another example where templatable urls can be usefull:
Im using alertmanager to send notifications about my jenkins slaves (ex. disk space usege). And i wanted to implement job triggering on alert to automaticly clear slaves disk. Templatable url would be helpful here, because i can create job to trigger with parameters in Jenkins, and use slave hostname as a parameter. Jenkins require parameter to be in a url.

[freeseacher]

@nehalraf Have you found a solution for templating opsgenie's heartbeats?