Alertmanager: Can't send API call to OpsGenie

[xmemy]

Hello,

What did you do?
Trying to get alertmanager sending a call to Opsgenie's API

What did you expect to see?
A call/alert into OpsGenie or at least an entry into Opsgenie's logs panel

What did you see instead? Under which circumstances?

Oct 02 10:39:16 Server1 alertmanager[32424]: level=error ts=2018-10-02T10:39:16.264984004Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="Post default/v2/alerts/cc9772a1ec901d26ef71dae861793e4d08979fa5030a3f0a5f3ba549f1f2d0b6/close?identifierType=alias: unsupported protocol scheme \"\""

• Alertmanager version:

alertmanager, version 0.15.2 (branch: HEAD, revision: d19fae3bae451940b8470abb680cfdd59bfa7cfa)
  build user:       root@3101e5b68a55
  build date:       20180814-10:53:39
  go version:       go1.10.3

• Prometheus version:

prometheus, version 2.4.2 (branch: HEAD, revision: c305ffaa092e94e9d2dbbddf8226c4813b1190a0)
  build user:       root@dcde2b74c858
  build date:       20180921-07:22:29
  go version:       go1.10.3

• Alertmanager configuration file:

global:
  #opsgenie_api_key: 'API KEY provided by OpsGenie's integration'
  #opsgenie_api_url: https://api.opsgenie.com/
templates:
- '/etc/alertmanager/template/*.tmpl'

route:
  repeat_interval: 1h
  receiver: saas-ops

receivers:
- name: 'saas-ops'
  opsgenie_configs:
  - api_key: 'API KEY provided by OpsGenie's integration'
    teams: 'dummy'

• Logs:

Oct 02 10:39:16 Server1 alertmanager[32424]: level=error ts=2018-10-02T10:39:16.264984004Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="Post default/v2/alerts/cc9772a1ec901d26ef71dae861793e4d08979fa5030a3f0a5f3ba549f1f2d0b6/close?identifierType=alias: unsupported protocol scheme \"\""

Of course before posting here, I tried to fix this but without luck.

Regards,

V.

[stuartnelson3]

@mxinden @simonpasquier do either of you have access to hipchat to test this? At first blush it would appear to me to be a misconfiguration based on the url error being logged.

https://github.com/prometheus/alertmanager/blob/master/notify/impl.go#L1164-L1169

[simonpasquier]

No I don't have an access to the OpsGenie API. @xmemy can you paste the full configuration from the AlertManager's Status page?

[xmemy]

as requested :
````
global:
resolve_timeout: 5m
http_config: {}
smtp_hello: localhost
smtp_require_tls: true
pagerduty_url: https://events.pagerduty.com/v2/enqueue
hipchat_api_url: https://api.hipchat.com/
opsgenie_api_url: https://api.opsgenie.com/
opsgenie_api_key:
wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
route:
receiver: saas-ops
repeat_interval: 1h
receivers:

• name: saas-ops
  opsgenie_configs:
  
  
  
  • send_resolved: true
    
    http_config: {}
    
    api_key:
    
    api_url: default/
    
    message: test
    
    description: '{{ template "opsgenie.default.description" . }}'
    
    source: '{{ template "opsgenie.default.source" . }}'
    
    templates:
  
  
• /etc/alertmanager/template/*.tmpl
  ````

[simonpasquier]

api_url is supposed to be an absolute URL (eg https://<host>/<path>), not a relative one.

https://prometheus.io/docs/alerting/configuration/#%3Copsgenie_config%3E

[stuartnelson3]

@simonpasquier beat me to it. Leave api_url blank to use the global value.

[simonpasquier]

That being said, the configuration should have fail on startup.

[xmemy]

Even if I specify the url it's not working as expected (same error message)

````
global:
#opsgenie_api_key: 'xxxx'
opsgenie_api_url: https://api.opsgenie.com/
templates:

• '/etc/alertmanager/template/*.tmpl'

route:
repeat_interval: 1h
receiver: saas-ops

receivers:

• name: 'saas-ops'
  opsgenie_configs:
  
  
  
  • api_key: 'xxxx'
    
    api_url: 'https://api.opsgenie.com/'
    
    teams: 'dummy'
    
    ````
  
  

Oct 02 13:14:16 Server1 alertmanager[32424]: level=error ts=2018-10-02T13:14:16.279963821Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="Post default/v2/alerts/cc9772a1ec901d26ef71dae861793e4d08979fa5030a3f0a5f3ba549f1f2d0b6/close?identifierType=alias: unsupported protocol scheme \"\"" Oct 02 13:14:16 Server1 alertmanager[32424]: level=error ts=2018-10-02T13:14:16.280881825Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="Post default/v2/alerts/cc9772a1ec901d26ef71dae861793e4d08979fa5030a3f0a5f3ba549f1f2d0b6/close?identifierType=alias: unsupported protocol scheme \"\""

[simonpasquier]

@xmemy can you double the configuration in the Status page?

[xmemy]

````
global:
resolve_timeout: 5m
http_config: {}
smtp_hello: localhost
smtp_require_tls: true
pagerduty_url: https://events.pagerduty.com/v2/enqueue
hipchat_api_url: https://api.hipchat.com/
opsgenie_api_url: https://api.opsgenie.com/
opsgenie_api_key:
wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
route:
receiver: saas-ops
repeat_interval: 1h
receivers:

• name: saas-ops
  opsgenie_configs:
  
  
  
  • send_resolved: true
    
    http_config: {}
    
    api_key:
    
    api_url: default/
    
    message: test
    
    description: '{{ template "opsgenie.default.description" . }}'
    
    source: '{{ template "opsgenie.default.source" . }}'
    
    templates:
  
  
• /etc/alertmanager/template/*.tmpl
  ````

Sounds identical which is weird

[xmemy]

I managed to fix this,
````
global:
opsgenie_api_key: "xxxxxxx"
opsgenie_api_url: "https://api.eu.opsgenie.com/"
route:
group_interval: 5m
repeat_interval: 1h
receiver: saas-ops

receivers:

• name: 'saas-ops'
  opsgenie_configs:
  
  
  
  • teams: 'dummy'
    
    description: 'test'
    
    message: 'test'
    
    ````
  
  

As summary, necessary to provide a description/message and use the proper url depending of the region.

[fgrzadkowski]

Actually it's just enough to provide the proper URL depending on the region. It works without description and message.