Alertmanager: wechat invalid APISecret for CorpID

[zong400]

What did you do?
send alert to wechat.

alerts1='[
{
"labels": {
"alertname": "DiskRunningFull",
"dev": "sdb2",
"instance": "example2"
},
"annotations": {
"info": "The disk sdb2 is running full",
"summary": "please check the instance example2"
}
}]'

curl -XPOST -d"$alerts1" http://10.172.138.114:9093/api/v1/alerts

What did you expect to see?
wechat show me the alert

What did you see instead? Under which circumstances?
nothing

Environment
prom/alertmanager:v0.14.0

• System information:
  Linux 4.4.0-74-generic x86_64

• Alertmanager version:
  prom/alertmanager:v0.14.0

• Prometheus version:
  prom/prometheus:v2.2.1

• Alertmanager configuration file:
  global:
  # The smarthost and SMTP sender used for mail notifications.
  resolve_timeout: 6m
  wechat_api_url: "https://qyapi.weixin.qq.com/cgi-bin/"
  # The directory from which notification templates are read.
  templates:

  • 'templates/*.tmpl'

  The root route on which each incoming alert enters.

  route:
  group_by: ['alertname']
  group_wait: 3s
  group_interval: 5m
  repeat_interval: 1h
  # A default receiver
  receiver: itesttech

  receivers:

  • name: "itesttech"
    wechat_configs:
    
    
    
    • send_resolved: true
      
      to_user: ""
      
      to_party: "4"
      
      to_tag: ""
      
      agent_id: "1000004"
      
      corp_id: "wx3xxxxxxe57c654c"
      
      api_secret: "oKxV3t9V8DxxxxxxxxxxrqdwDj9AX9mQfA"
    
    

• Logs:
  level=error ts=2018-05-17T03:10:10.324489751Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"wechat\" due to unrecoverable error: invalid APISecret for CorpID: wx38bxxxxxx7c654c"
  level=error ts=2018-05-17T03:15:10.257264553Z caller=notify.go:303 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"wechat\" due to unrecoverable error: invalid APISecret for CorpID: wx38bxxxxxx7c654c"
  level=error ts=2018-05-17T03:15:10.257449928Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"wechat\" due to unrecoverable error: invalid APISecret for CorpID: wx38bxxxxxx7c654c"

[simonpasquier]

It looks like the API secret doesn't match with the CorpID field.

[zong400]

impossible,
i tried with python.

[dingxiaoyue]

hi @zong400
I met the same problem. How did you solve it?
thanks.

[simonpasquier]

@zong400 can you reproduce the problem with AlertManager v0.15.1?

[wenfei3]

I reproduced with v0.15.1.

level=error ts=2018-08-14T13:33:48.129983854Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"wechat\" due to unrecoverable error: invalid APISecret for CorpID: ww728xxxxxxx9a1"
level=error ts=2018-08-14T13:33:48.130090838Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=24 err="cancelling notify retry for \"wechat\" due to unrecoverable error: invalid APISecret for CorpID: ww728xxxxxxxdd9a1"

there is the config

global:
  wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin'
  wechat_api_secret: i7ixxxxxxxxxxxxxxxxxxxxxxxxxxM
  resolve_timeout: 1m

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 30m
  receiver: wechat_receiver

receivers:
- name: wechat_receiver
  wechat_configs:
   - to_user: '@all'
     agent_id: '1000002'
     send_resolved: true
     corp_id: ww728xxxxxxxxxxxxxxxx1
     api_secret: i7ixxxxxxxxxxxxxxxxxxxxxxxxxxxxcM

and I could get access_token with the api

https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=ww728xxxxxxx9a1&corpsecret=i7ixxxxxxxxxxxxxxxxxxM

the response is

{
    "errcode": 0,
    "errmsg": "ok",
    "access_token": "5wbH37WTgeAstK8rhCylwrfalnT26wyw-LqD7H4T46m5jcOOXGi3whmcLKdqK2kU-xW8VWeF4AE4hF-wnbrMz8i4LEvyeNl2gBzLUICn2CqtQz1Wa5P4p1Zf6WUV-tnJWCFVZdLE91Z-WQdz0lS-jHussSEx8rk2rXZrFAvgOlpZXmoAt2x0MayRwNsK0dWx-0oRLvfhuZTwWiuW2yoigw",
    "expires_in": 7200
}

[simonpasquier]

@wenfei3 can you try commenting out the wechat_api_url line in the global section? The default value is https://qyapi.weixin.qq.com/cgi-bin/ which has a trailing "/" character compared to your setting?

You would also need to upgrade to v0.15.2 because of #1515 which also affects Wechat.

[wenfei3]

@simonpasquier
OK,I hava read the #1515 ,thanks for your suggestion.I will try to do it this afternoon.

[simonpasquier]

@wenfei3 any update? can we close this issue?

[eason0420]

I met the same problem.and I could get access_token with the api，but wechat not receive alert message，also have no error log. How did you solve it? thanks。

the email is work。

version：0.15.2 centos7

global:
resolve_timeout: 3m
wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/'
wechat_api_secret: '9Jvxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx20'

templates:

• '/server/alert/alertmanager/template/*.tmpl'

route:
group_by: ['prod_db_alert']
group_wait: 10s
group_interval: 10s
repeat_interval: 2m
receiver: 'dba'

receivers:

• name: 'dba'
  email_configs:
  
  
  
  • to: '[email protected]'
    
    html: '{{ template "mysql_email.html" . }}'
    
    headers: { Subject: "[WARN] MySQL报警邮件"}
    
    wechat_configs:
  
  
  • send_resolved: true
    
    to_party: '1'
    
    agent_id: '1000002'
    
    corp_id: 'wwexxxxxxxxxxxxb'
    
    api_secret: '9Jvxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx20'
    
    message: '{{ template "mysql_wechat.html" . }}'
  
  

inhibit_rules:

• source_match:
  severity: 'critical'
  target_match:
  severity: 'warning'
  equal: ['prod_db_alert', 'dev', 'instance']

[simonpasquier]

@eason0420 try adding a '/' to the wechat_api_url parameter.

[eason0420]

@simonpasquier on where？ like this wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/'

[simonpasquier]

@eason0420 yes

[eason0420]

@simonpasquier i try it，but it do not work

[simonpasquier]

@eason0420 I read again your initial message. If you've got no error in the logs and no message in WeChat then probably AlertManager doesn't have any notification to send at all. By the way, it would be more useful if to use our user mailing list rather than piggy backing on an existing issue that might not be related to your problem.

[eason0420]

@simonpasquier thanks，i will check itagain

[eason0420]

@simonpasquier can you provide a wechat alert template
i use my template，but $alert.Annotations get null value

{{ define "zhan_wechat.html" }}
{{ range $i, $alert := .Alerts.Firing }}
[报警项]:{{ index $alert.Labels "alertname" }}
[实例]:{{ index $alert.Labels "instance" }}
[报警阀值]:{{ index $alert.Annotations "value" }}
[报警内容]:{{ index $alert.Annotations "description" }}
[开始时间]:{{ $alert.StartsAt }}
{{ end }}
{{ end }}

[simonpasquier]

Sorry I don't use WeChat but what I can tell is that {{ index $alert.Annotations "value" }} returns the value of the annotation for the key value. So if it is nil, it means that the alert has no value annotation.

Does it mean that you can successfully authenticate to the WeChat API now?

[eason0420]

@simonpasquier yes，it cause by my wechat template。
now i use 【message: "告警信息-- \n[报警项]: {{ .CommonLabels.alertname }} \n[报警主机]: {{ .CommonLabels.instance }} \n[报警级别]: {{ .CommonLabels.severity }} \n[报警内容]: {{ .CommonAnnotations.description }}\n[当前值]: {{ .CommonAnnotations.value }}"】

[WOOOFEI]

Alertmanager v0.15.2

I met the same problem, and I'm sure that my secret is correct

[WOOOFEI]

I found the problem, the config works in "wechat_configs" but not in "global".

[hgkmail]

the "api_secret" config works in "wechat_configs" but not in "global"

receivers:
- name: "wechat"
  wechat_configs:
  - send_resolved: true
    to_user: "@all"
    to_party: ""
    agent_id: "xxx"
    corp_id: "xxx"
    api_secret: "xxx"

[marksugar]

Hello, friends
@hgkmail @WOOOFEI

I have encountered such an error:

level=error ts=2019-01-20T09:44:02.245621979Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="Get https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=ww0164f1XXXXXXXXX635bf0&corpsecret=llEV_QE_zw0XXXXXXXXXXoBiz_D8xYXdrT1fpGTjpcOw: context deadline exceeded"
level=error ts=2019-01-20T09:44:02.245695041Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="Get https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=ww016xxxxxxxxxx35bf0&corpsecret=llEV_QExxxxxxxxxxx5oBiz_D8xYXdrT1fpGTjpcOw: context deadline exceeded"

but

[root@master-0 alertmanager]# curl -XPOST 'https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=ww0164f1XXXXXXXXX635bf0&corpsecret=llEV_QE_zw0XXXXXXXXXXoBiz_D8xYXdrT1fpGTjpcOw'
{
"errcode":0,
"errmsg":"ok",
"access_token":"Cq39a-EIOG2fVJwrW6GJsQXZZF_H937C33fELkqCDVom8cNpLGmUL7rgIYiD-mxPetBAatcsHHb7gqeJakPT8d7SLB7dH4RhqBfLwc-3k5csIZedPwmaVsmZ4pCi58n5OstfUlVct4VSxgntGIxp9-Az9ZuZXEs_YmDJL51_lTzcSOVeyJd7oY4f-3OM3bc-HkMrHbMjo0XnfTdbErL-cg","expires_in":7200}

Not sure where the problem is

version: alertmanager:v0.15.3

[simonpasquier]

Indeed the global Wechat API secret is only used if you explicitly set the receiver's secret to "" like this:

global:
    wechat_api_secret: xxx

receivers:
- name: wechat
  wechat_configs:
    api_secret: ""

This is because the default value of the receiver's secret isn't empty:

https://github.com/prometheus/alertmanager/blob/705abf31c203c18c2e580da252be6da6f62906b2/config/notifiers.go#L103-L109

[marksugar]

@simonpasquier thank you very much

My problem may be that the k8s configuration caused the above problem. I just tested outside the k8s cluster and can send alerts normally. The above error occurs when I get to the k8s cluster. I continue to solve my problem.

[simonpasquier]

@marksugar your problem is that AlertManager can't connect to the Wechat API endpoints, it could be a firewall issue or something simillar.

[marksugar]

@simonpasquier
alertmanager runs as a pod in kubernetes and should have nothing to do with the firewall.
thanks

[simonpasquier]

Closing the issue as it was reported against an old version of AlertManager and too many people piled up on the original report which makes it hard to follow.