Alertmanager: Alertmanager continues to send messages after resolving
Alertmanager
branch unknown
buildDate 20160209-08:25:44
buildUser @30db80bfe262
goVersion 1.5.2
revision unknown
version 0.1.0-beta2
Prometheus
branch stable
buildDate 20160202-22:31:06
buildUser @9f8f0f8d724a
goVersion 1.5.3
revision 287d9b2
version 0.16.2
Alert manager config
route:
receiver: sms_team1
group_by: ['alertname', 'cluster']
group_wait: 0s
group_interval: 5s
repeat_interval: 5s
receivers:
- name: sms_team1
webhook_configs:
- url: http://sms-gate:8900/15551111/15552222/
send_resolved: true
slack_configs:
- api_url: https://hooks.slack.com/services/X000/X000/X000
channel: '#alerts'
Alert rule
ALERT TestStatsDAlert
IF irate(dispatcher_events_counter{processor="ff"}[15s]) > 1.5
SUMMARY "Test alerting"
DESCRIPTION "TEST!!!! {{$labels.instance}} processor={{$labels.processor}} dispatcher_events_counter (current value: {{$value}})"
Alerts working, but after resolving alertmanager continues send alerts. In prometheus alert tab all alerts is green but in alertmanager tab still present alert. It send alert messages with last alert value
Prometheus ui
Alertmanager UI
Webhook receiver logs
This behavior was observed in webhook and slack.
This environment work in docker compose
2tvenom
All 8 comments
How long did it continue to send the notifications for? 5 minutes by any chance?
brian-brazil
on 9 Feb 2016
Please wait, i'll check it
2tvenom
on 9 Feb 2016
Yes, 5 minutes. How turn off this?
2tvenom
on 9 Feb 2016
This is the intended behavior.
In the global configuration block you can set the resolve_timeout to somehting lower than 5 minute. But it should always be at least 3x of the longest rule evaluation interval of your Prometheus servers.
Prometheus 0.17.0 will send explicit resolved messages to Alertmanager, so you will no longer encounter this.
fabxc
on 9 Feb 2016
Got it! Thank you. And i have another question. In webhook i not receive resolved message, but in config i set field send_resolved: true
2tvenom
on 9 Feb 2016
Sorry about the late response – could you figure this out eventually. It should work as expected.
fabxc
on 29 Feb 2016
Hello, i observed very similar behavior, but in my case resolve_timeout is 5m and max evaluation time is 1m. I dont think that this is normal to receive more than 1 'resolved' message.
In my env i have 2 alertmanagers in cluster+2 prometheus which scrape the same data.
Alertmanager version: 0.5.1
Prometheus version: 1.4.1
# cat prometheus.yml
global:
scrape_interval: 15s
scrape_timeout: 15s
evaluation_interval: 1m
rule_files:
- "alerts.yml"
[...]
# cat alerts.yml
ALERT NovaContainerDown
IF time() - container_last_seen{container_name=~"nova-scheduler"} > 60
LABELS {
severity = "warning",
service = "nova",
affected_hosts = "{{ $labels.kubernetes_io_hostname }}"
}
ANNOTATIONS {
summary = "Nova Service {{ $labels.container_name }} down",
description = "{{ $labels.container_name }} is down on {{ $labels.kubernetes_io_hostname }}"
}
md5-172cfb946acd981b84ef26a96c0dc96c
# cat alertmanager.yml
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 60s
group_interval: 5m
repeat_interval: 3h
receiver: HTTP-notification
inhibit_rules:
# Mute critical when down is fired for given service
- source_match:
severity: 'down'
target_match:
severity: 'critical'
equal: ['cluster', 'service']
# Mute warning when down is fired for given service
- source_match:
severity: 'down'
target_match:
severity: 'warning'
equal: ['cluster', 'service']
# Mute warning when critical is fired for given alert
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'cluster', 'service']
receivers:
- name: 'HTTP-notification'
webhook_configs:
- url: http://localhost:8901/notify
send_resolved: true
md5-172cfb946acd981b84ef26a96c0dc96c
# cat alertmanager.log
time="2017-01-11T11:15:04Z" level=info msg="Starting alertmanager (version=0.5.1, branch=master, revision=0ea1cac51e6a620ec09d053f0484b97932b5c902)" source="main.go:101"
time="2017-01-11T11:15:04Z" level=info msg="Build context (go=go1.7.3, user=root@fb407787b8bf, date=20161125-08:14:40)" source="main.go:102"
time="2017-01-11T11:15:04Z" level=info msg="Loading configuration file" file="/opt/alertmanager/alertmanager.yml" source="main.go:195"
time="2017-01-11T11:15:04Z" level=info msg="Listening on :9093" source="main.go:250"
time="2017-01-11T11:21:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:21:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:22:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:22:37Z" level=debug msg="flushing [NovaContainerDown[18c0872][active] NovaContainerDown[dfa977e][active]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:22:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:22:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][active] NovaContainerDown[dfa977e][active]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:23:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:23:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:24:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:24:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:25:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:25:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:26:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:26:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:27:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:27:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:27:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved] NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:28:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:28:37Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:28:38Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:28:42Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:29:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:29:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:29:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:29:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:30:04Z" level=info msg="running maintenance" component=silences source="silence.go:233"
time="2017-01-11T11:30:04Z" level=info msg="running maintenance" component=nflog source="nflog.go:288"
time="2017-01-11T11:30:04Z" level=info msg="maintenance done" component=silences duration=165.34µs source="silence.go:250"
time="2017-01-11T11:30:04Z" level=info msg="maintenance done" component=nflog duration=2.196039ms source="nflog.go:305"
time="2017-01-11T11:30:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:30:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:30:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:30:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:31:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:31:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:31:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:31:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:32:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:32:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:32:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:32:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:33:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:33:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:33:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:33:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:34:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:34:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:34:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:34:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:35:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:35:37Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:35:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:35:42Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:36:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:36:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:36:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:36:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:37:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:37:37Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:37:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:37:42Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:38:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:38:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:38:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:38:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:39:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:39:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:39:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:39:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:40:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:40:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:40:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:40:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
md5-172cfb946acd981b84ef26a96c0dc96c
# grep POST /tmp/simple_notify.log |wc -l
26
md5-172cfb946acd981b84ef26a96c0dc96c
# cat /tmp/simple_notify.log
2017-01-11 11:22:37,869 - INFO - Got POST notify request with data '{"status": "firing", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "firing", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "0001-01-01T00:00:00Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}, {"status": "firing", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "0001-01-01T00:00:00Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:27:37,904 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}, {"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:28:42,965 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:28:47,969 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:29:47,843 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:30:42,858 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:30:47,869 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:31:42,828 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:31:47,834 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:32:37,929 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}, {"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:32:42,818 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:32:47,825 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:33:37,852 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}, {"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:33:42,838 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:33:47,865 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:34:37,819 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:34:47,818 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:35:47,837 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:36:47,823 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:37:47,873 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time%28%29+-+container_last_seen%7Bcontainer_name%3D~%22nova-%28compute%7Cscheduler%7Cconductor%7Capi%7Cnovncproxy%7Cvirtlogd%7Clibvirt%7Cconsoleauth%29%22%7D+%3E+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:38:47,821 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus
bkupidura
on 12 Jan 2017
👍4
@bkupidura I created a new ticket for our problem. Here you can find it: https://github.com/prometheus/alertmanager/issues/611
Would you be so nice and leave your comment there too? Thanks :)
tsboj17
on 24 Jan 2017
Was this page helpful?
0 / 5 - 0 ratings
Related issues
svenmueller
·
5Comments
fchiorascu
·
5Comments
pborzenkov
·
5Comments
ingshtrom
·
5Comments
yongzhang
·
5Comments
Most helpful comment
Hello, i observed very similar behavior, but in my case resolve_timeout is 5m and max evaluation time is 1m. I dont think that this is normal to receive more than 1 'resolved' message.
In my env i have 2 alertmanagers in cluster+2 prometheus which scrape the same data.
Alertmanager version: 0.5.1
Prometheus version: 1.4.1