Kapacitor: Kapacitor Deadman switch false positives

[alexganwd]

Kapacitor fires fake alerts with deadman switch feature. We are running Kapacitor and InfluxDB in the same box.
We are pushing metrics from servers using telegraf every 60s. We have only 3 streams configured in Kapcitord;

• Usage Disk
• Cpu
• Deadman Switch

The first two work as expected, The third one is failing every so often.

If we stop telegraf agent we get a proper alert on time and when telegraf is back we get a second alert, but every so often we get alerts from other servers which did not fail. Sometimes is just one node, other times 20 or 30. Around 2 minutes later we get recovery alerts.

Kapacitor and telegraf running on debug so we can see the points are been pushed every minute but kapacitor keeps firing these alerts randomly.

Operating system: CentOS
Full datails operating system: Amazon Linux AMI release 2016.09 - 4.4.35-33.55.amzn1.x86_64
Kapacitor Version: Kapacitor 1.2.0 (git: master 5408057e5a3493d3b5bd38d5d535ea45b587f8ff) installed from .rpm
Stream configuration

ID: deadman-test-projectname
Error:
Template:
Type: stream
Status: enabled
Executing: true
Created: 22 Apr 17 09:48 UTC
Modified: 22 Apr 17 16:14 UTC
LastEnabled: 22 Apr 17 16:14 UTC
Databases Retention Policies: ["projectName"."only_one_month"]
TICKscript:
var id = '{{ index .Tags "host"}}'

var message_dead = '** PROBLEM Service Alert: {{ index .Tags "host" }} is DEAD'

var message_alive = '** Service Alert: {{ index .Tags "host" }} is RECEIVING METRICS AGAIN'

var data = stream
    |from()
        .database('projectName')
        .retentionPolicy('only_one_month')
        .groupBy(['environment', 'host'])
        .where(lambda: TRUE)
    |deadman(0.9, 120s, lambda: "environment" == 'test')
        .id(id)
        .message('{{ if eq .Level "OK" }}' + message_alive + '{{ else }}' + message_dead + '{{ end }}.')
        .stateChangesOnly()
        .hipChat()
        .room('SaaS Alerts')
        .pagerDuty()
        .serviceKey('my_service_key')
    |influxDBOut()
        .database('kapacitor_alerts')

DOT:
digraph deadman-test-projectName {
graph [throughput="0.00 points/s"];   // Note: This value is different every time I run the command, its a coincidence is 0.00 now

stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="6177489"];

from1 [avg_exec_time_ns="35.168µs" ];
from1 -> noop3 [processed="6177489"];

noop3 [avg_exec_time_ns="0s" ];

stats2 [avg_exec_time_ns="199.096µs" ];
stats2 -> derivative4 [processed="111747"];

derivative4 [avg_exec_time_ns="2.048µs" ];
derivative4 -> alert5 [processed="111656"];

alert5 [alerts_triggered="697" avg_exec_time_ns="66.499µs" crits_triggered="348" infos_triggered="0" oks_triggered="349" warns_triggered="0" ];
alert5 -> influxdb_out6 [processed="697"];

influxdb_out6 [avg_exec_time_ns="16.541µs" points_written="697" write_errors="0" ];
}

Proof there are enough points to not fire an alert

A fresh example (sorry for the obfuscated names):
Log in kapacitor.log
Metrics received during two minutes with no issues and alert fired:

[httpd] 2017/04/24 09:07:00 D! write body received by handler: swap,dc=eu-west-1,environment=test,host=THIS_IS_THE_SERVER_NAME,influxdb_database_name=projectName free=0i,total=0i,used=0i,used_percent=0 1493024820000000000

2017/04/24 09:08:00 D! OK alert triggered id:THIS_IS_THE_SERVER_NAME msg:** Service Alert: THIS_IS_THE_SERVER_NAME is RECEIVING METRICS AGAIN. data:&{stats map[influxdb_database_name:projectName dc:eu-west-1 environment:test host:THIS_IS_THE_SERVER_NAME] [time emitted] [[2017-04-24 09:06:00 +0000 UTC 30]]}

[httpd] 2017/04/24 09:08:00 D! write body received by handler: swap,dc=eu-west-1,environment=test,host=THIS_IS_THE_SERVER_NAME,influxdb_database_name=projetName free=0i,total=0i,used=0i,used_percent=0 1493024880000000000

I hope you can shed some light on this.

Thanks,
Alejandro Gandara

[jcmcken]

This might be a similar problem to what's described here.

The solution in that case was to make sure your alert id is unique per stream. So if you're grouping by host and environment, make sure your id also has both of those data. Otherwise multiple streams will influence the same alert, and you'll get weird, out-of-order, unexpected state changes in your alerts.

[StefanSa]

Hi Alex,
did you find a solution to this problem?

[alexganwd]

We didn’t. We used lambda functions instead. The issue still persist in the
latest versions.

On Sun 10 Jun 2018 at 15:40, StefanSa [e-mail] wrote:

Hi Alex,
did you find a solution to this problem?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/influxdata/kapacitor/issues/1339#issuecomment-396054313,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJwIaTBNnKvswR_qbjKkX-n1Cds79cx_ks5t7S_xgaJpZM4NF9VF
.

[advaitvdeo]

We are facing same issue where deadman alerts are firing even though system is up and running.
We are getting lot of false positive alerts because of deadman.

As suggested by @jcmcken we are grouping by host and using host tag in task ID. But still false positive alerts are firing

Here is our TICK script

    |from()
        .measurement('cpu')
        .groupBy('host')
    |deadman(0.0, 600s)
        .id('{{ index .Tags "host"}}')
        .message(message_val)
        .details(details_val)
        .email('[email protected]')
        .stateChangesOnly()

Above link to google group is not working.
Any suggestions.

[jcmcken]

@advaitvdeo @alexganwd Are you sure you don't have other TICK scripts with the id set to {{ index .Tags "host" }}? Try using something like {{ .Tags.host }} deadman as the ID to make absolutely sure.

[advaitvdeo]

Actually we have around 15 deadman alerts (one for each component/application) and everyone uses {{ index .Tags "host" }} as ID.
Is this an issue? I was assuming that host name is different for each component so that would make ID as unique.

Please let us know.

[jcmcken]

@advaitvdeo It's definitely an issue if the IDs are not unique once the ID is rendered. If you have a server with an alert with an ID of {{ index .Tags "host" }} that resolves to example.com and an application running on that same host with a different alert, but with the same ID (example.com), then they're effectively sharing the same alert state. The ID is like the primary key of the alert state. It's like two threads modifying the same object in memory with no locks -- you'll get indeterminate behavior.

You should give your host alert an ID like {{ index .Tags "host" }} (host) and your app alert an ID like {{ index .Tags "host" }} (app) so each alert has independent state.

[advaitvdeo]

For all other alerts, we have given .ID as name of the alert itself. So it does not clash with deadman alert name

[advaitvdeo]

is it possible that too many stream alerts in kapacitor can make kapacitor "loose" some points in-between and can cause false deadman alerts. Because amount of data kapacitor is getting via stream is huge.

I tried logging using log() node in kapacitor deadman TICK script and checked the timestamp of points in kapacitor.log. I do see that points are not logged every 1min. Points are logged with random "gaps" in between. This is making me think if kapacitor is overloaded and needs to be scaled? is it not able to consume the data its getting via stream?

Please let me know your opinion.

[jcmcken]

@advaitvdeo

For all other alerts, we have given .ID as name of the alert itself. So it does not clash with deadman alert name

I don't understand. can you paste multiple configs in here? Paste a few deadman alerts for the host and apps. Your sentence makes it sound like you're doing this:

  |deadman(...options...)
   .id("{{ .ID }}")

...but I don't think this works. I'm also going to guess that's not what you meant, but I can't tell.

[advaitvdeo]

Here are the .id for few deadman alerts

    |from()
        .measurement('hadoopMonitoring')
        .groupBy('host')
    |deadman(0.0, 600s)
        .id('{{ index .Tags "host"}}')
        .message(message_val)
        .details(details_val)

    |from()
        .measurement('<measurement>')
        .groupBy('env','tag_5')
    |deadman(0.0, 5400s, lambda:"tag_5" == 'total-records')
        .id('{{ index .Tags "env"}} - {{ index .Tags "tag_5"}}')
        .message(message_val)
        .details(details_val)

    |from()
        .measurement('mysql')
        .groupBy('host')
    |deadman(0.0, 1200s)
        .id('{{ index .Tags "host"}}')
        .message(message_val)
        .details(details_val)

[jcmcken]

Are the first and third applied to the same host? If so, their alert IDs are colliding.