Kapacitor: Alert when host is offline

[qindj]

Dear:

I want to receive alerts when host is offline, is that possible in kapacitor?

As I tested with SNMP plugin, it shows me "Error reading from UDP XXXXX" on STDERR, can kapacitor catch it?

Thanks & Regards!

[rossmcdonald]

@qindj A deadman function was added recently which acts as a dead man's switch, with documentation available here. Is that what you are looking for?

[qindj]

@rossmcdonald thanks!

[qindj]

@rossmcdonald Sorry for reopen this issue!

One more question, can I use variables such as "host" or "ip" in deadman's template?

thanks!

[rossmcdonald]

@qindj I personally haven't experimented with it yet, but I believe so. The deadman function returns an alert node, which could then be used to send a custom message about the host that isn't responding (either through message or id).

[nathanielc]

@qindj Yes, you can use host and ip in the template assuming they are tags on the data points.
For example in the TICKscript

.message('host: {{ index .Tags "host"}} ip: {{ index .Tags "ip"}}')

or in the config:

message = "host: {{ index .Tags \"host\"}} ip: {{ index .Tags \"ip\"}}"

[qindj]

thanks @rossmcdonald @nathanielc

I tested but without success.

I have a db named "test", and using telegraf to get "sysuptime" from 5 linux machines through SNMP.

I want to monitor these machines, and when kapacitor didn't receive stream in 25s, send an email alert with "host" which can shows me who is offline.

As I tested according to @nathanielc , the "host" is always empty.

The data looks like following, "host" maybe different

select * from sysuptime limit 10
name: sysuptime
time host sysuptime
1454394310000000000 172.16.16.62 189383864
1454394320000000000 172.16.16.62 189384864
1454394330000000000 172.16.16.62 189385864
1454394340000000000 172.16.16.62 189386864
1454394350000000000 172.16.16.62 189387864
1454394360000000000 172.16.16.62 189388864
1454394370000000000 172.16.16.62 189389864
1454394380000000000 172.16.16.62 189390864
1454394390000000000 172.16.16.62 189391864
1454394400000000000 172.16.16.62 189392864

And TICKscript:

convertSNMPTimeticks is UDF

var data = stream
    .from()
        .database('test')
        .measurement('sysuptime')
        .groupBy('host')

data.deadman(0.0, 25s)
    .message('{{ index .Tags "host" }} is {{ .Level }} value:{{ index .Fields "value" }}')

data.convertSNMPTimeticks()
        .field('sysuptime')
        .as('uptime')
    .influxDBOut()
        .database('norgin')
        .retentionPolicy('uptime')
        .measurement('uptime') 

and the result is:
is CRITICAL value:<no value>

Did I miss something?

Thanks and regards!

[nathanielc]

The deadman's switch doesn't work with the groupBy node's the way you are expecting. This is a result of how groups are implemented. But I think your example is valid and should work.

Basically what you want is to be able to watch each individual group for a lack of points. Instead what you have right now is a deadman's switch on all hosts combined and no tag information on any host. This should be a simple addition to how the deadman's switch works.

Unfortunately I can't think of a good workaround at the moment.

[markgoffin]

Sorry if this is a silly question. Does deadman still work if there is no influx data coming in? I am using procstat from Telegraf and want to be able to alarm if the process is no longer alive. Is this possible with Kapacitor?

[nathanielc]

@markgoffin yes, that is the purpose of the deadman's switch. It monitors the flow of data through a part of the task, typically the start, and triggers an alert if the data stops flowing.

[markgoffin]

Ah ok, I can't seem to get it to work. My tick file looks like this:

    .from()
        .measurement('procstat')
        .where(lambda: "process_name" == 'myapp')
    .deadman(0.0, 20s)
        .log('/tmp/deadman.log')

I have Telegraf configured to send every 10s. I perform the following steps:

• Start myapp (have confirmed that procstat stats appear in influxDB)
• Enable the kapacitor task
• Stop myapp (and confirm that the stats stop appearing in influxDB)

And no log file is created in /tmp

Have I done something stupid? (kapacitor v0.10.1-1)

[nathanielc]

@markgoffin No, your TICKscript is logically correct but Kapacitor's implementation is confusing.

Basically what is happening is there are two values to track for each node:

1. collected points -- number of points the node has received
2. emitted points -- number of points the node has sent to its children

The deadman's switch is watching the collected metric. In your case the node is still receiving data for other processes but not the myapp process so the alert is not triggered. I am looking into how it would effect other uses cases to trigger the alert off the emitted metric which would take into account the where expression.

As a work around for now could add in a dumy node that receives the values from the stream node and trigger the deadman's switch.

Example:

    .from()
        .measurement('procstat')
        .where(lambda: "process_name" == 'myapp')
    //add no op node as workaround
    .eval(lambda: "cpu_user").as('cpu_user')
    .deadman(0.0, 20s)
        .log('/tmp/deadman.log')

The no op node should not be necessary so with the group by fixes i'll see how this can be improved.

[qindj]

Tested and "groupBy" is working perfect , thanks @nathanielc !!

[adityacs]

@qindj Could you please let me know how did "group by" work for you?. Below is my script. However, i get "null" in hostname

// Dataframe
var data = batch
|query('''SELECT 100 - (mean(usage_idle) + mean(usage_iowait)) as cpu_usage FROM "telegraf"."default"."cpu" ''')
.period(period)
.every(every)
.groupBy(time(10s), 'host')

// Thresholds
|alert()
.id('{{ index .Tags "host"}}/cpu_used')
.message('{{ .Level}}: {{ .Name }}: {{ index .Tags "host" }} has cpu usage: {{ index .Fields "cpu_usage" }}')
.warn(lambda: "cpu_usage" > warn)
.crit(lambda: "cpu_usage" > crit)
.log('/tmp/cpu_alerts.log')

data
|deadman(0.0, 1m)
.message('{{ index .Tags "host" }} is {{ if eq .Level "OK" }}alive{{ else }}dead{{ end }}.')
.log('/tmp/cpu_deadman_alerts.log')

[lfarkas]

is there anybody who can reopen this issue?
even though this issue state (and the deadman's message changes) deadman still do NOT got emitted values. here is the output of my kapacitor-1.4.1 and as you can see both before and after node got the emitted values but not the deadman

# kapacitor vars
...
   "f9771a78-60f9-4fef-a224-1f09b6c860b8": {
      "name": "edges",
      "tags": {
        "host": "localhost",
        "task": "a3_sia_hashrate",
        "parent": "from1",
        "child": "window2",
        "type": "stream",
        "cluster_id": "4ece60d6-f44d-4ec1-9485-5a4894af5b60",
        "server_id": "d42a1414-e7fa-4d66-ad3e-4d82f0b94191"
      },
      "values": {
        "collected": 6,
        "emitted": 6
      }
    },
    "ae382e9d-6c06-4cf8-be50-625a4874ab22": {
      "tags": {
        "id": "deadman",
        "cluster_id": "4ece60d6-f44d-4ec1-9485-5a4894af5b60",
        "server_id": "d42a1414-e7fa-4d66-ad3e-4d82f0b94191",
        "host": "localhost"
      },
      "values": {
        "collected": 3
      },
      "name": "topics"
    },
    "d5558e49-9aa0-4455-aa05-1cfef5851f26": {
      "name": "edges",
      "tags": {
        "type": "stream",
        "task": "task_master:main",
        "cluster_id": "4ece60d6-f44d-4ec1-9485-5a4894af5b60",
        "server_id": "d42a1414-e7fa-4d66-ad3e-4d82f0b94191",
        "host": "localhost",
        "parent": "stats",
        "child": "stream"
      },
      "values": {
        "collected": 14650,
        "emitted": 14650
      }
    },
...

[eohan01]

With all the confusion and having to write two streams etc, and the group by issue that i have read in every post i have found, couldn't you just allow a collection on point, say cpu = and trigger no data is returned and has 0 points from the measurement. not the value 0 but 0 points or a no data returned flag.

I am having the same issues trying to use the Tick Script for deadman on 100+ servers monitoring CPU. It does not seem to trigger the event for NO DATA RETURNED.
I have tried group by on the Tags, Group by *, no group by and the deadman does not seem to trigger. I really don't have to write a new rule for 100+ servers or databases. especially don't want to write a new rule each time a server or database is added to the collection

If I could just use a stream collector and trigger my own e-mail / stack etc, saying No metric data collected in the past xx minutes and go from there. Either Deadman need fixed, or further explained on how to group by, so the event triggers on no data and then data now coming in, or we need a NO DATA RETURNED flag in Chronograf / Kapacitor to Simulate what we are trying to accomplish with DEADMAN.