Following are the details
I tried with the exact same version of Influxdb,kapacitor and telegraf as was mentioned in the article but it seemed like many changes were made in kapacitor 1.0 beta later like
the old task.db database file since it is still needed to do the migration.
[storage]
boltdb = "/var/lib/kapacitor/kapacitor.db"
as I am receiving error exactly at this point.
1.First I tried with
Software Versions:
InfluxDB - 0.13.0
Telegraf - 0.13.1
Kapacitor - 0.13.1
Later on I uninstalled Kapacitor -0.13.1 and installed 1.1.1
Here are the configuration files
1.influxdb.conf
```
logging-enabled = true
pprof-enabled = false
[admin]
enabled = true
bind-address = ":8083"
https-enabled = false
https-certificate = "/etc/ssl/influxdb.pem"
[http]
enabled = true
bind-address = ":8086"
auth-enabled = false
log-enabled = true
write-tracing = false
logging-enabled = true
pprof-enabled = false
[admin]
enabled = true
bind-address = ":8083"
https-enabled = false
https-certificate = "/etc/ssl/influxdb.pem"
[http]
enabled = true
bind-address = ":8086"
auth-enabled = false
log-enabled = true
write-tracing = false
pprof-enabled = false
https-enabled = false
https-certificate = "/etc/ssl/influxdb.pem"
max-row-limit = 10000
[subscriber]
enabled = true
http-timeout = "30s"
[[udp]]
enabled = false
[continuous_queries]
log-enabled = true
enabled = true
Other details are omitted as of now
2.telegraf.conf
[agent]
interval = "1s"
[outputs]
[outputs.influxdb]
urls = ["http://localhost:8086"]
database = "kapacitor_example"
user_agent = "telegraf"
[cpu]
percpu = false
totalcpu = true
drop = ["cpu_time"]
3.Kapacitor.conf
hostname = "localhost"
data_dir = "/var/lib/kapacitor"
default-retention-policy = ""
[http]
bind-address = ":9092"
auth-enabled = false
log-enabled = true
write-tracing = false
pprof-enabled = false
https-enabled = false
hostname = "localhost"
data_dir = "/var/lib/kapacitor"
default-retention-policy = ""
[http]
bind-address = ":9092"
auth-enabled = false
log-enabled = true
write-tracing = false
pprof-enabled = false
https-enabled = false
https-certificate = "/etc/ssl/kapacitor.pem"
[config-override]
enabled = true
[logging]
file = "/var/log/kapacitor/kapacitor.log"
level = "INFO"
[replay]
dir = "/var/lib/kapacitor/replay"
[[influxdb]]
default = true
name = "localhost"
urls = ["http://localhost:8086"]
username = ""
password = ""
ssl-ca = ""
ssl-cert = ""
ssl-key = ""
insecure-skip-verify = false
timeout = "0"
disable-subscriptions = false
udp-buffer = 1000
udp-read-buffer = 0
startup-timeout = "0"
startup-timeout = "5m"
disable-subscriptions = false
subscription-protocol = "http"
4.cpu_alert.tick
stream
|from()
.measurement('cpu')
|alert()
.crit(lambda: "usage_idle" < 90 )
.log('/tmp/cpu_idle_alerts.log')
I followed [https://docs.influxdata.com/kapacitor/v0.12/introduction/getting_started/](url) for installation.
### Steps to reproduce
kapacitor define cpu_alert -type stream -tick ~/cpu_alert.tick -dbrp kapacitor_example.default
$ kapacitor show cpu_alert
ID: cpu_alert
Error:
Template:
Type: stream
Status: enabled
Executing: true
Created: 10 Dec 16 13:44 PST
Modified: 11 Dec 16 09:39 PST
LastEnabled: 11 Dec 16 09:39 PST
Databases Retention Policies: ["kapacitor_example"."default"]
TICKscript:
stream
|from()
.measurement('cpu')
|alert()
.crit(lambda: "usage_idle" < 90)
.log('/tmp/cpu_idle_alerts.log')
DOT:
digraph cpu_alert {
graph [throughput="0.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="0"];
from1 [avg_exec_time_ns="0s" ];
from1 -> alert2 [processed="0"];
alert2 [alerts_triggered="0" avg_exec_time_ns="0s" crits_triggered="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" ];
[ticku@localhost ~]$ kapacitor record stream -task cpu_alert -duration 20s -no-wait
922d81ca-9375-413c-9ffb-7c8fdd68a360
[ticku@localhost ~]$ kapacitor list recordings 922d81ca-9375-413c-9ffb-7c8fdd68a360
ID Type Status Size Date
922d81ca-9375-413c-9ffb-7c8fdd68a360 stream running 0 B 11 Dec 16 09:46 PST
[ticku@localhost ~]$ sudo cat /var/log/kapacitor/kapacitor.log
[run] 2016/12/10 18:16:39 I! Kapacitor starting, version 1.1.1, branch master, commit d1e0173df2b9f44c64b7bbf1a120a1bdb3ef683e
[run] 2016/12/10 18:16:39 I! Go version go1.7.4
[srv] 2016/12/10 18:16:39 I! Kapacitor hostname: localhost
[srv] 2016/12/10 18:16:39 I! ClusterID: e364dd76-166e-4e1c-bbe5-06c1c24aac1e ServerID: e859317a-0ea9-4272-a0bd-3b8ce8eeec03
[task_master:main] 2016/12/10 18:16:39 I! opened
[task_store] 2016/12/10 18:16:39 W! could not open old boltd for task_store. Not performing migration. Remove the task_store.dir configuration to disable migration.
[task_master:main] 2016/12/10 18:16:39 I! Started task: cpu_alert
[smtp] 2016/12/10 18:16:39 I! Starting SMTP service
[stats] 2016/12/10 18:16:39 I! opened service
[httpd] 2016/12/10 18:16:39 I! Starting HTTP service
[httpd] 2016/12/10 18:16:39 I! Authentication enabled: false
[run] 2016/12/10 18:16:39 I! Listening for signals
[httpd] ::1 - - [10/Dec/2016:18:17:21 -0800] "GET /kapacitor/v1/debug/vars HTTP/1.1" 200 1851 "-" "KapacitorClient" f264449f-bf47-11e6-8001-000000000000 4124
[httpd] ::1 - - [10/Dec/2016:18:17:55 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 612 "-" "KapacitorClient" 06b41a2c-bf48-11e6-8002-000000000000 3782
[httpd] ::1 - - [10/Dec/2016:18:17:55 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 614 "-" "KapacitorClient" 06b4d137-bf48-11e6-8003-000000000000 17560
[task_master:main] 2016/12/10 18:17:55 I! Stopped task: cpu_alert
[httpd] ::1 - - [10/Dec/2016:18:17:55 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 440 "-" "KapacitorClient" 06b79c77-bf48-11e6-8004-000000000000 12599
[task_master:main] 2016/12/10 18:17:55 I! Started task: cpu_alert
[httpd] ::1 - - [10/Dec/2016:18:17:55 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 600 "-" "KapacitorClient" 06ba122a-bf48-11e6-8005-000000000000 23309
[httpd] ::1 - - [10/Dec/2016:18:18:03 -0800] "GET /kapacitor/v1/tasks?dot-view=attributes&fields=type&fields=status&fields=executing&fields=dbrps&limit=100&offset=0&pattern=cpu%2A&replay-id=&script-format=formatted HTTP/1.1" 200 201 "-" "KapacitorClient" 0bc55423-bf48-11e6-8006-000000000000 2564
[httpd] ::1 - - [10/Dec/2016:18:18:04 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 600 "-" "KapacitorClient" 0c497bab-bf48-11e6-8007-000000000000 25805
[httpd] ::1 - - [10/Dec/2016:18:18:21 -0800] "GET /kapacitor/v1/tasks?dot-view=attributes&fields=link&limit=100&offset=0&pattern=cpu_alert&replay-id=&script-format=formatted HTTP/1.1" 200 124 "-" "KapacitorClient" 166f8ef7-bf48-11e6-8008-000000000000 1401
[httpd] ::1 - - [10/Dec/2016:18:18:21 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 609 "-" "KapacitorClient" 16700e1c-bf48-11e6-8009-000000000000 12661
[httpd] ::1 - - [10/Dec/2016:18:20:07 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 609 "-" "KapacitorClient" 554f65cc-bf48-11e6-800a-000000000000 14409
[run] 2016/12/10 18:26:14 I! Signal received, initializing clean shutdown...
[run] 2016/12/10 18:26:14 I! Waiting for clean shutdown...
[task_master:main] 2016/12/10 18:26:14 I! Stopped task: cpu_alert
[smtp] 2016/12/10 18:26:14 I! Closing SMTP service
[stats] 2016/12/10 18:26:14 I! closed service
[httpd] 2016/12/10 18:26:14 I! Closed HTTP service
[task_master:main] 2016/12/10 18:26:14 I! closed
[run] 2016/12/10 18:26:14 I! server shutdown completed
[run] 2016/12/10 18:26:17 I! Kapacitor starting, version 1.1.1, branch master, commit d1e0173df2b9f44c64b7bbf1a120a1bdb3ef683e
[run] 2016/12/10 18:26:17 I! Go version go1.7.4
[srv] 2016/12/10 18:26:17 I! Kapacitor hostname: localhost
[srv] 2016/12/10 18:26:17 I! ClusterID: e364dd76-166e-4e1c-bbe5-06c1c24aac1e ServerID: e859317a-0ea9-4272-a0bd-3b8ce8eeec03
[task_master:main] 2016/12/10 18:26:17 I! opened
[task_store] 2016/12/10 18:26:17 W! could not open old boltd for task_store. Not performing migration. Remove the task_store.dir configuration to disable migration.
[task_master:main] 2016/12/10 18:26:18 I! Started task: cpu_alert
[smtp] 2016/12/10 18:26:18 I! Starting SMTP service
[stats] 2016/12/10 18:26:18 I! opened service
[httpd] 2016/12/10 18:26:18 I! Starting HTTP service
[httpd] 2016/12/10 18:26:18 I! Authentication enabled: false
[run] 2016/12/10 18:26:18 I! Listening for signals
[httpd] ::1 - - [10/Dec/2016:18:26:35 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 609 "-" "KapacitorClient" 3d139b0e-bf49-11e6-8001-000000000000 19523
[httpd] ::1 - - [10/Dec/2016:18:26:35 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 612 "-" "KapacitorClient" 3d1714ff-bf49-11e6-8002-000000000000 25929
[task_master:main] 2016/12/10 18:26:36 I! Stopped task: cpu_alert
[httpd] ::1 - - [10/Dec/2016:18:26:36 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 438 "-" "KapacitorClient" 3d1b490d-bf49-11e6-8003-000000000000 20700
[task_master:main] 2016/12/10 18:26:36 I! Started task: cpu_alert
[httpd] ::1 - - [10/Dec/2016:18:26:36 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 599 "-" "KapacitorClient" 3d1e8e28-bf49-11e6-8004-000000000000 25508
[httpd] ::1 - - [10/Dec/2016:18:26:49 -0800] "GET /kapacitor/v1/tasks?dot-view=attributes&fields=link&limit=100&offset=0&pattern=cpu_alert&replay-id=&script-format=formatted HTTP/1.1" 200 124 "-" "KapacitorClient" 45614b9a-bf49-11e6-8005-000000000000 7634
[httpd] ::1 - - [10/Dec/2016:18:26:49 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 610 "-" "KapacitorClient" 45629fad-bf49-11e6-8006-000000000000 20701
[httpd] ::1 - - [10/Dec/2016:18:27:04 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 610 "-" "KapacitorClient" 4e24e582-bf49-11e6-8007-000000000000 16534
[run] 2016/12/10 18:31:11 I! Signal received, initializing clean shutdown...
[run] 2016/12/10 18:31:11 I! Waiting for clean shutdown...
[task_master:main] 2016/12/10 18:31:11 I! Stopped task: cpu_alert
[smtp] 2016/12/10 18:31:11 I! Closing SMTP service
[stats] 2016/12/10 18:31:11 I! closed service
[httpd] 2016/12/10 18:31:11 I! Closed HTTP service
[task_master:main] 2016/12/10 18:31:11 I! closed
[run] 2016/12/10 18:31:11 I! server shutdown completed
[run] 2016/12/10 18:32:01 I! Kapacitor starting, version 1.1.1, branch master, commit d1e0173df2b9f44c64b7bbf1a120a1bdb3ef683e
[run] 2016/12/10 18:32:01 I! Go version go1.7.4
[srv] 2016/12/10 18:32:01 I! Kapacitor hostname: localhost
[srv] 2016/12/10 18:32:01 I! ClusterID: e364dd76-166e-4e1c-bbe5-06c1c24aac1e ServerID: e859317a-0ea9-4272-a0bd-3b8ce8eeec03
[task_master:main] 2016/12/10 18:32:01 I! opened
[task_store] 2016/12/10 18:32:01 W! could not open old boltd for task_store. Not performing migration. Remove the task_store.dir configuration to disable migration.
[task_master:main] 2016/12/10 18:32:01 I! Started task: cpu_alert
[smtp] 2016/12/10 18:32:01 I! Starting SMTP service
[stats] 2016/12/10 18:32:01 I! opened service
[httpd] 2016/12/10 18:32:01 I! Starting HTTP service
[httpd] 2016/12/10 18:32:01 I! Authentication enabled: false
[run] 2016/12/10 18:32:01 I! Listening for signals
[httpd] ::1 - - [10/Dec/2016:18:32:14 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 610 "-" "KapacitorClient" 06a0858c-bf4a-11e6-8001-000000000000 3791
[httpd] ::1 - - [10/Dec/2016:18:32:20 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 610 "-" "KapacitorClient" 0a602e3d-bf4a-11e6-8002-000000000000 8258
[httpd] ::1 - - [10/Dec/2016:18:32:20 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 611 "-" "KapacitorClient" 0a6197b0-bf4a-11e6-8003-000000000000 14542
[task_master:main] 2016/12/10 18:32:20 I! Stopped task: cpu_alert
[httpd] ::1 - - [10/Dec/2016:18:32:20 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 438 "-" "KapacitorClient" 0a641f42-bf4a-11e6-8004-000000000000 9198
[task_master:main] 2016/12/10 18:32:20 I! Started task: cpu_alert
[httpd] ::1 - - [10/Dec/2016:18:32:20 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 600 "-" "KapacitorClient" 0a65ca90-bf4a-11e6-8005-000000000000 38318
[httpd] ::1 - - [10/Dec/2016:18:32:24 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 600 "-" "KapacitorClient" 0cb42c9f-bf4a-11e6-8006-000000000000 9891
[httpd] ::1 - - [10/Dec/2016:18:32:28 -0800] "GET /kapacitor/v1/tasks?dot-view=attributes&fields=link&limit=100&offset=0&pattern=cpu_alert&replay-id=&script-format=formatted HTTP/1.1" 200 124 "-" "KapacitorClient" 0f6127b7-bf4a-11e6-8007-000000000000 2811
[httpd] ::1 - - [10/Dec/2016:18:32:28 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 610 "-" "KapacitorClient" 0f620002-bf4a-11e6-8008-000000000000 16844
[run] 2016/12/10 18:38:28 I! Signal received, initializing clean shutdown...
[run] 2016/12/10 18:38:28 I! Waiting for clean shutdown...
[task_master:main] 2016/12/10 18:38:28 I! Stopped task: cpu_alert
[smtp] 2016/12/10 18:38:28 I! Closing SMTP service
[stats] 2016/12/10 18:38:28 I! closed service
[httpd] 2016/12/10 18:38:28 I! Closed HTTP service
[task_master:main] 2016/12/10 18:38:28 I! closed
[run] 2016/12/10 18:38:28 I! server shutdown completed
[run] 2016/12/10 18:38:52 I! Kapacitor starting, version 1.1.1, branch master, commit d1e0173df2b9f44c64b7bbf1a120a1bdb3ef683e
[run] 2016/12/10 18:38:52 I! Go version go1.7.4
[srv] 2016/12/10 18:38:52 I! Kapacitor hostname: localhost
[srv] 2016/12/10 18:38:52 I! ClusterID: e364dd76-166e-4e1c-bbe5-06c1c24aac1e ServerID: e859317a-0ea9-4272-a0bd-3b8ce8eeec03
[task_master:main] 2016/12/10 18:38:52 I! opened
[task_store] 2016/12/10 18:38:52 W! could not open old boltd for task_store. Not performing migration. Remove the task_store.dir configuration to disable migration.
[task_master:main] 2016/12/10 18:38:52 I! Started task: cpu_alert
[smtp] 2016/12/10 18:38:52 I! Starting SMTP service
[stats] 2016/12/10 18:38:52 I! opened service
[httpd] 2016/12/10 18:38:52 I! Starting HTTP service
[httpd] 2016/12/10 18:38:52 I! Authentication enabled: false
[run] 2016/12/10 18:38:52 I! Listening for signals
[httpd] ::1 - - [10/Dec/2016:18:39:13 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 610 "-" "KapacitorClient" 0090d939-bf4b-11e6-8001-000000000000 4844
[httpd] ::1 - - [10/Dec/2016:18:39:13 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 610 "-" "KapacitorClient" 0091c034-bf4b-11e6-8002-000000000000 24413
[task_master:main] 2016/12/10 18:39:13 I! Stopped task: cpu_alert
[httpd] ::1 - - [10/Dec/2016:18:39:13 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 437 "-" "KapacitorClient" 0095fb86-bf4b-11e6-8003-000000000000 16514
[task_master:main] 2016/12/10 18:39:13 I! Started task: cpu_alert
[httpd] ::1 - - [10/Dec/2016:18:39:13 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 600 "-" "KapacitorClient" 0098b80b-bf4b-11e6-8004-000000000000 26238
[httpd] ::1 - - [10/Dec/2016:18:39:20 -0800] "GET /kapacitor/v1/tasks?dot-view=attributes&fields=link&limit=100&offset=0&pattern=cpu_alert&replay-id=&script-format=formatted HTTP/1.1" 200 124 "-" "KapacitorClient" 048759ae-bf4b-11e6-8005-000000000000 1720
[httpd] ::1 - - [10/Dec/2016:18:39:20 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 611 "-" "KapacitorClient" 0487d50f-bf4b-11e6-8006-000000000000 8508
[httpd] ::1 - - [10/Dec/2016:18:39:24 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 611 "-" "KapacitorClient" 0707ad3e-bf4b-11e6-8007-000000000000 14243
[httpd] ::1 - - [11/Dec/2016:09:39:04 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 611 "-" "KapacitorClient" b57659a9-bfc8-11e6-8008-000000000000 37256
[httpd] ::1 - - [11/Dec/2016:09:39:04 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 615 "-" "KapacitorClient" b57dca07-bfc8-11e6-8009-000000000000 35760
[task_master:main] 2016/12/11 09:39:04 I! Stopped task: cpu_alert
[httpd] ::1 - - [11/Dec/2016:09:39:04 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 441 "-" "KapacitorClient" b58356b2-bfc8-11e6-800a-000000000000 19168
[task_master:main] 2016/12/11 09:39:04 I! Started task: cpu_alert
[httpd] ::1 - - [11/Dec/2016:09:39:04 -0800] "PATCH /kapacitor/v1/tasks/cpu_alert HTTP/1.1" 200 602 "-" "KapacitorClient" b5865a7c-bfc8-11e6-800b-000000000000 21753
[httpd] ::1 - - [11/Dec/2016:09:42:35 -0800] "GET /kapacitor/v1/tasks?dot-view=attributes&fields=type&fields=status&fields=executing&fields=dbrps&limit=100&offset=0&pattern=cp%2A&replay-id=&script-format=formatted HTTP/1.1" 200 201 "-" "KapacitorClient" 3342378a-bfc9-11e6-800c-000000000000 15366
[httpd] ::1 - - [11/Dec/2016:09:42:35 -0800] "GET /kapacitor/v1/tasks/cpu_alert?dot-view=attributes&replay-id=&script-format=formatted HTTP/1.1" 200 602 "-" "KapacitorClient" 33ced0aa-bfc9-11e6-800d-000000000000 5718
[httpd] ::1 - - [11/Dec/2016:09:44:37 -0800] "POST /kapacitor/v1/recordings/stream HTTP/1.1" 201 214 "-" "KapacitorClient" 7bfc88ff-bfc9-11e6-800e-000000000000 37514
[httpd] ::1 - - [11/Dec/2016:09:44:59 -0800] "GET /kapacitor/v1/tasks?dot-view=attributes&fields=type&fields=status&fields=executing&fields=dbrps&limit=100&offset=0&pattern=fccd05f3-be59-4e93-83db-67027ecd4a12&replay-id=&script-format=formatted HTTP/1.1" 200 43 "-" "KapacitorClient" 8913636c-bfc9-11e6-800f-000000000000 1115
[httpd] ::1 - - [11/Dec/2016:09:46:39 -0800] "POST /kapacitor/v1/recordings/stream HTTP/1.1" 201 213 "-" "KapacitorClient" c4f25780-bfc9-11e6-8010-000000000000 11718
[httpd] ::1 - - [11/Dec/2016:09:46:59 -0800] "GET /kapacitor/v1/recordings?fields=type&fields=size&fields=date&fields=status&limit=100&offset=0&pattern=922d81ca-9375-413c-9ffb-7c8fdd68a360 HTTP/1.1" 200 223 "-" "KapacitorClient" d0d0fa26-bfc9-11e6-8011-000000000000 645
[ticku@localhost ~]$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.13.0
InfluxDB shell version: 0.13.0
show subscriptions
name: telegraf
retention_policy name mode destinations
default kapacitor-e364dd76-166e-4e1c-bbe5-06c1c24aac1e ANY [http://localhost:9092]
retention_policy name mode destinations
monitor kapacitor-e364dd76-166e-4e1c-bbe5-06c1c24aac1e ANY [http://localhost:9092]
retention_policy name mode destinations
default kapacitor-e364dd76-166e-4e1c-bbe5-06c1c24aac1e ANY [http://localhost:9092]
[httpd] 2016/12/10 17:42:30 ::1 - - [10/Dec/2016:17:42:30 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 1416bc85-bf43-11e6-8032-000000000000 6.05208ms
[httpd] 2016/12/10 17:42:40 ::1 - - [10/Dec/2016:17:42:40 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 1a0b9efe-bf43-11e6-8033-000000000000 7.417738ms
[httpd] 2016/12/10 17:42:40 ::1 - - [10/Dec/2016:17:42:40 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 1a0b9fbc-bf43-11e6-8034-000000000000 10.765394ms
[httpd] 2016/12/10 17:42:50 ::1 - - [10/Dec/2016:17:42:50 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 200197ed-bf43-11e6-8036-000000000000 8.655367ms
[httpd] 2016/12/10 17:42:50 ::1 - - [10/Dec/2016:17:42:50 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 200197ed-bf43-11e6-8035-000000000000 11.39447ms
[httpd] 2016/12/10 17:43:00 ::1 - - [10/Dec/2016:17:43:00 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 25f6f89b-bf43-11e6-8037-000000000000 4.489744ms
[httpd] 2016/12/10 17:43:00 ::1 - - [10/Dec/2016:17:43:00 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 25f74eb0-bf43-11e6-8038-000000000000 4.027544ms
[httpd] 2016/12/10 17:43:10 ::1 - - [10/Dec/2016:17:43:10 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 2becdaa1-bf43-11e6-8039-000000000000 4.398767ms
[httpd] 2016/12/10 17:43:10 ::1 - - [10/Dec/2016:17:43:10 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 2bed6aa4-bf43-11e6-803a-000000000000 2.589761ms
[httpd] 2016/12/10 17:43:20 ::1 - - [10/Dec/2016:17:43:20 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 31e2dd1e-bf43-11e6-803b-000000000000 4.905191ms
[httpd] 2016/12/10 17:43:20 ::1 - - [10/Dec/2016:17:43:20 -0800] POST /write?consistency=any&db=telegraf&precision=s&rp=default HTTP/1.1 204 0 - InfluxDBClient 31e3037a-bf43-11e6-803c-000000000000 14.243782ms
[run] 2016/12/10 17:43:25 Signal received, initializing clean shutdown...
[run] 2016/12/10 17:43:25 Waiting for clean shutdown...
[copier] 2016/12/10 17:43:25 copier listener closed
[cluster] 2016/12/10 17:43:25 cluster service accept error: network connection closed
[shard-precreation] 2016/12/10 17:43:25 Precreation service terminating
[snapshot] 2016/12/10 17:43:25 snapshot listener closed
[continuous_querier] 2016/12/10 17:43:25 continuous query service terminating
[retention] 2016/12/10 17:43:25 retention policy enforcement terminating
[monitor] 2016/12/10 17:43:25 shutting down monitor system
[monitor] 2016/12/10 17:43:25 terminating storage of statistics
[subscriber] 2016/12/10 17:43:27 closed service
[run] 2016/12/10 17:43:27 server shutdown completed
[ticku@localhost influxdb]$
As it can be seen the error lies in
`[task_store] 2016/12/10 18:16:39 W! **could not open old boltd for task_store. Not performing migration. Remove the `task_store.dir` configuration to disable migration.**`
How to resolve this?
I tried all possible ways to do that:-
1.Added [subscriber] in kapacitor-0.13.1.
2.Referred to all the available forums and articles available online but none of them provided a solution.
3.I also tried in docker but on enabling the cpu_alert neither the kapacitor records any stream nor is the log file created.'show subscriptions' lists subscriptions correctly and there are no errors in influxdb logs
docker run -d --name influxdb -p 8083:8083 -p 8086:8086 influxdb
docker run -d --name telegraf --net=container:influxdb telegraf
After creating tick folder in root of the host I created tickscript - cpu_alert.tick to that ~/tick folder.
Add tickscript - cpu_alert.tick to that ~/tick folder.
Add ip of influxdb into kapacitor config file and run below command.
docker run -p 9092:9092 -d --link influxdb:influxdb --name kapacitor -v ~/Downloads/kapacitor.conf:/etc/kapacitor/kapacitor.conf:ro -v ~/tick/:/root -w=/root kapacitor
kapacitor define cpu_alert -type stream -tick ~/cpu_alert.tick -dbrp telegraf.autogen
kapacitor enable cpu_alert
```
Tried restarting the containers but it doesnot seem to work.
+1
@ipsha1545 The major version of InfluxDB and Kapacitor must match. Meaning if you are using 0.13.X InfluxDB you need to be using Kapacitor 0.13.X. Since you have upgraded Kapacitor to 1.1.X of Kapacitor you should also upgrade InflxuDB to 1.1.X.
Specifically there are lots of bug fixes for the subscriptions in version 1.0 of Kapacitor/InfluxDB so upgrading my solve your problem.
As it can be seen the error lies in
[task_store] 2016/12/10 18:16:39 W! could not open old boltd for task_store. Not performing migration. Remove thetask_store.dirconfiguration to disable migration.
This is not important, it is just complaining that it could not find an older (0.11) format of the kapacitor database file. That warning is going to be removed in the next release.
You may also find that the output of kapacitor stats ingress is useful in debugging ingest issues with Kapacitor.
I upgraded Influxdb to 1.1.x but still the issue persists.Is there any way to fix it?
Is there any specific version of kapacitor,influxdb and telegraph for which the entire setup will work fine without issues,the stream of data will be visible in kapacitor and the alerts can be visible in the log file either on docker or on localhost?
I'm having the same issue. I upgraded everything to the latest listed on the website today. I can't get anything to work, yet all the communications seem to be working.
@yatahaze @ipsha1545 Any version >= 1.0 should be working without issues. Can you share the output of kapacitor stats ingress?
Sure,
# kapacitor stats ingress
Database Retention Policy Measurement Points Received
_internal monitor cq 299
_internal monitor database 1196
_internal monitor httpd 299
_internal monitor queryExecutor 299
_internal monitor runtime 299
_internal monitor shard 23322
_internal monitor subscriber 2093
_internal monitor tsm1_cache 23322
_internal monitor tsm1_engine 23322
_internal monitor tsm1_filestore 23322
_internal monitor tsm1_wal 23322
_internal monitor write 299
_kapacitor autogen edges 888
_kapacitor autogen ingress 15628
_kapacitor autogen kapacitor 298
_kapacitor autogen runtime 298
telegraf default ceph 139308
telegraf default cpu 10764
telegraf default disk 31993
telegraf default diskio 28106
telegraf default disksmart 1144
telegraf default kernel 1196
telegraf default mem 1196
telegraf default mysql 299
telegraf default mysql_users 613
telegraf default net 16744
telegraf default processes 1196
telegraf default procstat 598
telegraf default sensors_custom 199
telegraf default swap 1196
telegraf default system 1196
telegraf default ups_apc 598
tm default customer_overdue 68
tm default error_checker 353
tm default inventory_count 50
tm default invoices 204
tm default invoices_90d 216
tm default invoices_ytd 220
tm default item_count 168
tm default order_status 77
tm default order_total 12
tm default order_total_status 24
tm default oversold 134
tm default sales_ptd 220
tm default sales_type 84
tm default sales_type2 588
tm default stock 1370
tm default stock_percentage 220
tm default transactions 208
tm default transactions_launch 964
tm default transactions_theme 1280
tm default transactions_themes 152
tm default transactions_type 668
tm default whos_online 238
tm default whos_online_provinces 105
tm default whos_online_total 50
Other random info I can provide:
@yatahaze Looks like Kapacitor is receiving the data, can you share a TICKscript that isn't working? The entire output of kapacitor show TASK_NAME would be best.
Here is the TICKscript and the output.
|from()
.measurement('cpu')
|alert()
.crit(lambda: "usage_idle" < 90)
.log('/tmp/cpu_idle_alerts.log')
```[ticku@localhost ~]$ kapacitor show cpu_alert
ID: cpu_alert
Error:
Template:
Type: stream
Status: enabled
Executing: true
Created: 19 Dec 16 23:29 PST
Modified: 19 Dec 16 23:29 PST
LastEnabled: 19 Dec 16 23:29 PST
Databases Retention Policies: ["telegraf"."autogen"]
TICKscript:
stream
|from()
.measurement('cpu-total')
|alert()
.crit(lambda: "usage_idle" < 90)
.log('/tmp/alerts.log')
DOT:
digraph cpu_alert {
graph [throughput="0.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="0"];
from1 [avg_exec_time_ns="0s" ];
from1 -> alert2 [processed="0"];
alert2 [alerts_triggered="0" avg_exec_time_ns="0s" crits_triggered="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" ];
```
Here's mine:
# kapacitor show cpu_alert
ID: cpu_alert
Error:
Template:
Type: stream
Status: disabled
Executing: false
Created: 19 Dec 16 10:21 PST
Modified: 20 Dec 16 09:03 PST
LastEnabled: 01 Jan 01 00:00 UTC
Databases Retention Policies: ["telegraf"."autogen"]
TICKscript:
stream
// Select just the cpu measurement from our example database.
|from()
.measurement('cpu')
|alert()
.crit(lambda: "usage_idle" < 100)
// Whenever we get an alert write it to a file.
.log('/tmp/alerts.log')
DOT:
digraph cpu_alert {
stream0 -> from1;
from1 -> alert2;
}
And the other I've been testing:
# kapacitor show disk_alert_stream
ID: disk_alert_stream
Error:
Template:
Type: stream
Status: disabled
Executing: false
Created: 19 Dec 16 11:14 PST
Modified: 19 Dec 16 11:51 PST
LastEnabled: 01 Jan 01 00:00 UTC
Databases Retention Policies: ["telegraf"."autogen"]
TICKscript:
// disk_alert_stream
// metric: used_percent
// available_fields: free","inodes_free","inodes_total","inodes_used","total","used"
// TELEGRAF CONFIGURATION
// [[inputs.disk]]
// ignore_fs = ["tmpfs", "devtmpfs"]
// DEFINE: kapacitor define disk_alert_stream -type stream -tick disk/disk_alert_stream.tick -dbrp telegraf.autogen
// ENABLE: kapacitor enable disk_alert_stream
// Parameters
var info = 75
var warn = 85
var crit = 92
var period = 10s
var every = 10s
// Dataframe
var data = stream
|from()
.database('telegraf')
.retentionPolicy('autogen')
.measurement('disk')
.groupBy('host')
|window()
.period(period)
.every(every)
|mean('used_percent')
.as('stat')
// Thresholds
var alert = data
|alert()
.id('{{ index .Tags "host"}}/disk_used')
.message('{{ .ID }}:{{ index .Fields "stat" }}')
.info(lambda: "stat" > info)
.warn(lambda: "stat" > warn)
.crit(lambda: "stat" > crit)
// Alert
alert
.log('/tmp/disk_alert_log.txt')
DOT:
digraph disk_alert_stream {
stream0 -> from1;
from1 -> window2;
window2 -> mean3;
mean3 -> alert4;
}
And my command:
# kapacitor record stream -task cpu_alert -duration 5s
Results in:
# kapacitor list recordings
ID Type Status Size Date
e39ced51-2b32-4a62-88a3-f9c3ccc92fbf stream running 0 B 20 Dec 16 09:04 PST
b1e7c9d4-4cec-46ff-a8ff-e22090ac4969 stream running 0 B 19 Dec 16 12:16 PST
(2nd one is from yesterday)
[ticku@localhost ~]$ kapacitor stats ingress
Database Retention Policy Measurement Points Received
_internal monitor cq 760
_internal monitor database 1520
_internal monitor httpd 760
_internal monitor queryExecutor 760
_internal monitor runtime 760
_internal monitor shard 5320
_internal monitor subscriber 3040
_internal monitor tsm1_cache 5320
_internal monitor tsm1_engine 5320
_internal monitor tsm1_filestore 5320
_internal monitor tsm1_wal 5320
_internal monitor write 760
_kapacitor autogen edges 3795
_kapacitor autogen ingress 18970
_kapacitor autogen kapacitor 759
_kapacitor autogen nodes 2277
_kapacitor autogen runtime 759
telegraf autogen cpu 3796
telegraf autogen disk 3800
telegraf autogen diskio 8916
telegraf autogen kernel 760
telegraf autogen mem 760
telegraf autogen processes 760
telegraf autogen swap 1520
telegraf autogen system 1520
``
+1
Having same issues with influxdb 1.1.1, kapacitor 1.1.1
confirmed that influxdb+kapacitor 1.1.0 works--was able to get through the getting started guide now. Hope this helps narrow things down to a range of changelists. ;)
same issue here
+1
i have the same issue
[dev_dean@VM_61_2_centos ~]$ kapacitor -url http://172.1.56.5:9092 stats ingress
Database Retention Policy Measurement Points Received
_internal monitor cq 220
_internal monitor database 440
_internal monitor httpd 220
_internal monitor queryExecutor 220
_internal monitor runtime 220
_internal monitor shard 1760
_internal monitor subscriber 3514
_internal monitor tsm1_cache 1760
_internal monitor tsm1_engine 1760
_internal monitor tsm1_filestore 1760
_internal monitor tsm1_wal 1760
_internal monitor write 220
_kapacitor autogen edges 573
_kapacitor autogen ingress 3224
_kapacitor autogen kapacitor 108
_kapacitor autogen nodes 291
_kapacitor autogen runtime 108
telegraf autogen cpu 4806
telegraf autogen disk 1314
telegraf autogen diskio 1314
telegraf autogen kernel 438
telegraf autogen mem 438
telegraf autogen mongodb 218
telegraf autogen net 876
telegraf autogen netstat 438
telegraf autogen processes 438
telegraf autogen redis 220
telegraf autogen redis_keyspace 660
telegraf autogen swap 876
telegraf autogen system 876
[dev_dean@VM_61_2_centos ~]$ kapacitor -url http://172.1.56.5:9092 show cpu_alert
ID: cpu_alert
Error:
Template:
Type: stream
Status: enabled
Executing: true
Created: 24 Jan 17 06:07 UTC
Modified: 24 Jan 17 06:08 UTC
LastEnabled: 24 Jan 17 06:08 UTC
Databases Retention Policies: ["telegraf"."default"]
TICKscript:
stream
// Select just the cpu measurement from our example database.
|from()
.database('telegraf')
.measurement('cpu')
|alert()
.crit(lambda: "usage_idle" < 100)
// Whenever we get an alert write it to a file.
.log('/tmp/alerts.log')
DOT:
digraph cpu_alert {
graph [throughput="0.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="0"];
from1 [avg_exec_time_ns="0s" ];
from1 -> alert2 [processed="0"];
alert2 [alerts_triggered="0" avg_exec_time_ns="0s" crits_triggered="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" ];
}
kapacitor logs:
[srv] 2017/01/24 06:06:12 D! opening service: *task_store.Service
[task_store] 2017/01/24 06:06:12 W! could not open old boltd for task_store. Not performing migration. Remove the `task_store.dir` configuration to disable migration.
[srv] 2017/01/24 06:06:12 D! opened service: *task_store.Service
i running influx and kapacitor in docker in kubernetes...
influxdb 1.1.1
kapacitor 1.1.1
I'm experience the same issues with
Here is some extra information and troubleshooting I've been doing, any extra input would be appreciated:
Hello, I'm having an issue with Kapacitor and data ingestion/alerting. The data appears
to be incoming from subscription channels from Influx, however Kapacitor streams
do not appear active.
Kapacitor and InfluxDB are running inside of separate docker containers on same
host.
Any help in diagnosing this problem is greatly appreciated.
The relevant configuration from my Kapacitor config is:
hostname = "kapacitor"
data_dir = "/root/.kapacitor"
default-retention-policy = "one_year"
[http]
bind-address = "0.0.0.0:9092"
auth-enabled = false
log-enabled = true
write-tracing = false
pprof-enabled = false
https-enabled = false
https-certificate = "/etc/ssl/kapacitor.pem"
[replay]
dir = "/root/.kapacitor/replay"
[[influxdb]]
enabled = true
default = true
name = "influxdb"
urls = ["http://influxdb:8086"]
username = ""
password = ""
timeout = 0
subscription-protocol="http"
udp-buffer = 1000
udp-read-buffer = 0
disable-subscriptions = false
[logging]
file = "STDERR"
level = "DEBUG"
I am trying to run 1 task, the cpu task. Kapacitor knows about it and thinks
it is running.
/ # kapacitor show cpu
ID: cpu
Error:
Template:
Type: stream
Status: enabled
Executing: true
Created: 27 Jan 17 19:57 UTC
Modified: 27 Jan 17 19:57 UTC
LastEnabled: 27 Jan 17 19:57 UTC
Databases Retention Policies: ["metrics"."default"]
TICKscript:
var data = stream
|from()
.database('metrics')
.measurement('docker_container_cpu')
data
|deadman(100.0, 10s)|log().prefix('DEADMAN')
data
|log().prefix('STANDARD')
DOT:
digraph cpu {
graph [throughput="0.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="0"];
from1 [avg_exec_time_ns="0s" ];
from1 -> log2 [processed="0"];
log2 [avg_exec_time_ns="0s" ];
log2 -> alert3 [processed="0"];
alert3 [alerts_triggered="0" avg_exec_time_ns="0s" crits_triggered="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" ];
stats4 [avg_exec_time_ns="0s" ];
stats4 -> derivative5 [processed="2"];
derivative5 [avg_exec_time_ns="0s" ];
derivative5 -> alert6 [processed="1"];
alert6 [alerts_triggered="1" avg_exec_time_ns="0s" crits_triggered="1" infos_triggered="0" oks_triggered="0" warns_triggered="0" ];
}
Here I'm trying to get Kapacitor log anything to STDERR, and if nothing comes in
the stream, then the deadman prints picks up.
I can confirm in InfluxDB there is a metrics table. I do not have a default
retention policy, or one by the name. However, I'm thinking the Databases
Retention Policies specification just means it will use the default retention
policy of the metrics table.
Many logs similar to this:
[httpd] 10.0.0.138 - - [27/Jan/2017:17:09:50 +0000] "POST /write?consistency=&db=metrics&precision=ns&rp=one_year HTTP/1.1" 204 0 "-" "InfluxDBClient" 699b238e-e4b3-11e6-ae28-000000000000 22832
Show up in the logs. I am taking these logs to mean that the InfluxDB
subscription is indeed working.
I have confirmed the InfluxDB subscriptions to the database exist by looking in
the InfluxDB and finding the kapacitor subscriptions, targeted at the correct host.
However, the deadman node kicks off every 10seconds:
[cpu:alert5] 2017/01/27 22:43:10 D! CRITICAL alert triggered id:node 'from1' in task 'cpu' msg:node 'from1' in task 'cpu' is dead: %!f(MISSING) points/10s. data:&{stats map[] [time emitted] [[2017-01-27 22:42:50 +0000 UTC 0]]}
[cpu:alert10] 2017/01/27 22:43:10 D! CRITICAL alert triggered id:node 'stream0' in task 'cpu' msg:node 'stream0' in task 'cpu' is dead: %!f(MISSING) points/10s. data:&{stats map[] [time emitted] [[2017-01-27 22:42:50 +0000 UTC 0]]}
[cpu:log6] 2017/01/27 22:43:10 I! DEADMAN {"Name":"stats","Database":"","RetentionPolicy":"","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":null,"Fields":{"emitted":0},"Time":"2017-01-27T22:42:50Z"}
This data stream is updated in InfluxDB every 10 seconds, and we never see any
logging from the |log().prefix('STANDARD') portion of the debugging TICK file.
If I turn on write tracing, I can see values for CPU to come through:
cpu,cpu=cpu-total,host=argon-odinperf usage_guest=0,usage_guest_nice=0,usage_idle=88.1011465381056,usage_iowait=0.028270771164317015,usage_irq=0,usage_nice=0,usage_softirq=0.21674257892678755,usage_steal=0,usage_system=2.2333909219421186,usage_user=9.420449191315278 1485557630000000000
So I know that I'm getting valid data from Influx.
i updated the kapacitor and influxdb to 1.2. Now it all working
Hi is the 1.2-alpine good image too? Cause I can see that I received something while i do Kapacitor Stats ingress. However, I neither get any alerts nor my kapacitor record stream, records anything. It just stays like its hung without exciting. Any help?
Looks like impossible to download matched versions of telegraf, influxdb and kapacitor, the current available versions are 1.6, 1.5 and 1.4 respectively. Don't know how is the release policy and why they must be consistent but can not be downloaded with same version. I met issue with data points are captured by kapacitor but alert are not sent like this.
+1 Also it's 2019
+1
Most helpful comment
Looks like impossible to download matched versions of telegraf, influxdb and kapacitor, the current available versions are 1.6, 1.5 and 1.4 respectively. Don't know how is the release policy and why they must be consistent but can not be downloaded with same version. I met issue with data points are captured by kapacitor but alert are not sent like this.