kapacitor record stream hangs

Created on 9 May 2016 · 14Comments · Source: influxdata/kapacitor

I am trying to go through the example in Introduction - Getting started, but when I try to run the following command it just hangs:

kapacitor record stream -name cpu_alert -duration 20s

Telegraf seems to be reporting to InfluxDB normally

More details here:

InfluxDB version: 0.12.2
Kapacitor version: 0.12.0 (git: master 32d4d1b8f688e2d310f4e9413b3fe6ac3ef73c70)
Telegraf version: 0.12.1
I used a pre-built package to install all of InfluxDB, Kapacitor & Telegraf.
OS: Centos 6.7
Kapacitor log:

[task_master] 2016/05/09 07:58:29 I! opened
[influxdb] 2016/05/09 07:58:29 I! linking subscriptions
[stats] 2016/05/09 07:58:29 I! opened service
[httpd] 2016/05/09 07:58:29 I! Starting HTTP service
[httpd] 2016/05/09 07:58:29 I! Authentication enabled: false
[httpd] 2016/05/09 07:58:29 I! Listening on HTTP: [::]:9092
[run] 2016/05/09 07:58:29 I! Listening for signals
[httpd] 2016/05/09 08:01:33 I! ::1 - - [09/May/2016:08:01:33 +0000] POST /task?dbrps=%5B%7B%22db%22%3A%22telegraf%22%2C%22rp%22%3A%22default%22%7D%5D&name=cpu_alert&type=stream HTTP/1.1 200 23 - Go-http-client/1.1 3f116d7c-15bc-11e6-8001-000000000000 17.978407ms
[httpd] 2016/05/09 08:01:33 I! ::1 - - [09/May/2016:08:01:33 +0000] GET /task?name=cpu_alert HTTP/1.1 200 310 - Go-http-client/1.1 3f143504-15bc-11e6-8002-000000000000 1.765559ms
[httpd] 2016/05/09 08:01:55 I! ::1 - - [09/May/2016:08:01:55 +0000] POST /record?duration=20s&name=cpu_alert&type=stream HTTP/1.1 200 84 - Go-http-client/1.1 4c219212-15bc-11e6-8003-000000000000 2.790549ms

kapacitor list recordings output:

ID                                      Type    Size      Created
90f5abf4-9c88-479b-82ff-592a613c4880    stream  0 B       09 May 16 08:01 UTC

Source

shanab

Most helpful comment

@jamischarles
Thank you. This is alienlogic; I wouldn't have figured this out on my own.

antonkolb on 6 Feb 2019

👍3

All 14 comments

I have the same issue on a debian machine.

entertainyou on 20 May 2016

Me too, experiencing the same.
Kapacitor: 0.13
InfluxDB: 0.13
OS: Amazon Linux 2015.09

minhdanh on 23 May 2016

Is this the same issue as #148?

nathanielc on 23 May 2016

I don't think so. My problem was that I generated Kapacitor config file using kapacitord config command using root, so the resulting dirs in the config file where pointing at /home/root/.kapacitor/....
And when I tried to run kapacitor as another user, it complained and I didn't notice it.

I ended up using a specific config file instead of generating one, and it worked as expected.

shanab on 24 May 2016

Hi @shanab

Would you mind posting your kapacitor config file ? It would be really helpful for my case

Regards

LaPetiteSouris on 26 May 2016

@LaPetiteSouris sure, here it is:
P.S: You should make sure that /opt/kapacitor is owned by kapacitor:kapacitor.

hostname = "localhost"
data_dir = "/opt/kapacitor/data"

[http]
  bind-address = ":9092"
  auth-enabled = false
  log-enabled = true
  write-tracing = false
  pprof-enabled = false
  https-enabled = false
  https-certificate = "/etc/ssl/kapacitor.pem"
  shutdown-timeout = "10s"

[replay]
  dir = "/opt/kapacitor/replay"

[task]
  dir = "/opt/kapacitor/tasks"
  snapshot-interval = "1m0s"

[[influxdb]]
  enabled = true
  name = "default"
  default = false
  urls = ["http://localhost:8086"]
  username = ""
  password = ""
  ssl-ca = ""
  ssl-cert = ""
  ssl-key = ""
  insecure-skip-verify = false
  timeout = "0"
  udp-buffer = 1000
  udp-read-buffer = 0
  startup-timeout = "5m0s"
  [influxdb.subscriptions]
  [influxdb.excluded-subscriptions]
    _kapacitor = ["default"]
    _internal = ["monitor"]
    telegraf = ["default"]

[logging]
  file = "STDERR"
  level = "INFO"

[collectd]
  enabled = false
  bind-address = ":25826"
  database = "collectd"
  retention-policy = ""
  batch-size = 5000
  batch-pending = 10
  batch-timeout = "10s"
  read-buffer = 0
  typesdb = "/usr/share/collectd/types.db"

[opentsdb]
  enabled = false
  bind-address = ":4242"
  database = "opentsdb"
  retention-policy = ""
  consistency-level = "one"
  tls-enabled = false
  certificate = "/etc/ssl/influxdb.pem"
  batch-size = 1000
  batch-pending = 5
  batch-timeout = "1s"
  log-point-errors = true

[smtp]
  enabled = false
  host = "localhost"
  port = 25
  username = ""
  password = ""
  no-verify = false
  global = false
  state-changes-only = false
  from = ""
  idle-timeout = "30s"

[opsgenie]
  enabled = false
  api-key = ""
  url = "https://api.opsgenie.com/v1/json/alert"
  recovery_url = "https://api.opsgenie.com/v1/json/alert/note"
  global = false

[victorops]
  enabled = false
  api-key = ""
  routing-key = ""
  url = "https://alert.victorops.com/integrations/generic/20131114/alert"
  global = false

[pagerduty]
  enabled = false
  url = "https://events.pagerduty.com/generic/2010-04-15/create_event.json"
  service-key = ""
  global = false

[sensu]
  enabled = false
  addr = ""
  source = "Kapacitor"

[slack]
  enabled = false
  url = ""
  channel = ""
  global = false
  state-changes-only = false

[hipchat]
  enabled = false
  url = ""
  token = ""
  room = ""
  global = false
  state-changes-only = false

[alerta]
  enabled = false
  url = ""
  token = ""
  environment = ""
  origin = ""

[reporting]
  enabled = true
  url = "https://usage.influxdata.com"

[stats]
  enabled = true
  stats-interval = "10s"
  database = "_kapacitor"
  retention-policy = "default"
  timing-sample-rate = 0.1
  timing-movavg-size = 1000

[udf]

[deadman]
  interval = "10s"
  threshold = 0.0
  id = "{{ .Group }}:NODE_NAME for task '{{ .TaskName }}'"
  message = "{{ .ID }} is {{ if eq .Level \"OK\" }}alive{{ else }}dead{{ end }}: {{ index .Fields \"emitted\" | printf \"%0.3f\" }} points/INTERVAL."
  global = false

[talk]
  enabled = false
  url = ""
  author_name = ""

shanab on 30 May 2016

same here, why this issue is closed ? I can't find the solution

lazywhite on 5 Oct 2016

@lazywhite The reason why the recording hangs is because Kapacitor is not receiving any data. There are many reasons why Kapacitor may not be receiving any data, dependent on the environment in which you are running Kapacitor. Can you post to the mailing list with the details of your situation so we can help you out?

The mailing list can be found here [email protected].

nathanielc on 5 Oct 2016

I was just bitten by this.
kapacitor record stream -task restart_alert7 -duration 60s

I assumed erroneously that I could manually submit a few data points to my influx db (my stream) and this recording would stop after 60s.

It appears the task will ONLY stop itself after it receives data AFTER the 60s has elapsed. So I had to send some data after, and then the recording completed.

Clearly the evaluation of the recording is triggered by data streaming in.

jamischarles on 28 Nov 2017

👍3

@jamischarles
I have the same issue I'm trying to insert some data manually what do you mean by the following sentence :

It appears the task will ONLY stop itself after it receives data AFTER the 60s has elapsed. So I had to send some data after, and then the recording completed.

I can not get it

zmellal on 3 Dec 2017

👍2

@zmellal You have to insert some data after the time limit has elapsed. This will run the logic that checks if the time has expired, then it'll end the recording.

jamischarles on 3 Dec 2017

👍3

@jamischarles
Thank you. This is alienlogic; I wouldn't have figured this out on my own.

antonkolb on 6 Feb 2019

👍3

Still does not work for me, Ubuntu 18