Kapacitor: Startup fails when subscription already exists

[cheribral]

While trying to test UDF code, I am constantly starting and stopping kapacitor, and quite often it is crashing. Very often when I restart it, I see:
run: open server: open service *influxdb.Service: subscription already exists

I then need to keep a console open on the influx server to delete the subcription before starting kapacitor.

Should it simply start listening on the correct port if the subscription is in place?

[nathanielc]

@cheribral Could you provide your startup log with debug level logs when it fails?

Subscriptions can exist before startup so something odd is going on.

[cheribral]

@nathanielc Here is what I have after an UDF function malfunction causes kapacitor to stop, with some lines from the crash included for context.

[task_store] 2016/07/13 02:21:53 E! task cpu finished with error: error restoring snapshot: 'state' object has no attribute 'modelParams'
[task_store] 2016/07/13 02:21:53 E! failed to save last error for task cpu
[httpd] 2016/07/13 02:21:53 I! Closed HTTP service
[srv] 2016/07/13 02:21:53 D! closed service: *httpd.Service
[edge:TASK_MASTER|write_points->stream] 2016/07/13 02:21:53 D! closing c: 14276 e: 14276
[task_master] 2016/07/13 02:21:53 I! closed
[run] 2016/07/13 02:21:53 I! server shutdown completed
[run] 2016/07/13 02:22:21 I! Kapacitor starting, version 0.13.0, branch master, commit ce28f6dced073e473b44f592517fa6c4d05b5e65
[run] 2016/07/13 02:22:21 I! Go version go1.6.2, GOMAXPROCS set to 4
[srv] 2016/07/13 02:22:21 I! Kapacitor hostname: laws-nag.telogis.local
[srv] 2016/07/13 02:22:21 I! ClusterID: 17b0b2de-49fd-4e4d-8765-99ebd2e56119 ServerID: 55b6633a-34b8-48a0-bf74-6db56ad12cbe
[task_master] 2016/07/13 02:22:21 I! opened
[srv] 2016/07/13 02:22:21 D! opening service: *udf.Service
[udf] 2016/07/13 02:22:22 I!P 2016-07-13 02:22:22,054 INFO:root: Starting Agent
[udf] 2016/07/13 02:22:22 I!P 2016-07-13 02:22:22,056 INFO:root: Agent finished
[udf] 2016/07/13 02:22:22 D! loaded UDF info "anomaly"
[srv] 2016/07/13 02:22:22 D! opened service: *udf.Service
[srv] 2016/07/13 02:22:22 D! opening service: *deadman.Service
[deadman] 2016/07/13 02:22:22 I! Deadman's switch is configured globally
[srv] 2016/07/13 02:22:22 D! opened service: *deadman.Service
[srv] 2016/07/13 02:22:22 D! opening service: *smtp.Service
[smtp] 2016/07/13 02:22:22 I! Starting SMTP service
[srv] 2016/07/13 02:22:22 D! opened service: *smtp.Service
[srv] 2016/07/13 02:22:22 D! opening service: *influxdb.Service
[influxdb] 2016/07/13 02:22:22 I! linking subscriptions
[udp:metrics.default] 2016/07/13 02:22:22 I! Started listening on UDP: [::]:57494
[influxdb] 2016/07/13 02:22:22 I! started UDP listener for metrics default
[srv] 2016/07/13 02:22:22 D! closing service: *udf.Service
[srv] 2016/07/13 02:22:22 D! closed service: *udf.Service
[srv] 2016/07/13 02:22:22 D! closing service: *deadman.Service
[srv] 2016/07/13 02:22:22 D! closed service: *deadman.Service
[srv] 2016/07/13 02:22:22 D! closing service: *smtp.Service
[smtp] 2016/07/13 02:22:22 I! Closing SMTP service
[srv] 2016/07/13 02:22:22 D! closed service: *smtp.Service
[srv] 2016/07/13 02:22:22 D! closing service: *influxdb.Service
[udp:metrics.default] 2016/07/13 02:22:22 I! Service closed
[srv] 2016/07/13 02:22:22 D! closed service: *influxdb.Service
[srv] 2016/07/13 02:22:22 D! closing service: *storage.Service
[srv] 2016/07/13 02:22:22 D! closed service: *storage.Service
[srv] 2016/07/13 02:22:22 D! closing service: *task_store.Service
[srv] 2016/07/13 02:22:22 D! closed service: *task_store.Service
[srv] 2016/07/13 02:22:22 D! closing service: *replay.Service
[srv] 2016/07/13 02:22:22 D! closed service: *replay.Service
[srv] 2016/07/13 02:22:22 D! closing service: *alerta.Service
[srv] 2016/07/13 02:22:22 D! closed service: *alerta.Service
[srv] 2016/07/13 02:22:22 D! closing service: *stats.Service
[srv] 2016/07/13 02:22:22 E! error closing service *stats.Service: error closing stats service: service not open
[srv] 2016/07/13 02:22:22 D! closed service: *stats.Service
[srv] 2016/07/13 02:22:22 D! closing service: *httpd.Service
[httpd] 2016/07/13 02:22:22 I! Closed HTTP service
[srv] 2016/07/13 02:22:22 D! closed service: *httpd.Service
[edge:TASK_MASTER|write_points->stream] 2016/07/13 02:22:22 D! closing c: 0 e: 0
[task_master] 2016/07/13 02:22:22 I! closed
[run] 2016/07/13 02:22:22 E! open server: open service *influxdb.Service: subscription already exists

[simonsdeng]

This crashes for me on the latest stable v0.13.1 but not on the latest unstable v1.0.0-beta4.

[thecodeassassin]

Still happens for me in the latest beta

[simonsdeng]

Nevermind, you're right, it still happens for me in the beta too, but I think it works in the latest git version.

[thecodeassassin]

@simonsdeng are you by any chance also using a loadbalancer/HA configuration?

If you are, add ip_hash to the upstream server list (or stickiness if you are using another loadbalancer) then it works :)

[simonsdeng]

@thecodeassassin No, not right now, but that is something I was planning on looking into. Thanks!

[nathanielc]

This is fixed in #811