Kapacitor: Kapacitor config to support multiple InfluxDB config

[jayannah]

Influx Config:

sender ---> InfluxRelay (hostname: influxwrite) --> Multiple InfluxDBs (for HA)
Reader --> LoadBalancer (hostname: influxread) --> one of the InfluxDB

So for seding data to InfluxDB, we use http://influxwrite:8086 (replay server) & for reading the data from InfluxDB, we use http://influxread:8086

Requirement: Use Kapacitor to read data, do stats and transformation, send back the data to InfluxDB if particular field value doesnt met the SLA. Basically writing the alert details back to InfluxDB along with sending to other message services like slack

At the bottom, you can see the kapacitor.conf file we have for influxdb.

Problem:
The kapacitor.conf has configuration to specify only one Influx DB instance details under [[influxdb]]. So the kapacitor uses the InfluxInstances mentioned under [[influxdb]] configs as default influx instance and connects while bootup or queries the data whatever we specify in the TICK script.

To send data back to InfluxDB, we have to use InfluxDBOut() and kapacitor will try to send it to default InfluxDB instance specified in [[influx]], which is NOT i want as the influxread:8000 is for reading the data, not for write since we have write to Relay server so that the data is written all the InfluxDB instances configured to receive from Relay server.

Feature request: Kapacitor configuration must allow users to define multiple influxDB instance configuration. Whatever defined under [[influxdb]] can be default to connect to and others should be able in InfluxDBOut.

E.g:
[[influxdb]]
urls = ["http://influxread:8086"]
..
..

[[influxdb_replay]]
url = ["http://influxwrite:8086"]
...

So in the InfluxDBOut() I should be able to use the InfluxDB instance configuration mentioned in the [[influxdb_aggregated_instance]] section.

Kapacitor can still connect to the default [[influxdb]] instance when bootup and querying the data.

kapacitor.conf

[[influxdb]]
  enabled = true
  name = "kapacitor"
  default = true
  urls = ["http://influxread:8086"]
  username = "admin"
  password = "admin"
  ssl-ca = ""
  ssl-cert = ""
  ssl-key = ""
  insecure-skip-verify = false
  timeout = "0"
  disable-subscriptions = false
  udp-buffer = 1000
  udp-read-buffer = 0
  startup-timeout = "30s"
  [influxdb.subscriptions]
  [influxdb.excluded-subscriptions]


please let me know for any clarifications.

[rossmcdonald]

The double brackets in the configuration ([[influxdb]], for example) means it can be specified multiple times. So to use Kapacitor with multiple InfluxDB instances, you'll need to create multiple [[influxdb]] sections. For example:

[[influxdb]]
  enabled = true
  name = "read"
  default = true
  urls = ["http://influxread:8086"]

[[influxdb]]
  enabled = true
  name = "write"
  default = false
  disable-subscriptions = true
  urls = ["http://influxwrite:8086"]

Note:

• The different name values
• The read InfluxDB instance is set to default = true, so it will be the default query endpoint
• The write InfluxDB instance has disable-subscriptions = true, which means that subscription data won't be forwarded to Kapacitor as data is written to it. This will help ensure that you're not receiving the data back from the write instance after you write to it.

And then to write to the write instance, specify it as the cluster:

|influxDBOut()
.cluster('write')
....