Telegraf: Send metrics to specific outputs

[s1m0]

It would be very useful to be able to output exec inputs to different Influxdb databases with differing retention policies and precisions. e.g.

[[inputs.exec]]
  command = "/usr/local/bin/my5minutescript.sh"
  interval = "5m"  
  data_format = "influx"
  influx_db = "database1"
  influx_retention = "default"
  influx_precision = "s"

[[inputs.exec]]
  command = "/usr/local/bin/my5secondcript.sh"
  interval = "5s"
  data_format = "influx"
  influx_db = "database2"
  influx_retention = "5days"
  influx_precision = "ns"

[titilambert]

And what about to be able to select which input send data to which outputs ?

[[inputs.exec]]
  command = "/usr/local/bin/my5minutescript.sh"
  interval = "5m"  
  data_format = "influx"
  # if outputs is empty, then send to all outputs
  outputs = ["influxdb1"]

[[inputs.exec]]
  command = "/usr/local/bin/my5secondcript.sh"
  interval = "5s"
  data_format = "influx"
  # if outputs is empty, then send to all outputs
  outputs = ["influxdb2"]

[[outputs.influxdb]]
  name = "influxdb1"
   urls = ["http://localhost:8086"]
  database = "telegraf"
  retention_policy = "default"
  precision = "s"

[[outputs.influxdb]]
  name = "influxdb2"
   urls = ["http://localhost:8086"]
  database = "telegraf2"
  retention_policy = "5days"
  precision = "ns"

[s1m0]

That approach would hopefully be applicable to all types of inputs and outputs. It would be a breaking change but well worth the disruption to make telegraf a lot more flexible, allowing you to 'wire-up' inputs to outputs, rather like Node-Red does visually.

[sparrc]

Sending inputs to specific outputs sounds like a useful feature, but there is a way of doing this already using measurement name filtering:

[[inputs.exec]]
  command = "/usr/local/bin/my5minutescript.sh"
  interval = "5m"  
  data_format = "influx"
  name_suffix = "_default"

[[inputs.exec]]
  command = "/usr/local/bin/my5secondcript.sh"
  interval = "5s"
  data_format = "influx"
  name_suffix = "_5days"

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "telegraf"
  retention_policy = "default"
  precision = "s"
  namepass = ["*_default"]

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "telegraf2"
  retention_policy = "5days"
  precision = "ns"
  namepass = ["*_5days"]

[titilambert]

@sparrc this is really nice :)

[sparrc]

note that you could also do this using custom tags and tagpass/tagdrop

[sparrc]

I'm going to close this because using namepass/tagpass is sufficient to accomplish this

[s1m0]

So if my measurement names are unique, I can use namepass on the influxdb output instance I want to use to get it to write the line to influxdb?
If I have one script that generates 2 measurements that go into different databases, will telegraf be intelligent enough to route them to different output instances?

E.g. My script produces lines like:-

Name1,type=ATP value1=34.56 124567890
Name2,type=abcd123 value1=123456 1234567890

My outputs could be configured like this:-

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "database1"
  retention_policy = "5days"
  precision = "s"
  Namepass = ["Name1"]

[[outputs.influxdb]]
  urls = ["http://12.34.56.78:8086"]
  database = "database2"
  retention_policy = "6months" 
  precision = "ms"
  namepass = ["Name2","Name3","Name4"]

[sparrc]

yes, exactly

[s1m0]

First problem:

~$ telegraf -config /etc/telegraf/telegraf.conf -debug
2016/03/12 02:31:18 line 84: field corresponding to `retention_policy' is not defined in `*influxdb.InfluxDB'
~$ telegraf -version
Telegraf - Version 0.10.4.1

Also, the time precision isn't being respected - as per my previous issue #761 , whatever the precision specified, it always uses "ns".

However, the good news is that the measurements are being written to the right databases albeit with the default rp and ns precision.

[sparrc]

Retention policy isnt available until 0.11, if you have an issue its best to open an issue rather than commenting on a closed issue

[chenl87]

Hi Guys,

Any chance you can consider reopening this issue?
I understand that "routing" the different inputs to the different outputs can be achieved using namepass/tagpass, but it would be a lot more intuitive using some sort of naming as @titilambert suggested (I would even add the feature of passing the names to the '--output-filter' argument in the cli) , plus you don't have to add unnecessary suffixes to your measurements.

Hope it makes sense..
Thanks!

[danielnelson]

@chenl87 There is a similar issue open with a comment that shows a way to accomplish this sort of routing https://github.com/influxdata/telegraf/issues/1778#issuecomment-247605473