Telegraf: Support MQTT templates for Telegraf input/output

[sparrc]

Looking for input on an MQTT "Templating" system similar to the InfluxDB graphite listener, where users can specify fields, measurements, and tags to turn into a topic.

For MQTT Output from telegraf, you could have an InfluxDB-style metric like this:

cpu,cpuX=cpu0,host=foo.com usage_idle=90

and a template:

telegraf/cpuX/host/measurement/field

you would get an MQTT output topic & payload like this:

telegraf/cpu0/foo.com/cpu/usage_idle 90

Similarly, for an MQTT input, you could template MQTT subscriptions into InfluxDB measurements (_measurement_ and _field_ are special keys, the rest are tags):

templated_topics = [
  "sensors/house/+/+        measurement/building/room/field"
]

which would translate the following MQTT values:

sensors/house/livingroom/temperature   19
sensors/house/diningroom/temperature   20
sensors/house/bedroom/temperature      17
sensors/house/livingroom/humidity      20

into influxdb metrics:

sensors,building=house,room=livingroom temperature=19,humidity=20
sensors,building=house,room=diningroom temperature=20
sensors,building=house,room=bedroom temperature=17

[nhimf]

I very much like the template idea, makes us a lot more flexible in how we implement our sensors.

What I do like to see is the ability to add some tags or other metadata, e.g. timestamps, value type. For some applications it's best to let the sensor send out the timestamp instead of the server (broker, InfluxDb or Telegraf) doing so.
While I am not deeply familiar with InfluxDB and Telegraf as of yet, I am not sure if there is a way to mark data as relative to the previous value instead of being absolute values.

Adding e.g. timestamps could be like this:

sensors/house/livingroom/temperature   19,time=1454657463,<other tags/meta-data>,....

[sparrc]

@nhimf Depends on the format of your data, for tags, there will definitely be a way to define additional tags within the template.

For timestamps and other data, I think the option would be to just format the message payload in the influxdb line-protocol, so then your sensors topic (or any data) would have messages like this:

sensors,building=house,room=living,foo=bar,mykey=myval temperature=19,humidity=10 1454657463

and these line-protocol messages would get parsed directly into Telegraf/InfluxDB

you can read more details about line-protocol here: https://docs.influxdata.com/influxdb/v0.10/write_protocols/line/

[nhimf]

The downside of using the line protocol in MQTT is that if you, for some reason, want to use the same topics with something other than influx you have to parse the Influx line protocol. One of the major reasons to use MQTT instead of directly pushing stuff into influx is I can use MQTT as an abstraction layer. Using the line protocol format kinda breaks this idea.

[sparrc]

but what is the trade-off of using line-protocol vs. the example you provided earlier? They look fairly similar to me. Allowing arbitrary data formats is a bit of a slippery slope, I'd prefer to stick with Graphite, Influx, and MQTT topics for now.

[nhimf]

E.g. if line protocol changes, I do not have to change my sensors. Also, not sure if it will work this way, the Telegraf template could ignore extra data which is not defined in the template.
So I can put extra data in the messages and use it elsewhere.
My personal idea is that, unless you do a direct transfer, transfers should be generic as possible up to where it is reasonable to do so

[sparrc]

@nhimf The problem is that there's an infinite number of formats to support, coming up with a generic templating system that works for literally _everything_ is not a trivial task...

[sparrc]

One option that could simplify things would be to break out the 4 main component parts of a telegraf Metric: measurement, fields, tags, and timestamp, and then a template could simply be a combination of those 4 components, so this template:

{{fields}}|{{timestamp}}|{{tags}}|{{measurement}}

which would accept metrics that look like:

temperature=17|1454657463|building=house,room=bedroom|sensors

What do you think of that?

[nhimf]

Seems fair enough, this way there is no direct lock-in for the line protocol but still quite similar and thus much easier to implement, but if something in the line protocol changes there is no need to change the MQTT topics. (Although the part of a possible change in the line protocol might be more of an academic discussion :-) )

[spidie]

Hi all, been 2.5 years since this was originally proposed and it seems like an awesome idea to me.

My current ugly workaround for this is that I write a small program that extracts fields from the topic and posts them on a new "unpacked" topic with the payload as json. I can then use the datatype json and the "tag_keys" to add specific json fields as tags in influxdb.

I have a couple of devices with heavily byte packed payloads and info in the topic (written for very low bandwidth) which I originally used this technique for - but works just as well for regular messages that use the topic for data too.

[CaffeinatedAndroid]

Just flagging that I was looking for this feature and would be great for reducing payload sizes.

I'm currently sending batches of data as a JSONArray, with device ID as part of the topic/url. As topic parsing is not implemented I have to have the device ID duplicated in each message. This would be fine if easch message was sent individually, but when batching them together in an array, it just leads to lots of duplicate data.

[danielnelson]

You might be able to do something similar these days using the parser processor and data_format = "grok". First collect the topic as a tag, then use the converter processor to convert the tag to a string field, finally use the grok parser to create new tags from the topic.