Telegraf: Make timestamp configurable for JSON parser

[bbczeuz]

Feature Request

Proposal:

• Add two configuration options to the JSON parser (e.g. socket_listener input plugin) that allow the timestamp to be collected from the data itself. e.g:

time_key = "time"
time_unit = "us"

Current Behavior:

• The timestamp in the JSON parser is currently being set by the parser (https://github.com/influxdata/telegraf/blob/master/plugins/parsers/json/parser.go#L61)
• With some hack in Kapacitor we might be able to use the time field as timestamp...

Desired Behavior:

• Keep the original timestamp of the data

Use Case:

• I plan to use telegraf to capture Juniper Network Analytics data. Supporting devices send interface/buffer metrics in JSON or GPB via UDP or TCP to remote devices.
• This would replace snmp polling reducing CPU load and latency
• The existing socket_listener input plugin supports the JSON format and tag_keys to process most data
• The original JSON data contain a time field with the timestamp of the measurement in microseconds since the beginning of the current Epoch (http://www.juniper.net/documentation/en_US/junos/information-products/pathway-pages/ex4300/network-management.pdf)

Example Data

{"record-type":"traffic-stats","time":1499268651471993,"router-id":"qfx5100-01","port":"et-0/0/50","rxpkt":82227,"rxucpkt":74232,"rxmcpkt":7985,"rxbcpkt":10,"rxpps":11,"rxbyte":6254098,"rxbps":55232,"rxdroppkt":0,"rxcrcerr":0,"txpkt":104340,"txucpkt":96366,"txmcpkt":7955,"txbcpkt":19,"txpps":13,"txbyte":9313288,"txbps":71552,"txdroppkt":0,"txcrcerr":0}

Proposed Example Configuration

[[inputs.socket_listener]]
    service_address = "udp://:50019"
    data_format = "json"
    tag_keys = [
        "router-id",
        "port",
        "record-type"
    time_key = "time"
    time_unit = "us"

[danielnelson]

Lets call it json_timestamp_units and json_timestamp_key to match existing options in the json serializer.

[srclosson]

+1, this would be very helpful for me, trying to access OPC data through a restful api.

[petetnt]

This issue proved to be a blocker for our use case of Telegraf, would you be interested in a PR?

[danielnelson]

@petetnt Absolutely

[suiluj]

hello i would be very interested in the timestamp feature, too.
any updates or solutions on this topic?

[maxunt]

An updated json parser will have an optional 'json_time_key' and a 'json_time_format' config. This will extract one timestamp per metric from the specified key, so if your data has an array of objects you can extract multiple different timestamps. Would this suit the above use cases?

[pmouawad]

Hello,
Can you confirm current below documentation:
https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md#json

References new features like json_time_key and json_time_format that are not yet available as of version 1.7.4 , and that will be available in 1.8.0 ?

Is release date expected on 21 september ?

Thanks for your help

[danielnelson]

@pmouawad I recently updated the JSON documentation, these options are documented but can you take a look and let me know if you have any feedback.

Any documentation links to the master branch are for the latest development version, so you may want to check the release branch for release specific docs:

https://github.com/influxdata/telegraf/tree/release-1.8/plugins/parsers/json

[littlespace]

@bbczeuz were you able to get this working with the UDP streaming? I am trying to do the same, would you mind share your experience?

Thanks

[bbczeuz]

@littlespace I didn't work with that feature for a long time and am ususally offline these days, so you might have to test it yourself