Telegraf: Logstash integration

[sparrc]

Looking for community input.

We'd like Telegraf to have some integration with logstash. Not being a user of logstash myself, what should this look like? An output that can send arbitrary telegraf data to logstash, or a plugin that could consume logstash output and forward to InfluxDB?

[jrxFive]

I'm a pretty heavy logstash user. I think it would be cool to do both if possible but I favor a pure output. Since logstash is a great router to different services, it could in itself send to InfluxDB as logstash output.

It could be alot like the Kafka/NSQ output. With it being formatted into the logstash protocol. Since we already using tags, that can be an exact mapping for logstash. Type configured from the configuration file, defaulting to telegraf. I'm unsure if anyone has created a codec in logstash for the new line protocol yet.

Additionally we do have outputs that logstash uses as inputs very heavily amqp,kafka,redis. It could just be as simple as adding the logstash protocol option to those. Its not very often you will directly send to logstash without a broker in the middle.

[jrxFive]

@sparrc, any suggestion on which direction? I probably have some time to start this.

[sparrc]

@jrxFive Sorry I missed this comment. I think that supporting the logstash encoding would be good. I was recently thinking about ways to do this, one option would be to have "protocol" plugins that could parse the client.Point object to various different protocols, such as logstash. Currently we pretty much just do line-protocol, JSON would be easy and an obvious one to support as well.

So let me think on how that would look architecturally, it would be nice if users could plug different protocols into input and output plugins adhoc.

[sslupsky]

The current output in logstash does not work well. We have experienced issues where influx drops data because the logstash output does not backoff when influx is busy. I am assuming Telegraf knows to do so? So a plugin that consumes logstash output would be nice.

[zstyblik]

@sslupsky what do you mean by "busy"?

[sslupsky]

@zstyblik When Influx cannot accept data, it throws a 500 error. When it does that you need to back off for a while. Unfortunately, the influx output for logstash does not back off and keeps shoving data at influx.

My developer mentioned that if the influx plugin for logstash was a codec, then he thought maybe you could use the logstash http output to send data to influx instead.

[zstyblik]

@sslupsky I see and thank you for explanation. As for http plugin in logstash, it doesn't allow batching as far as my experience goes :-s

[deanefrati]

@sparrc another idea for logstash integration could be to integrate with elastic's beates framework. It's also written in go so the integration may not be that difficult. The beats framework knows how to ship data to various outputs including logstash and elasticsearch and does some nice things like reducing the send frequency when logstash/elasticsearch are "busy". We would be very interested in a logstash output plugin.

[elvarb]

Having Kafka as a middle man for both Input and Output plugins in Logstash would be in my opinion the best way. Solves all possible performance problems.

Another use case for a Logstash integration would be a InfluxDB filter. That way you could configure Logstash to query InfluxDB when a certain event comes in. For example if an error message comes in it could query InfluxDB for current active sessions, average sessions for the past 10 minutes and for the past 60 minutes.

[alimousazy]

@sparrc
Is this will handle aggregation also, Please take a look at this https://github.com/influxdata/telegraf/issues/1349 ?

[sparrc]

aggregation will not be specific to any one plugin, see #380

[robinsmidsrod]

If you want specific events from logstash into InfluxDB then you can also use this output configuration in Logstash:

# Output HTTP access log info to Telegraf TCP listener which ends up in InfluxDB
if "http-access" in [type] {
    tcp {
        host => "127.0.0.1"
        port => 8094
        codec => line {
            format => "http_access,program=%{program},host=%{host},vhost=%{vhost},port=%{port},status=%{status},scheme=%{scheme},method=%{method},severity=%{severity} bytes=%{bytes}i,duration=%{duration},count=1i %{@timestamp_ns}"
        }
    }
}

You'll need this in your filter section as well:

if [@timestamp] {
    ruby {
        # event.get('@timestamp').time is an object of class 'Time'
        # See http://ruby-doc.org/core-2.2.0/Time.html for details
        # Convert to rational number (fraction) and multiple with 1e9, store as integer
        code => "event.set('@timestamp_ns', ( event.get('@timestamp').time.to_r * 1000000000 ).to_i )"
    }
}