Logstash: Time difference calculation in logstash

[purbon]

In the scenario where you can to calculate a difference between two timestamps in an event, for example to calculate the elapsed time, there are no filters, or options within LogStash to do it, except of using a very simple ruby filter.

For example:

ruby {
        code => "event['elapsed'] = (event['delta']-event['@timestamp'])"
}

As this is expected to be a common operation, it would be easier to have this integrated within the flow. There are some options to implement this:

• As a new filter.
• Within the mutate filter, as a new option.
• Within the filter base, so we can use it within add_field option present in every filter.

For example:

  mutate {
    add_field => {   elapsed => "%{[@timestamp]-[end]}"   }
  }

So what option is the one you like the most?

[purbon]

I think this module could be extended to do some internal document aggregations, if they not really going to change from time to time, it makes not much sense to be done inside the database, so having a something on the Logstash level would be great.

[JPvRiel]

Bumped my head into this as well. Had not so much fun when copying fields using add_field and unwittingly converting the time object to a string. Since I'm not a ruby dev, just a person trying to parse logs and calculate new fields, I ended up with exceptions like: "TypeError: can't convert Timestamp into Rational", despite trying to parse a timestamp with the ruby Time.parse function. After a lot of hacking around, converting Time to floats, etc, I got some working examples. The logstash time-test.conf I wrote helps demonstrates the pain.

input { 
  generator {
    message => "2003-10-11T22:14:15.003Z"
    count => 1
  }
}

filter {
  # copy logstash timestamp
  ruby {
    code => "event['timestamp_logstash_ruby'] = event['@timestamp']"
  }
  mutate {
    add_field => { "timestamp_logstash_add_field" => "%{@timestamp}" }
  }
  # get source timestamp and copy it into @timestamp
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp_source}" }
  }
  date {
    match => [ "timestamp_source", "ISO8601" ]
  }
  # add field for time elapsed
  ruby {
    code => "event['time_elapsed_ruby'] = event['timestamp_logstash_ruby'] - event['@timestamp']"
  }
  ruby {
    code => "event['time_elapsed_add_field'] = Time.parse(event['timestamp_logstash_add_field']).to_f - event['@timestamp'].to_f"
    # above results in "exception"=>#<TypeError: can't convert Timestamp into Rational" if not converted to floats
  }
}

output { stdout { codec => rubydebug } }

[timbunce]

I hit the "TypeError: can't convert Timestamp into Rational" in a ruby filter block.
This change to my code fixed it for me:

-            msg_age = Time.now - event['@timestamp']
+            msg_age = Time.now.to_i - event['@timestamp'].to_i

[untergeek]

I believe this functionality is now covered by the elapsed plugin. If you believe this is in error, please feel free to re-open the ticket.

[anhlqn]

@untergeek I believe the elapsed plugin does not do what this issue describe. Elapsed plugin tracks two separate messages bound by an unique id field then calculates the elapsed time between the two messages. On the other hand, this scenario does a subtract between two timestamp fields in a single message. For numbers, I usually use scripted fields in Kibana to do such thing, but it would be great if the mutate filter offers similar feature.

[gmoskovicz]

Using the ruby filter while works is not a great experience. I would recommend to add a way to check this with conditionals.

[guyboertje]

@gmoskovicz
The math filter is nearly ready - it will do timestamp arithmetic.
PR https://github.com/logstash-plugins/logstash-filter-math/pull/5