Logstash: Multiline: message stays in buffer until new log entry

[wvdongen]

Hello,

Currently when using the multiline codec a message will only be outputted when Logstash is able to determine the end of the log entry. The end of a multiline log message can only be determined when a new one comes in. When watching a log that doesn't receive many log entries this is annoying. It would be nice if it was possible to set somekind of timeout on the multiline functionality so the the buffer is outputted before the next log message comes in.

[runningman84]

This seems to work if you use multiline as an input codec.

[wvdongen]

What do you mean? The above behavior happens when using the multiline input codec.

[xkr47]

This issue has been lurking over at Jira for quite some time now.. https://logstash.jira.com/browse/LOGSTASH-202

I feel tempted to start implementing something along the lines of @jordansissel's suggestion on 03/Dec/12 12:30 PM in Jira.. :)

[xkr47]

Perhaps #1545 is related too?

[colinsurprenant]

yes, #1545 will solve the filter flushing and currently there is no timeout based flushing in the codecs.

[suyograo]

Closing this. #1545 is merged in

[ChrisMagnuson]

In case anyone else finds this and is trying to get the multiline codec to output the last log record that it has in its buffer after a timeout, the parameter you want is auto_flush_interval which was introduced with Logstash 2.2.

The value is specified in seconds you want to wait before the codec flushes whatever it has into a new event.

Here is an example from a windows based logstash configuration:

input {
    file {
        path => "$LogstashFilePathValue"
        type => "DemandwareError"
        tags => "$EnvironmentName"
        start_position => "beginning"
        sincedb_path => "NUL"
        codec => multiline {
            pattern => "\A\[%{TIMESTAMP_ISO8601:demandware_timestamp} GMT\]"
            negate => true
            what => previous
            auto_flush_interval => 10
        }
    }
}