Telegraf: Multiline support in the log parser (similar to LogStash and FluentD)

[DmitryKiselev]

Feature Request

Proposal: Enable the Telegraf to span regex patterns beyond one text line.

In other words the regex should be able to continue beyond the line breaks until it finds a predefined "line end" (or "new log line") regex pattern.
Note: the grok parser and the the log tail implementation currently used, are capable of that).

Current behavior:

Currently Telegraf can only apply regex patterns inside each text line in a log file.

Desired behavior:

To support an option where the log is broken into individual records fully by virtue of regex and without taking into account the line breaks (like in Log Stash and FluentD).

Use case:

Then the Telegraf can be freely used as a replacement for FluentD and/or LogStash when preferred.
That is especially desirable when InfluxDB is used as a consolidated log storage.

[mattpoel]

Would love to see this feature! I am parsing the logfiles of an application, which is using multiline messages quite a lot.

Example with multiline messages:

20151129 09:00:18.990 TEST    Test::Testing                  Text begins here
additional text
even more text

text after an empty line
20151129 09:00:18.991 TEST    Test::Testing                  Next log file entry

Unfortunately execution times for operations are even written to two different log lines:

20151129 09:00:18.990 BEGIN   Test::Testing                  Performing operation XY
which is described over
multiple lines

and even with empty lines in between
20151129 09:00:19.181 END     Test::Testing                  190ms LONG

It would be good if patterns could be defined to store multiline messages (example 1) and to be able to merge two lines to parse the execution time information for an operation as shown in example 2.

[botzill]

What is the status of this feature? Is there any way to parse multiline logs ?

Thx!

[chenhangbiao]

What is the status of this feature? Is there any way to parse multiline logs ?

Thanks very much!

[glinton]

I believe more conversation is in the related issue #3228

[Phukety]

What is the status of this feature? Is there any way to parse multiline logs ?

Thanks very much!

[ssoroka]

This is now supported in inputs.tail in and will be officially released in 1.16. Note that logparser is deprecated and the recommendation is to use inputs.tail.