Fluent-bit: Option "Parser" is not ignored when "Multiline" is ON

[JavaCS3]

Bug Report

Describe the bug

config.conf

[SERVICE]
    Parsers_File    parsers.conf

[INPUT]
    Name              tail
    Path              ./demo.log
    Parser            demo
    Multiline         On
    Parser_Firstline  demo
    Parser_1          demo_1

[OUTPUT]
    Name   stdout
    Match  *

parsers.conf

[PARSER]
    Name        demo
    Format      regex
    Regex       ^\d+\s(?<level>\w{3})\s(?<time>\w{3}\s\d+\s\d+:\d+:\d+\.\d+)\s(?<message>.*)
    Time_Key    time
    Time_Format %b %d %H:%M:%S.%L
    Time_Keep   On

[PARSER]
    Name        demo_1
    Format      regex
    Regex       ^\s+.*

demo.log

1442 INF Jul 28 17:30:01.035659 AAAAAAA
1444 NOT Jul 28 17:30:01.263385 BBBBBBB
    MULTILINE
1445 NOT Jul 28 17:30:01.263741 CCCCCCC
    MULTILINE
1446 NOT Jul 28 17:30:01.786663 DDDDDDD
    MULTILINE
1447 NOT Jul 28 17:30:02.199434 EEEEEEE
1448 NOT Jul 28 17:30:02.628430 FFFFFFF
1449 NOT Jul 28 17:30:03.020809 GGGGGGG
1450 NOT Jul 28 17:30:03.423898 HHHHHHH
1452 NOT Jul 28 17:30:05.307595 IIIIIII
1451 NOT Jul 28 17:30:03.795889 JJJJJJJ

To Reproduce

• run fluent-bit -c config.conf

Fluent-Bit v0.13.5
Copyright (C) Treasure Data

[2018/07/29 06:57:03] [ info] [engine] started (pid=251)
[2018/07/29 06:57:03] [ warn] [in_tail] the 'Parser demo' config is omitted in Multiline mode
"}] tail.0: [1532799001.035659000, {"level"=>"INF", "time"=>"Jul 28 17:30:01.035659", "message"=>"AAAAAAA
"}] tail.0: [1532799001.263384999, {"level"=>"NOT", "time"=>"Jul 28 17:30:01.263385", "message"=>"BBBBBBB
"}] tail.0: [1532847423.102942400, {"log"=>"    MULTILINE
"}] tail.0: [1532799001.263741000, {"level"=>"NOT", "time"=>"Jul 28 17:30:01.263741", "message"=>"CCCCCCC
"}] tail.0: [1532847423.102952400, {"log"=>"    MULTILINE
"}] tail.0: [1532799001.786663000, {"level"=>"NOT", "time"=>"Jul 28 17:30:01.786663", "message"=>"DDDDDDD
"}] tail.0: [1532847423.102960600, {"log"=>"    MULTILINE
"}] tail.0: [1532799002.199434000, {"level"=>"NOT", "time"=>"Jul 28 17:30:02.199434", "message"=>"EEEEEEE
"}] tail.0: [1532799002.628430000, {"level"=>"NOT", "time"=>"Jul 28 17:30:02.628430", "message"=>"FFFFFFF
"}] tail.0: [1532799003.020809000, {"level"=>"NOT", "time"=>"Jul 28 17:30:03.020809", "message"=>"GGGGGGG
"}]] tail.0: [1532799003.423898000, {"level"=>"NOT", "time"=>"Jul 28 17:30:03.423898", "message"=>"HHHHHHH
"}]] tail.0: [1532799005.307595000, {"level"=>"NOT", "time"=>"Jul 28 17:30:05.307595", "message"=>"IIIIIII
"}]] tail.0: [1532799003.795889000, {"level"=>"NOT", "time"=>"Jul 28 17:30:03.795889", "message"=>"JJJJJJJ

The MULTILINE log is not appended to the previous log. But if you remove the lineParser demo it will work as expected.

luent-Bit v0.13.5
Copyright (C) Treasure Data

[2018/07/29 07:00:06] [ info] [engine] started (pid=253)
"}] tail.0: [1532799001.035659000, {"level"=>"INF", "time"=>"Jul 28 17:30:01.035659", "message"=>"AAAAAAA
[1] tail.0: [1532799001.263384999, {"level"=>"NOT", "time"=>"Jul 28 17:30:01.263385", "message"=>"BBBBBBB
"}]     MULTILINE
[2] tail.0: [1532799001.263741000, {"level"=>"NOT", "time"=>"Jul 28 17:30:01.263741", "message"=>"CCCCCCC
"}]     MULTILINE
[3] tail.0: [1532799001.786663000, {"level"=>"NOT", "time"=>"Jul 28 17:30:01.786663", "message"=>"DDDDDDD
"}]     MULTILINE
"}] tail.0: [1532799002.199434000, {"level"=>"NOT", "time"=>"Jul 28 17:30:02.199434", "message"=>"EEEEEEE
"}] tail.0: [1532799002.628430000, {"level"=>"NOT", "time"=>"Jul 28 17:30:02.628430", "message"=>"FFFFFFF
"}] tail.0: [1532799003.020809000, {"level"=>"NOT", "time"=>"Jul 28 17:30:03.020809", "message"=>"GGGGGGG
"}] tail.0: [1532799003.423898000, {"level"=>"NOT", "time"=>"Jul 28 17:30:03.423898", "message"=>"HHHHHHH
"}] tail.0: [1532799005.307595000, {"level"=>"NOT", "time"=>"Jul 28 17:30:05.307595", "message"=>"IIIIIII
"}] tail.0: [1532799003.795889000, {"level"=>"NOT", "time"=>"Jul 28 17:30:03.795889", "message"=>"JJJJJJJ

Your Environment

• Version used: v0.13.5

[edsiper]

That's the default and expected behavior: if multiline mode is enabled, the only parsers registered are the ones that start with Parser_, e.g:

• Parser_Firstline
• Parser_1
• Parser_2
• Parser_N

[JavaCS3]

@edsiper But why Parser demo will affect the result when Multiline is on? I don't understand how multiline is worked.

In my opinion, I guess fluent-bit will use Parser_Firstline to decide whether a line of log is the first line of multiline log. If not matched, then it is multiline content and will use Parser_1, Parser_2 ... and append then to a buffer. If matched, it will flush that buffer, and begin the next cycle.

[edsiper]

If a Parser is defined, Multiline routine is not processed (despite is On) . I guess the right fix is to trigger a warning about that situation to avoid confusion. I've pushed the following fix:

• 7eb7aeed

When Multiline is On, if a line matched Parser_Firstline, continuation lines will be matched against Parser_N parsers (just if a previous Parser_Firstline match exists)