Graylog2-server: Multiline Grok

[joschi]

The Grok extractor currently only supports single line matches which makes it hard to match log messages containing stack traces or other multiline content.

Logstash's implementation of Grok supports multiline matches by using the (?m) modifier in the pattern, but Graylog's Grok implementation doesn't.

[pramodanarase]

+1

[nhsalexb]

+1

[areynolds77]

This also affects the pipeline grok function.

[wildaxe]

I use different modifier (?s) in my pipeline functions to handle multi-line messages. It works well on 2.1.2.

[areynolds77]

That works perfectly, thanks @Nemchin!

[EvertMDC]

(?s) does not seem to work in the extractors.
There does not seem to be a lot of momentum to close this, which I find curious. May I know an estimation of when this will be fixed?

I would have assumed that any enterprise using Java will run into this. Or is it a better practice to use the pipelines?

[joschi]

@EvertMDC There are GELF appenders for virtually every Java logging framework which can be used to send log messages including multiline stack traces directly to Graylog, so there's no need to read these messages line-by-line from a file and merge them again at a later point.

[EvertMDC]

Hello @joschi
True, that makes sense. However all the output of all my containers all go to stdout, which docker sends to syslog, which is then read by filebeat. It would be weird to adapt my applications just to send the stacktraces through GELF when everything else is working.
I like the filebeat way of working because of the buffer it uses and the fact that the application does not have to care about it.

It is also best practice in the Twelve-Factor App

A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout.

[aliasmee]

+1

[danvaida]

I agree with @EvertMDC, this should definitely not be the concern of the application.
It'd be nice to have GELF supporting this natively.

[kb-elmo]

Any news on when this will be fixed?
It's pretty annoying to work with filebeat inputs without the ability to use the extractor properly.

[rcny]

Same problem as above. Is there any workaround for the Grok extractor or the only solution right now is to use pipelines for the extraction?

[afaqbabar]

Hi all, I want to extract fields from modsecurity audit log, which is multiline log with sections A,B,..Z. I can merge multiline messages of a single event using multiline feature in Graylog input.
How can I extract fields of all sections of a single event of a modsecurity log?

[jalogisch]

@afaqbabar your issues looks more like a problem on ingest. You need to ingest the multi-line message as one.

This issue is about the ability to match multiline messages with one GROK pattern not to merge them together!

@afaqbabar please use the community that will help you with your request.

[jayakumar-ml]

+1

[matiza5]

+1

[alexandervasylev]

+1

[jmacku]

+1

[albertMkhitaryan]

+1

[jzandbergen]

+1

[pericajakimov-sng]

+1

[markoknez]

+1

[xtruthx]

Dear Graylog Team,

I can confirm that this is working in a pipline_rule

I use different modifier (?s) in my pipeline functions to handle multi-line messages. It works well on 2.1.2.

I did not test in extractors.
If we could find a clear way to address this topic.

Thx