Graylog2-server: GROK Extractor with OR does not work reliable

[jalogisch]

Expected Behavior

When using GROK Pattern to extract content from fields with OR like (%{INT:login}|%{WORD:login}) it should match on one of the Patterns.

Current Behavior

If the second option in the OR is NULL the field is not created.

Steps to Reproduce (for bugs)

1. select a field to create a GROK Extractor from
2. create a grok pattern include OR (example above)
3. use named_matches_only
4. Hit try to see what is matched.

Context

When you use multiple nested pattern it can look like it is not possible to have a working solution, sometimes a match happens and sometimes not.

Your Environment

• Graylog Version: 2.4.3

[petracvv]

I am seeing something similar using the pipeline grok() function. ORs do not seem to work as they should.

Would that be the same issue as here?

[kmerz]

I just tested it on 3.0.0 pre alpha and it seems there is yet a different result again:

[jalogisch]

it looks like the 3.0 release will behave now like other GROK implementations out in the wild. It should be verified but I guess that it can be closed as resolved then.

[edmundoa]

I tested this against 3.0.0-rc.1 and it worked fine for me. I'm closing the ticket, @jalogisch please reopen if you can reproduce the original issue.

[kmerz]

It was decided that the correct behavior is that the return value should be without null:

So DELETE instead of [DELETE, null].

[kmerz]

The implemented solution has a unexpected behavior:
2019-03-27 11:48:16,049 ERROR: org.graylog2.filters.ExtractorFilter - Could not apply extractor "UFW SHORT" (id=c4b53200-2a20-11e9-91c5-00e18cb9c35a) to message d377ab03-507d-11e9-b2bb-00e18cb9c35a io.krakens.grok.api.exception.GrokException: key 'IPV4' has multiple non-null values, this is not allowed in flattened mode, values:'192.168.2.108', '239.255.255.250' at io.krakens.grok.api.Match.lambda$capture$0(Match.java:175) ~[grok-0.1.9-graylog-1.jar:?] at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684) ~[?:1.8.0_191] at io.krakens.grok.api.Match.capture(Match.java:134) ~[grok-0.1.9-graylog-1.jar:?] at io.krakens.grok.api.Match.captureFlattened(Match.java:109) ~[grok-0.1.9-graylog-1.jar:?] at org.graylog2.inputs.extractors.GrokExtractor.run(GrokExtractor.java:94) ~[classes/:?] at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:214) ~[classes/:?] at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:77) [classes/:?] at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:100) [classes/:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:114) [classes/:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:100) [classes/:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:77) [classes/:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [classes/:?] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [disruptor-3.4.2.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [metrics-core-4.0.3.jar:4.0.3] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

We might need to implement out own flatten mechanism.