Graylog2-server: Pipeline processing will not stop for dropped messages

[jtkkex]

When trying to filter out noise from event logs, the pipeline rules seem as excellent option, as complex conditions are available.

Expected Behavior

When setting up noise reduction pipeline, it would seem logical to make the most voluminous drops at stage 0, and then have less messages to process in later stages.

Current Behavior

At the moment, there are two things that happen

1. If there is no "empty rule":

rule "empty"
when 
  (1 == 1)
then
end

processing does not continue to the next stages for the messages that are not dropped.

1. After adding the empty rule to the first stage, all messages are processed on the second stage, even the dropped ones.

Possible Solution

Two suggestions:

1. Add a new condition to pipeline stage processing: "continue processing on the next step if none of the rules match condition"
2. When message is dropped, stop evaluating other rules for that message

Steps to Reproduce (for bugs)

1. Make a stream Windows Events and route a Windows Event log entry there
2. Make a pipeline "Filter unnecessary Windows Events". Connect it to the Windows Events stream
3. Stage 0, create a rule that drops messages with condition EventID == 4662 (for example - if your test setup creates these events)
4. Stage 1, create a rule that forwards the messages to a specific new stream "TestStream"

Context

Your Environment

• Graylog Version: 2.4.5
• Elasticsearch Version: 5.6.9
• MongoDB Version: 3.4
• Operating System: CentOs7
• Browser version: any

[jalogisch]

@jtkkex depending on your conditions the fastest possible solution would be to have multiple conditions:

alert "multi con"
when
  has_field("EventID") AND
  to_string($message.eventid) == 4662 OR
  to_string($message.eventid) == 4672 OR
  to_string($message.eventid) == 4662
then
  drop_message();
end

Currently your suggestion would be against the design principles of the processing pipelines that you need to select on purpose what is processed and what not.

[jtkkex]

Actually, I do use multiple conditions. The "problem" is that after drop_message(), the processing of the message does not stop. So having stage 0 drop messages that are not wanted and then stage 1 processing fields of (also) the dropped messages is so unnecessary. If for example 2000 msgs/s are processed at stage 0. 90% of them are dropped. Then at stage 1, again 2000 msgs/s are processed, in stead of the 200msg/s that would seem logical to me.

[florianpopp]

This looks more like a bug and we will take a look at it.

[GaneshbabuRamamoorthy]

Hi @jalogisch @florianvolle

I am facing the similar issue in my env where graylog is running 3.1.1 version. Let us know if there is any update on this request.

https://community.graylog.org/t/pipeline-processing-continues-after-messages-getting-dropped-in-previous-stage-of-pipeline/13023

Thanks,
Ganeshbabu R

[tmacgbay]

Beyond drop_message(), It would be helpful to have a separate exit_pipeline() function to reduce unnecessary processing and/or pipeline stages. Being able to set rule execution sequence within a stage would help too... but is slightly off topic.

[jalogisch]

@tmacgbay you might want to open a feature request for that function.