Graylog2-server: Stream rules match but no messages routed into a stream

[emitev]

Hi,

I am using
Graylog v2.1.2+50e449a

Experiencing a problem with a setting a stream for Juniper SRX models - 110 210 100

I can see the logs from those devices coming in to graylog (see pics attached). I can do a regular search for source:{NameOfFireWall} and that gets me the results I want. I created a stream for ease of management. The stream matches on field source criteria "must contain" and here I am using the name of the firewall the same name as the one that I use when doing a regular search.

In the stream tab view in the graylog web interface the stream shows up ok and indicates that it is matching 40-60 msgs/s [ 73 messages/second, Must match all of the 1 configured stream rule(s). ]

When I click on the specific stream it shows no messages:

Nothing found in stream Johannesburg
Your search returned no results, try changing the used time range or the search query.

If I do a search it then finds the relevant messages and I can test a message against the stream it states that this message is a match and it would be routed to the stream but in reality it is not.

Picture 1 - Here it is showing that it is matching 30+ msgs/sec

Picture 2 - Here is a specific test for an exact message. It states the message would be routed to this stream.

Picture 3 - Here again it is showing that it is matching 30+ msgs/sec

Picture 4 - Here I did a normal search with the exact same condition Source:{NameOfFirewall} . It is showing all the messages as expected

Picture 5 - Here is the contents of the stream when I click on it. It shows no messages.

The exact same setup is working fine with juniper Netscreen SSG devices but there is something about the SRX that is different and is not working fine with the streams.

Expected Behavior

I expect the stream to route messages which match the confired rule inside the stream

Current Behavior

It shows that it is matching the messages but it is not routing them in the stream.

Possible Solution

Steps to Reproduce (for bugs)

1. Juniper SRX 210 110 or 100 device UDP Syslog logging port 5140
   2.Pretty standard Graylog deployment.
   3.Try to filter messages into any kind of stream matching on source or other field values
   4.

Context

Your Environment

• Graylog Version: Graylog v2.1.2+50e449a
• Elasticsearch Version:
• MongoDB Version:

• Operating System: Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-57-generic x86_64)

• Browser version:

[emitev]

so I left it over night. Did not do anything and this morning it is working fine showing logs routed into the correct streams. It takes a while for graylog to pickup the config i guess. Not sure what the explanation is.

[dennisoelkers]

This should not take that long. Can you reproduce the error by creating a new stream, adding a few (matching) rules and checking how long it takes before messages are routed into that stream?

[jalogisch]

@emitev can you please check the timezone of your Juniper and your Graylog Setting. What you had written sounds like the time or the time settings are not the same on all devices.

[jalogisch]

This issue is fairly old and there hasn't been much activity on it. Closing, but please re-open if it still occurs.

[edsonmarquezani]

Having the exact same issue right now. Single message matches but stream is empty, no messages being routed to it. Will wait for a while to see what happens.

[joschi]

@edsonmarquezani Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

[edsonmarquezani]

@joschi Ok. Thanks.

Edit: I figured it out by myself, and I think other people could benefit from my experience, as I found it a bit tricky, so I'm going to leave it reported here.

The problem was related to the field name, which contains dots originally (like io.kubernetes.pod.namespace). At a given point of the default Graylog's pipeline, dots are replaced by underscores, becoming io_kubernetes_pod_namespace.

Nonetheless, it turns out that events are processed by _streams_ before the field is renamed. So, the solution was to use the original field name in the matching rule.

[CMoH]

This seems like a design bug: all those test rules features should be consistent with this behaviour.

[joschi]

@CMoH I agree, but it will stay this way until we drop support for Elasticsearch 2.x, probably around Graylog 3.0.0.

[edsonmarquezani]

@joschi Wouldn't it be nice to have it documented somewhere, at least? It would save a lot of people precious time.

[joschi]

@edsonmarquezani You mean like in a FAQ section in the documentation?

http://docs.graylog.org/en/2.3/pages/faq.html#my-field-names-contain-dots-and-stream-alerts-do-not-match-anymore

[edsonmarquezani]

@joschi Oh, my bad, I had not seen it! That's perfect! Thanks.

[diranged]

This bit us as well.. even though its in the FAQ, its not an obvious thing to look for. :/

[bhoss12]

@joschi

can I have few examples how to setup few streams to trigger alert notifications as per streams.