Graylog2-server: Graylog 3.0 grok extractors broken

[leftorbit23]

After upgrading to Graylog 3.0, I noticed that many of my grok extractors didn't load.

server.log errors

2019-02-20T07:02:54.687-05:00 ERROR [InputServiceImpl] Cannot build extractor from persisted data. Skipping.
java.util.regex.PatternSyntaxException: named capturing group is missing trailing '>' near index 14
%ASA-\d-(?<asa_messageid>611101):

2019-02-20T07:02:54.684-05:00 ERROR [InputServiceImpl] Cannot build extractor from persisted data. Skipping.
java.util.regex.PatternSyntaxException: named capturing group is missing trailing '>' near index 14
%ASA-\d-(?<asa_messageid>106001): (?<name0>\S+) (?<name1>\S+) connection denied from (?<name2>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](
?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name3>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) to (?<name4>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?
:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name5>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) flags (?<asa_flags>.+)  on interface (?<name6>\S+)

2019-02-20T07:02:54.682-05:00 ERROR [InputServiceImpl] Cannot build extractor from persisted data. Skipping.
java.util.regex.PatternSyntaxException: named capturing group is missing trailing '>' near index 14
%ASA-\d-(?<asa_messageid>106023): (?<asa_action>Deny) (?<name0>\S+) src (?<name1>\S+):(?<name2>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.]
(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name3>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) dst (?<name4>\S+):(?<name5>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-
9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name6>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) by access-group (?<name7>(?>(?<!\\)(?>"(?>\\.|[^\\"]+)+
"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``)))


2019-02-20T07:02:54.680-05:00 ERROR [InputServiceImpl] Cannot build extractor from persisted data. Skipping.
java.util.regex.PatternSyntaxException: named capturing group is missing trailing '>' near index 14
%ASA-\d-(?<asa_messageid>302016): (?<asa_action>Teardown) (?<asa_proto>UDP) connection (?<name0>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) for (?<name1>\S+):(?<name2>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-
5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name3>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) to (?<name4>\S+):(?<
name5>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name6>(?<![0-9.+-])(?>[+-]?(?:(?:[
0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) duration (?<name7>(?!<[0-9])(?<name8>(?:2[0123]|[01]?[0-9])):(?<name9>(?:[0-5][0-9]))(?::(?<name10>(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))(?![0-9])) bytes (?<name11>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?
:\.[0-9]+)?)|(?:\.[0-9]+))))


2019-02-20T07:02:54.678-05:00 ERROR [InputServiceImpl] Cannot build extractor from persisted data. Skipping.
java.util.regex.PatternSyntaxException: named capturing group is missing trailing '>' near index 1421
(?<name0>(?:(?<name1>(?:(?<name2>((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-
f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\
d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{
1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\
d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?<name3>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]
{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))))|(?<name4>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)))) %ASA-\d-(?<asa_
messageid>106015): (?<asa_action>Deny) (?<asa_proto>TCP) (?<name5>.*?) from (?<name6>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[
0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name7>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) to (?<name8>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0
-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name9>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) flags (?<name10>.*?) on interface (?<name11>.*?)$

2019-02-20T07:02:54.676-05:00 ERROR [InputServiceImpl] Cannot build extractor from persisted data. Skipping.
java.util.regex.PatternSyntaxException: named capturing group is missing trailing '>' near index 1422
^(?<name0>(?:(?<name1>(?:(?<name2>((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa
-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?
\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]
{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?
\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?<name3>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9
]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))))|(?<name4>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)))) %ASA-\d-(?<asa
_messageid>710005): (?<asa_proto>TCP) request (?<asa_action>discarded) from (?<name5>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[
0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name6>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) to (?<name7>.*?):(?<name8>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])
[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name9>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))$


2019-02-20T07:02:54.674-05:00 ERROR [InputServiceImpl] Cannot build extractor from persisted data. Skipping.
java.util.regex.PatternSyntaxException: named capturing group is missing trailing '>' near index 14
%ASA-\d-(?<asa_messageid>302015): (?<asa_action>Built) (?<name0>\S+) (?<asa_proto>UDP) connection (?<name1>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) for (?<name2>\S+):(?<name3>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4]
[0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name4>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) \((?<name
5>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name6>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]
+(?:\.[0-9]+)?)|(?:\.[0-9]+))))\) to (?<name7>\S+):(?<name8>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[
0-5]))(?![0-9]))/(?<name9>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))) \((?<name10>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0
-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))/(?<name11>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))\)
              ^

Graylog 3.0 unable to process the following grok pattern:

(?<asa_proto>UDP)

Data sample:

Feb 20 2019 07:44:35: %ASA-6-302016: Teardown UDP connection 43191210 for outside:1.1.1.1/123 to inside:2.2.2.2/123 duration 0:04:01 bytes 985

Error:

We were not able to run the grok extraction because of the following error: named capturing group is missing trailing '>' near index 6 (?<asa_proto>UDP) ^

I was able to rewrite one of the grok patterns to get it to work

Before:

ASA-\\d-(?<asa_messageid>302016): (?<asa_action>Teardown) (?<asa_proto>UDP) connection %{BASE10NUM:asa_conn_id} for %{NOTSPACE:asa_interface_in}:%{IPV4:asa_src_ip}/%{BASE10NUM:asa_src_port} to %{NOTSPACE:asa_interface_out}:%{IPV4:asa_dst_ip}/%{BASE10NUM:asa_dst_port} duration %{TIME:asa_conn_durration} bytes %{BASE10NUM:asa_conn_bytes;long}

After:

ASA-\d-%{WORD:asa_messageid:int}: %{WORD:asa_action} %{WORD:asa_proto} connection %{BASE10NUM:asa_conn_id} for %{NOTSPACE:asa_interface_in}:%{IPV4:asa_src_ip}/%{BASE10NUM:asa_src_port} to %{NOTSPACE:asa_interface_out}:%{IPV4:asa_dst_ip}/%{BASE10NUM:asa_dst_port} duration %{TIME:asa_conn_durration} bytes %{BASE10NUM:asa_conn_bytes;long}

[Hetann]

remove the underscore from the capture group name

[leftorbit23]

I rewrote all my grok patterns using the workaround in my original post.

[kmerz]

@leftorbit23 I already looked into the problem (#5563). Can you please confirm that the patterns worked prior 3.0.

[Hetann]

@kmerz yes it was working before 3.0, we had a lot of patterns with underscore in capture group too

[leftorbit23]

@kmerz

I rewrote all my grok patterns using the workaround in my original post.

The example I provided worked prior to the upgrade

(?<asa_proto>UDP)

The following still works in 3.0:

%{WORD:asa_proto}

[kmerz]

@leftorbit23 I found the issue. We updated a library we use and they dropped the underscore support (unintentionally). I opened a issue there and we discuss now internally how to handle that.

[JSylvia007]

So I just upgraded from 2.5.x to 3.0, and I believe I'm having the same issue. I finally sorted out all the other warnings/errors on the graylog server.log file. I believe that I only have two remaining issues related to pfsense log extraction. I deleted the two extractors, both gave errors in the webUI and when accessing the UI produced the able error in the log. Unfortunately, the errors are still there even after a server restart. It looks like all my other data sources are fine, but my pfSense source isn't.

I'm not sure how to fix any of this, or if I can. I thought I had it figured out and that I'd be able to just remove those two extractors and then the data would then correctly be parsed again. That is apparently not the case though.

Is there anything I can do myself? Can someone point me in the right direction? I don't know how it can still be throwing the errors after I removed those two extractors.

If it helps, I used this guide to add the information to grafana: https://github.com/opc40772/pfsense-graylog

[kmerz]

For help you are better of asking in the community forum: https://community.graylog.org/
There I would provide a more detailed view on your server logs. That would definitely help your case.

But as I said in the community forum, since here is a place to discuss issues and how to fix them!

[JSylvia007]

For help you are better of asking in the community forum: https://community.graylog.org/
There I would provide a more detailed view on your server logs. That would definitely help your case.

But as I said in the community forum, since here is a place to discuss issues and how to fix them!

Thanks @kmerz. I just posted there, but I think I'm going to be screwed as I don't know what I need to fix and it looks like it's an upstream issue.

[bernd]

@leftorbit23 @Hetann @JSylvia007 This has been fixed in master and will be backported into the upcoming 3.0.1 release. That means in 3.0.1 you will be able to use underscores again.