Graylog2-server: Unable to parse grok patterns

[colttt]

Expected Behavior

working grok patterns

Current Behavior

some grok patterns are not working

Possible Solution

???

Steps to Reproduce (for bugs)

1. I use grok patterns from here https://github.com/whyscream/postfix-grok-patterns/blob/master/postfix.grok
2. I import it
3. I want to use the grok pattern %{POSTFIX_SMTPD}
4. first time it works, but i missconfigure it (forgott "Named captures only"),
5. try to add/update this, but i got an error message
6. I deleted this extractor
7. add a new one and check "Named captures only"
8. it doesn't work (error message see below), other grok patterns work fine
9. if I don't use "Named captures only" it can be added

Your Environment

• Graylog Version: 2.2.3-1
• Elasticsearch Version: 2.3.4
• MongoDB Version: 1:2.4.9-1ubuntu2
• Operating System: Ubuntu 14.04.5 LTS
• Browser version: Google Chrome Version 59.0.3071.109 (Official Build) (64-bit)

server.log:

2017-06-30T11:26:44.274+02:00 ERROR [GrokExtractor] Unable to parse grok patterns
oi.thekraken.grok.api.exception.GrokException: Deep recursion pattern compilation of %{POSTFIX_SMTPD}
        at oi.thekraken.grok.api.Grok.compile(Grok.java:356) ~[graylog.jar:?]
        at org.graylog2.inputs.extractors.GrokExtractor.<init>(GrokExtractor.java:79) [graylog.jar:?]
        at org.graylog2.inputs.extractors.ExtractorFactory.factory(ExtractorFactory.java:65) [graylog.jar:?]
        at org.graylog2.rest.resources.system.inputs.ExtractorsResource.buildExtractorFromRequest(ExtractorsResource.java:346) [graylog.jar:?]
        at org.graylog2.rest.resources.system.inputs.ExtractorsResource.create(ExtractorsResource.java:121) [graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
2017-06-30T11:26:44.274+02:00 ERROR [ExtractorsResource] Cannot create extractor. Missing configuration.
org.graylog2.ConfigurationException: Unable to parse grok patterns
        at org.graylog2.inputs.extractors.GrokExtractor.<init>(GrokExtractor.java:82) ~[graylog.jar:?]
        at org.graylog2.inputs.extractors.ExtractorFactory.factory(ExtractorFactory.java:65) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.inputs.ExtractorsResource.buildExtractorFromRequest(ExtractorsResource.java:346) [graylog.jar:?]
        at org.graylog2.rest.resources.system.inputs.ExtractorsResource.create(ExtractorsResource.java:121) [graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

[joschi]

@colttt Please attach the contents of the "grok_patterns" collection in MongoDB from that Graylog setup.

You can use mongoexport to export it as CSV or JSON file. Alternatively, please create a content pack containing the Grok patterns (see System / Grok Patterns).

[colttt]

@joschi in attach the content pack with the grok patterns
postfix_content_pack.zip

[xtruthx]

I ve got the same issue here with default grok pattern which you can find on grokdebug.herokuapp.com or in the logstash core grok-pattern.

Expected Behavior

Current Behavior

gl2_processing_error

For rule 'kemp message processing': In call to function 'grok' at 8:17 an exception was thrown: Deep recursion pattern compilation of %{SYSLOG5424PRI}%{SYSLOGPROG}: %{GREEDYDATA:message}

Possible Solution

Steps to Reproduce (for bugs)

1.
2.
3.
4.

Context

This how the grok function in my rule looks like. I got the same behavior in an grok extractor on a input.
let action = grok(pattern: "%{SYSLOG5424PRI}%{SYSLOGPROG}: %{GREEDYDATA:message}", value: message_field, only_named_captures: true);

Your Environment

• Graylog Version: 2.4.3+2c41897, codename Wildwuchs
• Elasticsearch Version: 5.6.6
• MongoDB Version: 1:3.2.11-2+deb9u1

• Operating System: PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
  NAME="Debian GNU/Linux"
  VERSION_ID="9"
  VERSION="9 (stretch)"
  ID=debian
  HOME_URL="https://www.debian.org/"
  SUPPORT_URL="https://www.debian.org/support"
  BUG_REPORT_URL="https://bugs.debian.org/"

• Browser version: Google Chrome Version 63.0.3239.132 (Offizieller Build) (64-Bit)

[joschi]

@xtruthx
See https://github.com/Graylog2/graylog2-server/issues/3949#issuecomment-312224268

[xtruthx]

content_pack.zip.zip

[joschi]

@xtruthx It looks like you're missing the POSINT grok pattern, which is why SYSLOGPROG cannot resolve properly, and the NONNEGINT pattern, which is why SYSLOG542PRI cannot resolve properly.

{
   "streams" : [],
   "lookup_tables" : [],
   "lookup_data_adapters" : [],
   "dashboards" : [],
   "description" : "0815 pattern",
   "inputs" : [],
   "category" : "default grok",
   "name" : "default_grok_pattern",
   "grok_patterns" : [
      {
         "name" : "SYSLOG542PRI",
         "pattern" : "<%{NONNEGINT:syslog5424_pri}>"
      },
      {
         "pattern" : "(?:[\\w._/%-]+)",
         "name" : "PROG"
      },
      {
         "pattern" : "%{PROG:program}(?:\\[%{POSINT:pid}\\])?",
         "name" : "SYSLOGPROG"
      },
      {
         "pattern" : ".*",
         "name" : "GREEDYDATA"
      }
   ],
   "outputs" : [],
   "lookup_caches" : []
}

[xtruthx]

sorry i forgot to put them into the conent_pack due to the fact that they are default grok pattern which are already exist in graylog.

[joschi]

@xtruthx Please provide all Grok patterns that have been configured in Graylog or attach a complete dump of the "grok_patterns" collection in MongoDB.

[bernd]

@xtruthx The content pack imports the SYSLOG542PRI pattern but in the rule you are using SYSLOG5424PRI. The grok pattern name is misspelled (missing 4 after the 2) or you have to use the correct name in the pipeline rule.

This works for me with the content pack you provided. (I changed the pattern name in the pipeline rule)

rule "issue-3949"
when true
then
  let message_field = to_string($message.message);

  let action = grok(pattern: "%{SYSLOG542PRI}%{SYSLOGPROG}: %{GREEDYDATA:message}", value: message_field, only_named_captures: true);
  debug(action);
  set_fields(action);

  let syslogpri = expand_syslog_priority($message.syslog5424_pri);

  set_fields({facility: syslogpri.facility, level:syslogpri.level});
end

Input:

<45>syslog-ng[7208]: syslog-ng starting up; version='3.5.3'

Output:

2018-02-09 18:22:10,680 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {message=syslog-ng starting up; version='3.5.3', pid=7208, program=syslog-ng, syslog5424_pri=45}

[xtruthx]

Ok at least blame on me. It was really a typo. Sorry. You can count it on a bad resulution quality and the error message which not show clearly that it misses a pattern. At least due to the fact that there is still a issue pending that can not resolve recursion at specific level forced me to belive that i hit this issue too.
Which i already experienced last week but i am not able to reproduce it.

It should give you a hint that there is a missing pattern.
So may want to think about my suggestion to adapt the error message if a pattern is missing.

[joschi]

@xtruthx Thanks for your feedback!

There's a reason this issue is still open and closing it will involve having better error messages in case of missing Grok patterns.

[xtruthx]

@joschi sorry. i cleared my last comment. Cause i did not want to complain about the open issue.

[joschi]

@xtruthx No offense taken. I simply wanted to let you know that the error message for missing Grok patterns will be improved when this issue has been resolved. 😉