Beats: [Filebeat] Elasticsearch module - regular expression has redundant nested repeat operator

[andrewkroh]

Elasticsearch is logging warnings as a result of the Elasticsearch Filebeat module pipeline.

regular expression has redundant nested repeat operator * ...

Versions:

• Filebeat 8.0.0-SNAPSHOT (daf90991ac032376ace85db2626cef17ff5201cf)
• Elasticsearch 8.0.0-SNAPSHOT (b7af85242288)

Here's the full log output from Elasticsearch (as collected by Filebeat (super meta)).

{
    "agent": {
      "hostname": "es",
      "id": "f619f7c9-d4fe-4efc-8f3f-6df680f57380",
      "ephemeral_id": "446eb7aa-4831-40fe-b478-e720430d2abe",
      "type": "filebeat",
      "version": "8.0.0"
    },
    "log": {
      "file": {
        "path": "/var/log/elasticsearch/elasticsearch.log"
      },
      "offset": 21118272,
      "level": "WARN"
    },
    "message": "regular expression has redundant nested repeat operator * /(?:(?:(?:(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?): (?<BASE10NUM:elasticsearch.gc.jvm_runtime_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))):)|(?:\\[(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)\\]\\[(?<POSINT:process.pid>\\b(?:[1-9][0-9]*)\\b)\\]\\[(?<DATA:elasticsearch.gc.tags>.*?)(?:\\s*)*\\])) Total time for which application threads were stopped: (?<BASE10NUM:elasticsearch.gc.threads_total_stop_time_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) seconds, Stopping threads took: (?<BASE10NUM:elasticsearch.gc.stopping_threads_time_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) seconds)|(?:(?:(?:(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?): (?<BASE10NUM:elasticsearch.gc.jvm_runtime_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))):)) \\[GC \\((?<DATA:elasticsearch.gc.phase.name>.*?)\\) \\[YG occupancy: (?<BASE10NUM:elasticsearch.gc.young_gen.used_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) K \\((?<BASE10NUM:elasticsearch.gc.young_gen.size_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) K\\)\\](?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))): \\[Rescan \\(parallel\\) , (?<BASE10NUM:elasticsearch.gc.phase.parallel_rescan_time_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) secs\\](?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))): \\[weak refs processing, (?<BASE10NUM:elasticsearch.gc.phase.weak_refs_processing_time_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) secs\\](?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))): \\[class unloading, (?<BASE10NUM:elasticsearch.gc.phase.class_unload_time_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) secs\\](?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))): \\[scrub symbol table, (?<BASE10NUM:elasticsearch.gc.phase.scrub_symbol_table_time_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) secs\\](?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))): \\[scrub string table, (?<BASE10NUM:elasticsearch.gc.phase.scrub_string_table_time_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) secs\\]\\[1 CMS-remark: (?<BASE10NUM:elasticsearch.gc.old_gen.used_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\((?<BASE10NUM:elasticsearch.gc.old_gen.size_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\)\\] (?<BASE10NUM:elasticsearch.gc.heap.used_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\((?<BASE10NUM:elasticsearch.gc.heap.size_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\), (?<BASE10NUM:elasticsearch.gc.phase.duration_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) secs\\] (?:\\[Times: user=(?<BASE10NUM:elasticsearch.gc.phase.cpu_time.user_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) sys=(?<BASE10NUM:elasticsearch.gc.phase.cpu_time.sys_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))), real=(?<BASE10NUM:elasticsearch.gc.phase.cpu_time.real_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) secs\\]))|(?:(?:(?:(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?): (?<BASE10NUM:elasticsearch.gc.jvm_runtime_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))):)) \\[GC \\((?<DATA:elasticsearch.gc.phase.name>.*?)\\) \\[(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) CMS-initial-mark: (?<BASE10NUM:elasticsearch.gc.old_gen.used_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\((?<BASE10NUM:elasticsearch.gc.old_gen.size_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\)\\] (?<BASE10NUM:elasticsearch.gc.heap.used_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\((?<BASE10NUM:elasticsearch.gc.heap.size_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\), (?<BASE10NUM:elasticsearch.gc.phase.duration_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) secs\\] (?:\\[Times: user=(?<BASE10NUM:elasticsearch.gc.phase.cpu_time.user_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) sys=(?<BASE10NUM:elasticsearch.gc.phase.cpu_time.sys_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))), real=(?<BASE10NUM:elasticsearch.gc.phase.cpu_time.real_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))) secs\\]))|(?:(?:\\[(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)\\]\\[(?<POSINT:process.pid>\\b(?:[1-9][0-9]*)\\b)\\]\\[(?<DATA:elasticsearch.gc.tags>.*?)(?:\\s*)*\\]) GC\\((?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))\\) ParNew: (?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K-\\>(?<BASE10NUM:elasticsearch.gc.young_gen.used_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\((?<BASE10NUM:elasticsearch.gc.young_gen.size_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\))|(?:(?:\\[(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)\\]\\[(?<POSINT:process.pid>\\b(?:[1-9][0-9]*)\\b)\\]\\[(?<DATA:elasticsearch.gc.tags>.*?)(?:\\s*)*\\]) GC\\((?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))\\) Old: (?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K-\\>(?<BASE10NUM:elasticsearch.gc.old_gen.used_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\((?<BASE10NUM:elasticsearch.gc.old_gen.size_kb>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+))))K\\))|(?:(?:(?:(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?): (?<BASE10NUM:elasticsearch.gc.jvm_runtime_sec>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))):)|(?:\\[(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)\\]\\[(?<POSINT:process.pid>\\b(?:[1-9][0-9]*)\\b)\\]\\[(?<DATA:elasticsearch.gc.tags>.*?)(?:\\s*)*\\])) (?<GREEDYMULTILINE:message>(.|",
    "fileset": {
      "name": "server"
    },
    "input": {
      "type": "log"
    },
    "@timestamp": "2020-04-01T15:38:52.340Z",
    "ecs": {
      "version": "1.5.0"
    },
    "elasticsearch": {
      "server": {},
      "node": {
        "name": "es"
      },
      "component": "stderr"
    },
    "service": {
      "type": "elasticsearch"
    },
    "host": {
      "name": "es"
    },
    "event": {
      "timezone": "+00:00",
      "created": "2020-04-01T15:39:01.053Z",
      "kind": "event",
      "module": "elasticsearch",
      "category": "database",
      "type": "info",
      "dataset": "elasticsearch.server"
    }
  }

[andrewkroh]

This looks really similar to https://github.com/elastic/beats/issues/15840.

[kaiyan-sheng]

@ycombinator Do you have any insight on this?

[ycombinator]

Hmm, I thought we fixed this in https://github.com/elastic/beats/pull/15900, including the ingest grok pattern for the elasticsearch.gc dataset but perhaps we missed some spots? Or perhaps grok pattern definitions were recently updated in ES so now some wildcards in the dataset's grok pattern have become redundant? Either way, let's keep this issue open so we can investigate.

[adriansr]

We're seeing this issue in other modules too. Just received a contribution to fix mysql module: https://github.com/elastic/beats/pull/17156 I am running a test to see how many modules cause these errors in ES.

I think we should communicate this to the Elasticsearch (ingest?) team so they can also check where is that msg being printed, seems a library written straight to stderr, while this msg should go to debug.

[adriansr]

The only module that produced this warning during system-tests is activemq (x-pack). Tested with ES 7.6.0 and 7.6.2.

I'll submit a patch.

[willemdh]

I think we are experiencing this issue for some time now. This problem is filling up our disks as it is logging huge amount of stuff like this to /var/log/messages:

Apr  1 11:57:56 ourelasticnode elasticsearch: regular expression has redundant nested repeat operator * /^# User@Host: (?<USER:user.name>(?:[a-zA-Z0-9._-]+))(\[(?<USER:mysqurce.domain>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))? \[(?<IP:source.ip>(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d{1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-:([ #
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*))(Id:(?:\s*)(?<NUMBER:mysql.thread_id:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(Thread_id:(?:\s*)(?<NUMBER:mysql.thread_id>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(Schema:(?:\s*)(?<WORD:mysql.slowlog.schema>\b\w+\b)?(?:([ #
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(Last_errno: (?<NUMBER:mysql.slowlog.last_errno:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(Killed: (?<NUMBER:mysql.slowlog.killed:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:(
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(QC_hit: (?<WORD:mysql.slowlog.query_cache_hit>\b\w+\b)(?:([ #
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(Query_time: (?<NUMBER:temp.duration:float>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(Lock_time: (?<NUMBER:mysql.slowlog.lock_time.sec:float>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(Rows_sent: (?<NUMBER:mysql.slowlog.rows_sent:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(Rows_examined: (?<NUMBER:mysql.slowlog.rows_examined:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0
Apr  1 11:57:56 ourelasticnode elasticsearch: ]*)))?(Rows_affected: (?<NUMBER:mysql.slowlog.rows_affected:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0

@adriansr We tried stopping Filebeat, but that didn't help. Is there any workaround we can apply to stop this from happening or do we have to wait for 7.6.3?

Case number is 00510847

[adriansr]

@willemdh that particular warning is fixed by #17156. As it might take some time until the fix is released, I think an easy workaround is to replace your current ingest pipeline (/etc/share/filebeat/module/mysql/slowlog/ingest/pipeline.json) with the fixed pipeline, delete your current pipeline in Elasticsearch and the patched one will be installed once Filebeat is started.

[willemdh]

@adriansr Thanks for the suggestion, will try that tomorrow!

[adriansr]

@andrewkroh It looks like you're using an elasticsearch/gc pipeline without the fix in #15900. The regexp in the error message has one extra *.

What's the output of:

curl 'http://elasticsearch:9200/_ingest/pipeline/filebeat-8.0.0-elasticsearch-gc-pipeline?pretty' | grep '^\s*"JVM9'

it shouldn't have a star at the end:

- "JVM9HEADER" : "\\[%{TIMESTAMP_ISO8601: <...> %{SPACE}*\\]",
+ "JVM9HEADER" : "\\[%{TIMESTAMP_ISO8601: <...> %{SPACE}\\]",

[adriansr]

Turns out Elasticsearch had the old pipeline installed. I guess an older 8.0.0 has been used in this cluster in the past, and the pipeline is not updated if the version number is the same.

I updated it, problem should be gone.

Another cause for this message could be having pipelines for older versions installed. This will cause the error to appear every time an Elasticsearch instance starts.

[willemdh]

@adriansr

"Another cause for this message could be having pipelines for older versions installed. "

Correctly, after updating the mysql slowlog pipeline we were still seeing these regex logs. Only after deleting all old slowlog mysql pipelines, the issues seems to be resolved.

[andrewkroh]

Thanks @adriansr for diving into this issue!

[kayaktri]

I wanted to let everyone know that I upgraded to 7.7.0 and still was having this issue - "regular expression has redundant nested repeat operator ...". As adriansr pointed out, it was because I had pipelines (not the same ones as the other users above) from older versions that were left installed on the cluster.

You can check to see if you have older pipelines in your config by running this command:
curl -X GET "(localhost or your IP address):9200/_ingest/pipeline?pretty"
I was able to see I had pipelines from version 7.6, 7.5, 7.4, 7.3, 7.2, etc.

I was able to delete these by running the command:
curl -X DELETE "(localhost or your IP address):9200/_ingest/pipeline/*"

Next, I stop and started the service (systemctl stop/start elasticsearch in my case).

Next, I reran the command to list the pipelines ES was using (see above) and I now only had version 7.7.0 pipelines.

To determine if this corrected the issue, I ran the following command to show me the elasticsearch log after I restarted the service:

journalctl -u elasticsearch --no-pager

And all the entries for the "redundant regular expression..." were no longer there. Problem solved.