Telegraf: The regex processor does not correctly process capture groups

[neben]

Relevant telegraf.conf:

[[processors.regex]]
  [[processors.regex.tags]]
    key = "topic"
    pattern = '^Sensor_?(\d+)'
    replacement = "${1}"

[[processors.printer]]

System info:

Telegraf version 1.15.2
Docker 19.03.12 on Debian

Docker

Steps to reproduce:

1. Process data with tag topic="Sensor22_humi"

Expected behavior:

The topic tag should have the value "22", as the capture group should only capture digits. This is also the result of this Go regex tester.

Actual behavior:

The topic tag has the value "22_humi".

[sjwang90]

Same solution as #8044

https://docs.influxdata.com/telegraf/v1.15/administration/configuration/#measurement-filtering

[neben]

Same solution as #8044

https://docs.influxdata.com/telegraf/v1.15/administration/configuration/#measurement-filtering

How is this related? This issue has nothing to do with measurement filtering. Please reopen this issue.

[sjwang90]

Sorry read too quickly. Can you please provide the output you are receiving?

[neben]

Well, from the input Sensor22_humi the capture group captures 22_humi instead of 22. Any other info I can provide?

[Hipska]

That is correct behaviour, what you are specifying in the config results that Sensor22 will be replaced by 22. And that is actually the case.

You might want the regex to be ^Sensor_?(\d+).*

[neben]

That is correct behaviour, what you are specifying in the config results that Sensor22 will be replaced by 22. And that is actually the case.

No, unfortunately this is not the case, Sensor22_humi is replaced with 22_humi.

2020-09-02 09:48:57 | mqtt_consumer,host=b3b68196d864,topic=Sensor22_humi humidity=50.9 1599032937961636936
2020-09-02 09:48:57 | mqtt_consumer,host=b3b68196d864,topic=22_humi humidity=50.9 1599032937961636936

[Hipska]

No, did you try my regex?

You have the string Sensor22_humi, your regex is set to match only the part Sensor22 and you specified to match the last decimals of that part in a group. Then you specify that the replacement for the total match (being Sensor22) is the content of matching group 1 (being 22). When you replace Sensor22 in the total string of Sensor22_humi by 22, the result will then obviously be 22_humi.

My proposal of RegEx will match the complete string, resulting in a replacement like what you are expecting. You simply put the wrong RegEx. See the example on RegExr and compare with your RegEx by removing the last .* and see what the effect is.

[neben]

No, did you try my regex?

I already had been using that as a workaround for the presumed faulty plugin, but I thought it had to do with greedy matching.

You have the string Sensor22_humi, your regex is set to match only the part Sensor22 and you specified to match the last decimals of that part in a group. Then you specify that the replacement for the total match (being Sensor22) is the content of matching group 1 (being 22). When you replace Sensor22 in the total string of Sensor22_humi by 22, the result will then obviously be 22_humi.

I see, I missed the part that not the entire string is replaced, but only the matching pattern. I assumed the pattern was only used for filtering, where in reality it's being used to select the part being replaced. Which makes total sense. Thank you for clarifying!

[sjwang90]

Thanks @Hipska!