Vector: `rename_fields` transform does not parse field names properly

[MOZGIII]

This is branched of from master @ 2fdbd762ef80ebd325bfb4fc92a5bcb00adcdd2a.

[transforms.rename]
  inputs = [".."]
  fields.k8s_logs_ingestion_timestamp = "..."

k8s_logs_ingestion_timestamp is not recognized properly.

[jszwedko]

To add some additional context, this appears to be due the deserialization of the field lookup stopping when it gets to the 8:

Oct 28 19:51:41.823  WARN transform{component_kind="transform" component_name=rename component_type=rename_fields}: vector::internal_events::rename_fields: Field did not exist. field=k rate_limit_secs=30

This appears to be due to https://github.com/timberio/vector/blob/master/src/mapping/parser/grammar.pest#L23 not allowing numbers.

0.10.0 parses this field fine so I'd consider this a regression.

2477 may be related.

[Hoverbear]

4277 might be related.

[MOZGIII]

Is there a workaround we could use for now? Just for completeness.

[Hoverbear]

Users can use stable for now!

Otherwise we should switch these transforms back to strings?

I'm not sure how far off #4277 is along. :(

[JeanMertz]

@Hoverbear it'll be merged tomorrow.

edit: although yes, @jszwedko is correct, this is because our grammar doesn't support numbers in fields, which we of course should. I can tackle that in the same PR, or someone can create a separate one.

[JeanMertz]

I believe this issue can be broadened a bit, as I think I've constructed a couple of tests that show master and 0.10 working differently:

[transforms.rename_fields_lookup_error_at]
  inputs = []
  type = "rename_fields"
  [transforms.rename_fields_lookup_error_at.fields]
    "@timestamp" = "aa"
[[tests]]
  name = "rename_fields_lookup_error_at"
  [tests.input]
    insert_at = "rename_fields_lookup_error_at"
    type = "log"
    [tests.input.log_fields]
      "@timestamp" = true
  [[tests.outputs]]
    extract_from = "rename_fields_lookup_error_at"
    [[tests.outputs.conditions]]
      "aa.equals" = true

[transforms.rename_fields_lookup_error_escape]
  inputs = []
  type = "rename_fields"
  [transforms.rename_fields_lookup_error_escape.fields]
    "a.b\\.c" = "aa"
[[tests]]
  name = "rename_fields_lookup_error_escape"
  [tests.input]
    insert_at = "rename_fields_lookup_error_escape"
    type = "log"
    [tests.input.log_fields]
      "a.b\\.c" = true
  [[tests.outputs]]
    extract_from = "rename_fields_lookup_error_escape"
    [[tests.outputs.conditions]]
      "aa.equals" = true

Specifically:

Paths with characters that were valid in 0.10 are no longer valid in master

In the first test, using a path such as foo.@timestamp became invalid in master. This is because Lookup uses the more strict definition of paths used in the remap-language as opposed to what's possible in the existing Vector config file.

I'm not sure what other characters are no longer allowed, but basically only alphanumeric and - and _ is allowed with the new set-up.

Character escaping that worked in 0.10 no longer works in master

In 0.10, if you wanted to escape a dot in a path segment, you'd write foo.bar\.baz. In master, this is written as foo."bar.baz". The former no longer works as expected, and will break configurations.

---

We'll have to fix both issues to be able to use the new Lookup-based path logic in 0.11, or we have to revert that change for now, and ship it in 0.12.

[Hoverbear]

4804 triages this