Vector: Feature request: Add support for grouping parsed JSON fields under a specific field

[AlexGustafsson]

---

name: Feature Request
about: Suggest a new feature or other enhancement.

labels: Type: Enhancement

Current Vector Version

Docker, 0.9.1-alpine

Use-cases

1. To mitigate the potential merge collisions mentioned here: https://vector.dev/docs/reference/transforms/json_parser/#merge-conflicts.
2. To help structure the result of logs for services such as Traefik and Kibana which provide many values with names such as type, req, tags etc.

Attempted Solutions

Not attempted, but the runtime transform could be used with a third-party JSON library.

Proposal

JSON Parser Transform:

[transforms.parse-json]
  type = "json_parser"
  inputs = ["in"]
  drop_field = true
  drop_invalid = true
  field = "message"
  # New optional value
  namespace = "my.log.namespace"

JSON log:

{"id": "1", "foo": "2"}

JSON output:

{
  "timestamp": "2020-07-13T14:32:13+0000",
  "my": {
    "log": {
      "namespace": {
          "id": "1",
          "foo": "2",
       }
    }
  }
}

References

[lukesteensen]

@AlexGustafsson I believe we support this now with the target_field option: https://vector.dev/docs/reference/transforms/json_parser/#target_field

Does that meet your requirements? The only difference I see is that it might not supported writing to a nested key, which is something we can fix.

[AlexGustafsson]

That seems to be what I'm after. I don't have any personal use for nested keys and I can't think of any use case right now, so I assume my feature request is invalid for now.

Thanks for the help! I'm leaving the issue open, if you decide to move forward with nested keys. Feel free to close it.

[lukesteensen]

Sounds good. We really do appreciate users opening issues like this, so thank you! Even if the feature is present, this tells us that the documentation should be clearer.

[AlexGustafsson]

I think it’s only right I take this opportunity to applaud you for an absolutely fantastic website. The layout, graphics and documentation is all superb! It’s comprehensible and technical enough where it needs to be.