Vector: Transform to allowlist fields

[jszwedko]

There are currently transforms for adding fields (add_fields) and removing fields (remove_fields) but there isn't an easy way to filter for only allowed fields which can be useful for sources that have a large number of fields, but we only care about a subset of them. Maybe it could be called filter_fields?

This came from gitter https://gitter.im/timberio-vector/community?at=5f374ae3aecae32e8ec3bf0d

I think this could also be incorporated into the work happening with the new remap transform as well as something to consider with event mapping at the source or sink level currently under consideration.

[binarylogic]

I believe @Jeffail already has a function for this in https://github.com/timberio/vector/pull/3341. @Jeffail would you mind linking this to the PR if that's true?

[Jeffail]

Yeah this is now part of https://github.com/timberio/vector/pull/3341 as of https://github.com/timberio/vector/pull/3341/commits/af0b6aeaadf74f30bf4d7e7037801a65c6bca531, it's a function called only_fields and looks like this in the wild:

[transforms.mapper]
  inputs = [ "foo" ]
  type = "remap"
  mapping = "only_fields(.foo, .bar.baz, .message, .timestamp, .host)"

[jszwedko]

That is pretty wild.

:tada:

[mdfranz]

Very cool!