Vector: Ability to include multiple Vector configuration files

[karlseguin]

Don't think this is currently supported, but is there any appetite for an include directive to include other config files? Helps organize things a little better (distinct sources -> transform -> sink flows can be in their own file).

[Jeffail]

I think we can go in one (or both) of two significantly different directions here:

Import fields

The syntax could be very simple. Something like an array field imports at the root of a config file:

imports = [
    "./path/to/foo.toml",
    "./path/to/bar.toml",
    "./path/to/${ENV}.toml",
]

However, this syntax limits us in the future if we wanted to allow more specialized imports. If we wish to leave that door open we can instead support import tables:

[[imports]]
  path = "./path/to/foo.toml"

Which is a little more to write, but allows us to expand the spec later on for higher levels of import customization:

[[imports]]
  path = "./path/to/bar.toml"
  components = [
      "sinks.send_logs_to_kafka",
  ]

N path args in config flag

Another possibility is to allow multiple values for the --config flag. Paths could also be directories, and potentially support ellipsis syntax for instructing vector to walk a directory for any .toml files:

vector -c "./foo.toml" -c "./and_also/${ENV}" -c "./and_also_these/..."

This is much more limited than using configuration parameters but is probably easier for users to pick up. It also ties nicely in with https://github.com/timberio/vector/issues/98#issuecomment-542660074 as it would complement vector --dry-run -c ./config/examples/....

However, this is also very easy to go wrong for users. Allowing Vector to walk a directory for .toml files also sets it up nicely to gobble up all kinds of unrelated configs unbeknownst to our user.

[lukesteensen]

I like the imports array idea! The table idea is neat but I don't have a good idea of which use cases we'd want to implement beyond just basic imports.

[binarylogic]

Yeah, the imports directive is interesting. Although, I prefer the path args approach:

Another possibility is to allow multiple values for the --config flag. Paths could also be directories, and potentially support ellipsis syntax for instructing vector to walk a directory for any .toml files

That's pretty standard and less surprising. I'm also 👍 on supporting globbing.

[Jeffail]

I'm going to create some specs for https://github.com/timberio/vector/issues/232 next and depending on how that goes there might be ways in which these proposals tie in.

[Jeffail]

One interesting use case I would quite like to solve would be to import config snippets multiple times with different insertion points. For example, if we have a pretty complex set of transforms for doing something that is common to us we might break it out into a snippet, e.g. ./snippets/quack_to_woof.toml:

[transform.foo]
  type = "foo"
  inputs = ["source_quack"]

[transform.bar]
  type = "bar"
  inputs = ["foo"]

[transform.resulting_woof]
  type = "baz"
  inputs = ["bar"]

However, we can only import that snippet once into a root config as it has a static source source_quack and a static sink resulting_woof. What if we want to use this snippet in multiple places? Without a solution we would have to copy/paste that snippet, and if it evolves over time we would need to duplicate our effort (both writing and unit testing) across all copies of it.

Solutions to this wouldn't necessarily need to be complicated. A very simple approach would be to use environment variables within the snippet:

[transform.foo]
  type = "foo"
  inputs = ["${QUACK_SOURCE}"]

[transform.bar]
  type = "bar"
  inputs = ["foo"]

[transform.${WOOF_SINK}]
  type = "baz"
  inputs = ["bar"]

And if we implement the table based import syntax we can add the ability to set custom environment variables for each import:

[sources.foo_source]
  type = "file"
  include = ["/var/log/nginx/*.log"]

[[import]]
  path = "./snippets/quack_to_woof.toml"
  [import.environment]
    QUACK_SOURCE = "foo_source"
    WOOF_SINK = "woofs"

[sinks.bar_sink]
  type = "console"
  inputs = ["woofs"]
  target = "stdout"

[binarylogic]

Thanks, @Jeffail, that's definitely an interesting and creative approach. To keep this issue focused, I opened https://github.com/timberio/vector/issues/1061 which represents the "snippets" idea. I think this is separate from including multiple configuration files which we can solve via the config flags proposed in https://github.com/timberio/vector/issues/832#issuecomment-542698801

[binarylogic]

I'm closing this in favor of https://github.com/timberio/vector/issues/1445. Let me know if I'm missing anything, but I think the approach outlined there is simpler and does not require a specicial include directive.