Vector: Configuration unit tests

[LucioFranco]

As a user I want tools that I can use to test vector locally without much setup, this may mean creating a cli tool to submit and accept logs from vector. This should allow the user to play around with different sources, transforms and sinks locally with minimal setup.

[Jeffail]

Dumping some thoughts before writing specs out.

Goals

Ability to complement configs with unit tests

I think we should aim for a solid specification for unit testing Vector configs, something that we can recommend as best practices for maintaining large deployment projects.

Simplified execution

It should be simple to run a group of unit tests. Something like vector --test ./configs, maybe expanding to vector --test ./configs/**/*.toml and/or vector --test ./.... Our spec therefore needs to make the relationship between test definitions and their target config files explicit.

Tests should not need to cover an entire config

It will overwhelmingly be the case that users want to test the interactions between transforms with their tests, we therefore need to accommodate that. However, Vector makes it very easy to express rather complex graphs of components and the more complex a system being unit tested is the more brittle the tests become.

Our spec should allow tests to be defined for an explicit subset of a config, allowing users to break them down into logical groups and test them in isolation.

Tests should be easy to find

The relationship between a config and its test definitions should be clear to a user editing that config. We want to make it easy for large teams to maintain their Vector configs.

For example, we could enforce that test definition files are named according to the config file they are testing (foo.test.toml to your foo.toml), or we could enforce that tests are defined at the bottom of the config file they are designed to test.

[Jeffail]

Proposal 1

DISCLAIMER: This is only a partial proposal and likely to get fleshed out over time.

Unit tests for a config file are defined within the config file itself at the bottom. Each test allows you to specify one input event, to be injected in the place of a component (as a pseudo source), and >1 output events, each to be expected from a component in order for the test to pass.

[sources.foo]
  type = "kafka"
  bootstrap_servers = "10.14.22.123:9092,10.14.23.332:9092"
  group_id = "consumer-group-name"
  topics = ["topic-1"]

[sources.buz]
  type = "some_other_thing"

[transforms.bar]
  type = "something"
  inputs = [ "foo" ]

[transforms.baz]
  type = "something_else"
  inputs = [ "bar", "buz" ]

[sinks.qux]
  type = "kafka"
  inputs = ["baz"]
  bootstrap_servers = ["10.14.22.123:9092", "10.14.23.332:9092"]
  encoding = "json"
  key_field = "TODO"
  topic = "topic-1234"

[[test]]
  name = "foo test"

  [test.input_event]
    insert_at = "foo"
    type = "log"
    [test.input_event.log_fields]
      foo = "bar"
      baz = "buz"

  [[test.output_event]]
    expect_from = "bar"
    type = "log"
    [test.output_event.log_fields]
      foo = "bar"
      baz = "buz"

  [[test.output_event]]
    expect_from = "baz"
    type = "log"
    [test.output_event.log_fields]
      foo = "bar"
      baz = "buz"

[[test]]
  name = "bar test"

  [test.input_event]
    insert_at = "buz"
    type = "metric"
    [test.input_event.metric]
      type = "counter"
      name = "foocounter"
      val = 1
      timestamp = "TODO"
      [test.input_event.metric.tags]
        tag_one = "foo"
        tag_two = "bar"

  [[test.output_event]]
    expect_from = "baz"
    type = "log"
    [test.output_event.log_fields]
      foo = "bar"
      baz = "buz"

Problem: Test input data

Users need to be able to define their input data when writing tests, what should that look like? This ties in with https://github.com/timberio/vector/issues/704. For now I've mocked up a spec that allows explicit event typing, but I think most users would want to simply specify lines of raw data like this:

[[test]]
  name = "baz test"
  insert_at = "bar"
  input = '{"field":"this is way simpler"}'

  [[test.output_event]]
    expect_from = "baz"
    value = '{"field":"but Im not sure of the feasibility of this"}'

[binarylogic]

Thanks, @Jeffail! These are good proposals and agree with all of your goals 👍 . I also agree that the original example is a little verbose, the JSON example is much nicer.

One comment:

Users need to be able to define their input data when writing tests, what should that look like? This ties in with #704.

For all intents and purposes the user should think the event is nested. The fact that we flatten it internally should not be exposed to the user.

Onto the testing design. I find proposal 1 to be rather verbose. Most of that is probably due to defining the test input data (which you pointed out). What do you think about this?

Proposal 2

# 
# Pipeline
#

[sources.in]
  type = "stdin"

[transforms.parse]
  type = "tokenizer"
  inputs = [ "info" ]
  field_names = ["timestamp", "level", "message"]

  [transforms.parse.types]
    timestamp = "timestamp|%Y-%m-%dT%H:%m:%sZ"

[sinks.out]
  type = "console"

#
# Tests
#

[tests.parse]
  # Sytax is {input|output}.<component-id>.<type>.<field-name>
  input.parse.log.message = '2019-10-19T12:22:22Z debug "I have the best words"'
  output.parse.log.timestamp = 2019-10-19T12:22:22Z
  output.parse.log.level = "debug"
  output.parse.log.message = "I have the best words"

# Another example demonstrating different assertions
[tests.conditions]
  input.parse.log.message = '2019-10-19T12:22:22Z debug "I have the best words"'
  output.parse.log.message.ne = "Does not equal this"
  output.parse.log.message.starts_with = "I have"
  output.parse.log.timestamp.gt = 0

# Another example demonstrating asserting the entire event
[tests.full_document]
  input.parse.log.message = '2019-10-19T12:22:22Z debug "I have the best words"'
  output.parse.log = '{"timestamp": "2019-10-19T12:22:22Z", "level": "debug", "message": "I have the best words"}'

This is just a _rough_ example. I realize there are many more edge cases we need to cover, but this is a starting point for a more succinct syntax.

CLI

In terms of the CLI. I think this will be fairly straight-forward:

vector test

Targeting specific test(s):

vector test --test=parse --test=conditions

Targeting specific components:

vector test --component=parse

[lukesteensen]

I like Proposal 1 and am not too worried about being verbose at this point.

I do think a lot of these tests will involve some kind of parsing, so it will likely be common to supply only a raw log message as input. We can probably special case that to avoid the extra nesting under input_event.log_fields.

In general, I think there's plenty of work to be done just getting this working. We can iterate on the UI and come up with a more succinct representation once we have some experience writing these tests.

[binarylogic]

In general, I think there's plenty of work to be done just getting this working. We can iterate on the UI and come up with a more succinct representation once we have some experience writing these tests.

Agree. I'm fine with proposal 1. Although, it doesn't hurt to think through the interface some before getting started. Are there any specific reasons you like proposal 1? And I would like to agree on an actual example before getting started. Ex: one with JSON input data.

[lukesteensen]

Are there any specific reasons you like proposal 1?

I just think the deep nesting of things like output.parse.log.message.starts_with is a little dense and maybe not as discoverable. Not a strong preference either way though. I'm also hoping that starting with raw inputs will make proposal 1 less verbose to start with.

[binarylogic]

Cool. A few changes that I think we should consider:

1. Assigning unique IDs to tests. This will make it easier to reference and run individual tests.
2. The ability to supply raw input.
3. Thinking about different assertion operators (not equal, greater than, begins with, etc)

@Jeffail do you want to take a crack at a modified version of proposal 1? I'd definitely like to see 1 and 2, 3 is optional if we can work it in.

[Jeffail]

I think I prefer defining tests as an array of tables rather than a table of tables since it makes the nested tables shorter to define (shortens [tests.do_some_foo.input] to [[test.input]]). Each table then needs to specify a unique name for referencing and we can enforce uniqueness during parse time. It's only a minor difference and I'm happy to concede on this though.

Redefined Proposal 1

Input Definition

First thing to expand on proposal 1 is to simplify the input definition. I propose here that we have a type field supporting either "log", "metric" or a default value "raw". We allow users to omit the field entirely, in which case we accept a raw message string in a field value:

[[test]]
  name = "foo test"

  [test.input]
    insert_at = "foo"
    value = '{"field":"but Im not sure of the feasibility of this"}'

This allows users to define explicit types using the type field when necessary:

[[test]]
  name = "foo test"

  [test.input]
    insert_at = "foo"
    type = "log"
    [test.input.log_fields]
      message = '{"field":"but Im not sure of the feasibility of this"}'
      foo = "bar"
      baz = "buz"

Output Conditions

The second expansion on proposal 1 is the output conditions. I'm still thinking through this one as it's easy to write a spec for conditions when the output will _always_ be one event. However, there's a range of ways in which we can end up with >1 events at the end of a topology execution. That could be something like the log_to_metric transform, or maybe a user is testing a diamond shaped topology.

I solved this in Benthos by allowing users to define an array of output condition objects: https://docs.benthos.dev/configuration_testing. However, this relies on a deterministic ordering of the output events, which I'm not sure we can support with Vector.

A solution might be to define general conditions that aren't specific to a singular event. So your standard content_equals condition might actually be interpretted as at_least_one_event_equals. The spec for a condition would be pretty simple, each one being a table in an array with a type:

  [[test.conditions]]
    type = "log_field_equals"
    key = "message"
    value = "foo bar baz"

  [[test.conditions]]
    type = "log_field_contains"
    key = "something_else"
    value = "ashrules"

If any condition fails then the test fails (logical AND). We might want to open up the possibility of defining some conditions chained as an OR. Considering the case where we have >1 resulting events we might want to be able to specify that a group of conditions should only pass if they all pass on a _single_ event.

More thoughts needed on this. I'm going to step away and come back again.

[Jeffail]

Since the concept of implementing conditions within Vector crosses over with https://github.com/timberio/vector/issues/882 I think it makes sense to at least try and solve them both at the same time.

We could do this by implementing shared condition types within the scope of this ticket and releasing config unit tests as an experimental feature to start with. That allows us to play around with conditions until we're happy and then stabilize it for filter transforms.

I'd like to try and implement a spec for unit tests that looks like this:

[[test]]
  name = "foo test"

  [test.input]
    insert_at = "foo"
    value = '{"field":"but Im not sure of the feasibility of this"}'

  [[test.outputs]]
    extract_from = "bar"
    [[test.outputs.conditions]]
      type = "check_fields"
      "status.eq" = 500
      "bar.contains" = "giraffe"

And also supports this:

[conditions.is_thing]
  type = "check_fields"
  "status.eq" = 500
  "bar.contains" = "giraffe"

[[test]]
  name = "foo test"

  [test.input]
    insert_at = "foo"
    value = '{"field":"but Im not sure of the feasibility of this"}'

  [[test.outputs]]
    extract_from = "bar"
    conditions = [ "is_thing" ]

[binarylogic]

I'm happy with that. @lukesteensen what do you think?

Regarding the quoted conditions ("status.eq"), couldn't we just do status.eq? That would actually create a map. Just pointing that out in case you find that cleaner.

[lukesteensen]

That looks good to me. This feels like something we'll want to iterate on a bit once we have an initial implementation. We can probably dog food it by converting some transform unit tests to use it.