K6: Investigate telegraf integration in k6

Created on 27 Jun 2019 · 21Comments · Source: loadimpact/k6

After https://github.com/loadimpact/k6/issues/1060 and and https://github.com/loadimpact/k6/pull/1032#issuecomment-506349787, I think it makes some sense to investigate potentially integrating telegraf in k6. And I don't just mean sending metrics from k6 to a telegraf process, since that should currently be possible with no extra changes in k6 - telegraf has an InfluxDB listener input plugin that k6 can send metrics to.

Rather, since telegraf is a Go program with a seemingly fairly modular and clean architecture, it may be worth investigating if we can't use parts of it as a library in k6. If we can figure out a way to use it, we'd pretty much have a very universal metrics output "for free":

it has various processor and aggregator plugins
it has over 35 output plugins, a few of which we also have (influxdb, datadog, kafka), but most of which we lack
it's a very active project, but I think the base architecture would change very rarely, so I doubt we'd have much issues updating the dependency

The two main sticking points I foresee are the configuration and metric mismatches between it and k6. So, instead of a huge refactoring effort, my initial idea is to investigate if we can't just add a telegraf output in k6 that can accept any telegraf options. That is, instead of one new k6 output type per one telegraf output type, we could have a single "universal" k6 output type that could, via configuration, be used to filter/aggregate/output k6 metrics in every way telegraf supports.

This way, we don't have to refactor a lot of k6 - we can transparently convert the k6 metrics to whatever telegraf expects. The configuration would be trickier, since telegraf expects its configuration in a TOML format... And I'm not sure there's any way to change that, considering that even the simple file output just has toml struct tags and apparently that's enough, since the constructor just returns the empty struct (which I assume the config is unmarshaled into).

We can try to convert JSON to TOML, though I don't think it's worth it, since the k6 outputs can't be configured from the exported script options yet anyway (https://github.com/loadimpact/k6/issues/587). Instead, we probably should stick with the TOML config and just pass it via the CLI, like how the JSON output file is currently being specified: k6 run script.js --out telegraf=my_telegraf.conf or something like that.

Another thing we should evaluate is how big of a dependency telegraf would be. The current repo has ~200k Go LoC, but its vendor has around 5 million... I think a lot of those would be dropped, since we won't need any of its 150+ input plugins and other things, but there's still a good chance that this dependency would actually be bigger than the rest of k6 :smile: Even so, I don't think that would be a huge issue, since with the number of plugins it has, I assume that the base APIs are very stable... It's just something that we need to keep in mind, given that we vendor our dependencies in our repo (and they don't).

enhancement evaluation needed feature

Source