Vector: Proposal: update how configmap is populated

[spencergilbert]

Current Vector Version

vector 0.11.0 (g50a9c3a x86_64-unknown-linux-musl 2020-09-10)

Use-cases

With the currently implemented (helm chart) templating of vector's config it makes using vector's builtin unit testing tricky, as you need to first template the configmap to get a usable config for vector to test against.

Attempted Solutions

Not attempted, but you could extract the configuration out of the configmap after templating in CI to run unit tests

Proposal

Updating the design to use the .Files object users would be able to unit test their configuration prior to having it deployed via kubernetes.

Similarly, for the kustomize version - a configMapGenerator could be used to the same ends

References

[MOZGIII]

This looks like a case for an external config map. I.e. if you want to unit test the ConfigMap outside of the Helm Chart - the easiest option is to keep it outside of the Helm Chart.

That said, we could, in theory, add support for unit testing the config as part of the Helm Chart via helm test, but I'm not sure how useful that is, given that by the time you deploy you'd usually have a config that's tested...

[JeanMertz]

given that by the time you deploy you'd usually have a config that's tested...

That is highly dependent on the deployment strategy used. I could see a case for having a continuous deployment strategy that includes both the Vector config and the Kubernetes config in one repo, and then deploy whatever gets merged to master.

In such a case, it _would_ be nice if the unit tests ran before deployment.

Writing this out though, I think I see what you're saying now; since the Vector config (with the unit tests) lives in the repo as well, you might as well run a separate CI pipeline to run the Vector tests and block a PR merge for a bad config. Any master deploy could then still run that same CI pipeline before deployment, and stop the deployment if it fails.

[MOZGIII]

given that by the time you deploy you'd usually have a config that's tested...

That is highly dependent on the deployment strategy used. I could see a case for having a continuous deployment strategy that includes both the Vector config and the Kubernetes config in one repo, and then deploy whatever gets merged to master.

I was talking specifically about the Helm tests - https://helm.sh/docs/topics/chart_tests/. They run after (or during) the deployment, and are supposed to tell you if the deployment is well, or not. So, I was thinking about having a Vector Pod that would unit test the effective config - but that's not really a good place (or rather time in the pipeline) for such an assertion.

Writing this out though, I think I see what you're saying now; since the Vector config (with the unit tests) lives in the repo as well, you might as well run a separate CI pipeline to run the Vector tests and block a PR merge for a bad config. Any master deploy could then still run that same CI pipeline before deployment, and stop the deployment if it fails.

Yep, that's the idea. We allow users to retain full control of the config - it can even be managed independently of the Helm Chart itself - i.e. you have a config in one repo (you test and run CI on it there) and a Helm Chart management setup in another. It is possible to configure the Helm Chart to use an external config.

On the other hand, if you want to generate the config via Helm Chart - there's no way around templating it. Or maybe I don't get the idea behind the .Files object proposal.

[spencergilbert]

I definitely want to just run vector test _not_ look to test it through helm. @MOZGIII my idea behind using .Files would be to have the vector.toml in the same repo as the helm chart, and then template it entirely into the configmap - while still allowing a separate CI job to vector test the toml

[MOZGIII]

Btw, my usual k8s/helm management setup consists of the scrips like this:

#!/bin/bash
exec helm upgrade \
  --install \
  --atomic \
  --namespace loki \
  --create-namespace \
  --values values.yaml \
  --post-renderer ../../helm-kustomize.sh \
  loki \
  loki/loki \
  "$@"

Where ../../helm-kustomize.sh is

#!/bin/bash
set -euo pipefail
cd kustomize
cat >helm-base.yaml
exec kubectl kustomize

and a kustomize/kustomization.yaml:

namespace: loki

resources:
  - helm-base.yaml
  - pvc.yaml

This allows me to apply kustomize atop of the Helm Chart - which is very useful for last-minute configuration.

It's not vector-specific, but I guess this might help in your use case - you could keep your "external" ConfigMaps at the Kustomization layer, and have it testable - and all this without keeping the local copy of the chart around!

[spencergilbert]

That looks interesting - but I'm not sure how convenient that would be to build into our current use of ArgoCD. I can draft a PR if it helps

[MOZGIII]

Yep, a PR would be great. So, you see this as another option for a config managed by Helm Chart, but not templated, and instead taken verbatim from a file that is in the same file tree as the Helm Chart? It would only make sense if you keep a local copy of a chart, right?

[spencergilbert]

Yeah... ugh - it might mostly be useful just for how we end up using charts with ArgoCD. I think it _would_ be good to "natively" support vector test but I might just need to do that myself and use the "existingConfigMap" option. I guess it also wouldn't be the _worst_ thing in the world to helm template and pull the toml out of the configmap to unit test

[MOZGIII]

Looks like ArgoCD can do it: https://github.com/argoproj/argocd-example-apps/tree/master/plugins/kustomized-helm
UPD: ...can apply kustomize atop of helm

[spencergilbert]

Looks like ArgoCD can do it: https://github.com/argoproj/argocd-example-apps/tree/master/plugins/kustomized-helm

:+1: yeah - sorry I was unclear, it's doable - but I'm hesitant to add more different ways to configure deploys if I don't have to (and try and fit things into our existing flows)

EDIT: though we have had some some times that we've wanted to do some "last mile" adjustments that doing it that way could be useful

[MOZGIII]

I gave it some thought, and I think implementing though would be valuable. I wonder if it's possible to do correctly though - I wonder if the execution order of templates is deterministic or will it cause issues.

[spencergilbert]

Definitely could be a nice feature if we can figure a good way of doing it. At this time I'm building the vector.toml directly and creating a pre-existing configmap for the chart to reference