Vector: Deploy Vector as an aggregator in Kubernetes

[binarylogic]

v1 of our Kubernetes integration dealt with deploying Vector as a daemon on everyone node to collect app/service logs, but it did not cover deploying Vector as an aggregator (service). We should extend our Helm charts to offer this as an option, as well as updating our documentation.

[spencergilbert]

I'm looking at replacing my existing logstash statefulset that reads from kafka, writes to elasticsearch. I'd be happy to upstream that chart once I have it working.

[MOZGIII]

Here's the plan for restructuring the Helm Charts that I came up with.

First of all, the Helm charts will form this dependency graph:

Diagram source

@startuml Vector Helm Chart Model v2

hide circle
hide members

class "vector-shared"
note left
Common templates.
Library chart, non-installable.
end note

class "vector-agent"
note left
Collect logs from nodes.
end note

class "vector-aggregator"
note right
Aggregate data from
agent vectors.
end note

class "vector"
note left
A facade for single-command
deployment of all roles.
end note

"vector-agent" -[hidden]> "vector-aggregator"

"vector-agent" --> "vector-shared"
"vector-aggregator" --> "vector-shared"

"vector" --> "vector-agent"
"vector" --> "vector-aggregator"

@enduml

This structure enables the flexibility for the advanced users to deploy only the necessary subset of vector components, but also provides an option of deploying all vector components with one command/release.

Some additional design details:

• all components are deployed to the same namespace (vector by default);
• each component has it's own RBAC/SA for fine-grained control;
• charts configuration will be designed to be compatible and composable, such that you can use the single-entry-point vector chart, but customize the dependencies (vector-agent and vector-aggregator) - either via global options, or individually, depending on user needs;
• all charts can share common template functions (helpers) via vector-shared (which will be a library chart).

It is easy to add new components later on, and the extensibility of this design should be enough for the foreseeable future.

We'll be able to, later on, rely on this single-entry-point vector to implement #3008.

[jamtur01]

Is this done? I see it's in Review/QA?

[MOZGIII]

It is in review, and we're waiting for feedback from some of the early adopters. The plan is to merge this not before the 0.11.

[spencergilbert]

@MOZGIII as promised, went through replacing my existing handrolled implementation and identified a few things!

1. template/service.yaml and template/service-headless.yaml create an extra line break under ports key. 7 | {{- if .Values.vectorSource.enabled -}} I think adding hyphen at the end of this template (ln 7, templates/_helpers.tpl should resolve this.
2. vectorSource.enabled: false generates invalid service resource, since there are no ports configured. We could solve this by wrapping entire service.yaml with a conditional checking for vectorSource.enabled or service.ports being present.
3. Add a hash of the vector-aggregator ConfigMap in the StatefulSet template annotations to force redeployment of Vector when the config changes.
4. I was having trouble specifying the managedPVC, thought it might be related to ArgoCD

Persistentvolumeclaim "vector-aggregator-data-dir" not found
vector-aggregator-data-dir-vector-aggregator-0 created

1. Enabling both rbac + psp appears to break the chart?

❯ helm template . --set rbac.enabled=true --set psp.enabled=true
Error: template: vector-aggregator/templates/podsecuritypolicy.yaml:17:16: executing "vector-aggregator/templates/podsecuritypolicy.yaml" at <not>: wrong number of args for not: want 1 got 0

[MOZGIII]

I'm happy to update that all the feedback was addressed, except for extra linebreaks (because doing this properly in Helm, while possible, would be too much boilerplate and not worth it). Thanks @spencergilbert!

I wonder what's wrong with the managed PVC, helm template . --set storage.mode=managedPersistentVolumeClaim --set storage.managedPersistentVolumeClaim.storageClass=- --debug output looks ok. We can chat about it, it might've been a fluke, or might as well be an actual issue with our config.

[spencergilbert]

I'll see if I can replicate...

[spencergilbert]

@MOZGIII still fails when using the managedPVC - I tested your command above with both my existing cluster and a kind cluster I just started locally. I'm not sure what, but there's definitely something weird there.

[MOZGIII]

Can you send the output of kubectl describe pod <vecotr pod>?

I guess we'd ultimately want to add E2E tests for this, I'll have to think about how we can do it in a portable way...

[MOZGIII]

@spencergilbert if you don't mind, let's move the implementation-related discussion to the PR - https://github.com/timberio/vector/pull/4404