Vector: Collect `host_metrics` in Kubernetes

[binarylogic]

Vector aims to be the single agent to collect, process, and send all observability data. In order to deliver on this claim in a Kubernetes cluster, Vector must collect host-level metrics to avoid the need to deploy the Prometheus node_exporter agent.

Why?

Collecting metrics for everything on a K8s Pod can be a messy endeavor. In a perfect world, everything on the Pod exposes a /metrics endpoint, with perfectly named labels, that Prometheus auto-discovers and scrape on an interval. Unfortunately, it is often not this simple:

1. Legacy apps might not expose a Prometheus endpoint (ex: Statsd), requiring app-level changes or a utility to convert Statsd to Prometheus (Vector's job!).
2. Operators might want to derive metrics from logs or events. Again, Vector's job.
3. Vector can simplify Prometheus metrics ingestion by exposing a _single_ endpoint that Prometheus scrapes instead of multiple.
4. Finally, because of point 3, operators have a central place to augment, normalize, and control their metrics data before it leaves the Pod. This gives them control to reduce cost and protect against bad data (ex: the tag_cardinality_limit transform).

Requirements

• [ ] Enable the new host_metrics source in K8s deployments by default.
• [ ] Allow operators to opt-out of this if desired.
• [ ] Make it clear in the docs that this is enabled and that exporters like the node_exporter and statsd_exporter are not needed.

[MOZGIII]

It might be tricky to collect host metrics from a container environment. We might need a dedicated role for this, also a DaemonSet, but with less cgroup namespaces or direct access to the host namespaces /proc//sys fs.
In other words - before we do this, we need to make sure our host_metrics supports operating in the container.

[jamtur01]

The heim crate we use can detect if it's running in a container (https://docs.rs/heim/0.0.10/heim/virt/index.html) so that's a start - we can expand that support if required. I don't know if we'd want to get host metrics specifically (or at least use the current host_metrics source) or rather rely on adding a storage adapter to something like cadvisor (https://github.com/google/cadvisor/blob/master/docs/storage/README.md).

[MOZGIII]

The heim crate we use can detect if it's running in a container (https://docs.rs/heim/0.0.10/heim/virt/index.html) so that's a start - we can expand that support if required.

Well, the tricky part is that if heim reads /proc or /sys fs, we need a way to ensure it captures host metrics, and not container metrics specifically. It looks like just making sure vector properly captures host metrics in docker would be enough.

The virtualization detection is useful, but it's more like a user-visible data point, rather than something we could make use of to get access to host's /proc//sys from the container. We most likely need to passthough them via custom mount points or volumes, and maybe so some other stuff like setcap/etc, as well as, obviously, making heim use custom mount points/volumes. It might be a solved thing in heim, we just need to validate that it's covered.

[binarylogic]

What does node_exporter do? I was assuming we’d do exactly that since our intent is to replace it.

[jamtur01]

@binarylogic You wouldn't generally use node_exporter for this - https://github.com/prometheus/node_exporter#using-docker - you'd generally use something like cadvisor.

[MOZGIII]

They mount the host /sys and /proc like this:
https://github.com/prometheus-community/helm-charts/blob/3aa6446822cf292c7ed7627353e84b0bcc63c83a/charts/prometheus-node-exporter/templates/daemonset.yaml#L125-L130
https://github.com/prometheus-community/helm-charts/blob/3aa6446822cf292c7ed7627353e84b0bcc63c83a/charts/prometheus-node-exporter/templates/daemonset.yaml#L70-L75

[MOZGIII]

I'm not sure how does cadvisor relate here, @jamtur01 could you elaborate?

[MOZGIII]

If this is what you mean - we don't need to ship cadvistor manually - it's built into the kubelets now. And getting data from it would be covered by scraping the Prometheus server.

But I didn't get the storage reference - I think it's just to store the data, and we'll just be passing through our topology.

[jamtur01]

@MOZGIII Yes - if it's built-in kubelet's now then it's moot - I didn't realize they'd done that.

[jamtur01]

I just checked the heim code and it uses /proc/ and /sys and we can just do the same as node_exporter and mount these in the Daemonset.

[MOZGIII]

Yep, we can't mount as /sys as the container scoped sysfs lives there (and the runtime kind of needs it as is). We'd need to mount as /host/sys, and make heim work with that. Same with /proc. This might be tricky, as we'd need some required level of in-kernel support for this (should be there since ~3.10, but still we should probably figure out the exact details to be able to provide support for this).

Ah, and for host metrics as in metrics for apps deployed on the host (not kernel metrics) - it's correct, node_exporter doesn't do it, and we shouldn't aim to do it with host_metrics.

What does node_exporter do? I was assuming we’d do exactly that since our intent is to replace it.

https://github.com/prometheus/node_exporter#enabled-by-default
It focuses on exposing metrics that the kernel gathers on itself.

[jamtur01]

It looks like heim hardcodes the proc et al path. I'll open a ticket over there. :(

[MOZGIII]

We should create a tracking task on our end too. To validate that the end result works. Doesn't have to be in the k8s env, just that in docker it works.

[jamtur01]

Added issue: https://github.com/heim-rs/heim/issues/276

[MOZGIII]

I'll be submitting a PR to heim as part of my work on this issue.

[MOZGIII]

In addition to the inability to collect host metrics from the container environment, heim also has some limitations when used with musl - https://github.com/heim-rs/heim/issues/141. It's not clear if it affects our functionality, but we'll have to keep it in mind to avoid unimplemented! panics.