Cadvisor: Prometheus integration

[beorn7]

After our VC discussion, I thought I open an issue here to track the options and efforts for Prometheus integration.

I have looked a bit at the cadvisor code, and it seems to me that it should be relatively easy to (optionally) export a sub-set of metrics directly by the cadvisor binary (via an additional HTTP endpoint) using https://github.com/prometheus/client_golang . The sub-set should be defined by metrics that fit well into the Prometheus data model and are interesting to have available together with metrics from instrumented applications running in the same cluster. Technically, you would probably implement a prometheus.Collector that takes info.ContainerInfo objects and extracts Prometheus metrics from it on scrape time (in the Collect method). I'm happy to assist with questions, or I could provide a prototype implementation. I think, this approach makes more sense than implementing and running a separate binary that scrapes cadvisor's JSON and re-exports it as Prometheus metrics.

Note that there is https://github.com/docker-infra/container_exporter already, which might serve as a source of inspiration. (Alternatively, one might simply decide to run a container_exporter in addition to cadvisor.)

I'm a bit lost when trying to figure out which parts of cadvisor are Docker specific, which are general container things, and which are Kubernetes specific. So I can't really judge what the best course of action would be here...

[vmarmol]

This approach makes sense, I agree that we'd rather export the metrics here than have a separate collector process. We've been looking to export other cAdvisor metrics (latency and usage data) through Prometheus so this lines up nicely. It looks like we can register the custom Collector as well as the other collectors we'd want to use, nice! Looking forward to trying this out :) We'll certainly be poking you with questions.

cAdvisor is primarily container-agnostic, the API exports all containers on the machine. We're able to gather stats on Docker and non-Docker containers simultaneously. We do have some custom code for Docker containers to get some extra information and export those in a separate namespace in the API (since it's a very common use).

On a slight side note, the other portion of this is in Heapster. There we'd like to collect Prometheus metrics from clients and possibly re-export them to a Prometheus backend. It seems like there is a library for the extraction (awesome!) and a custom Collector as you mentioned may help with re-exporting.

[vishh]

Is there a reason why the Prometheus server doesn't support a 'push' based
model? The reason I ask is that we can very easily have a cadvisor side-car
that can periodically push metrics to prometheus server. This is the model
we are trying to move towards.

On Tue, Feb 10, 2015 at 8:52 AM, Victor Marmol [email protected]
wrote:

This approach makes sense, I agree that we'd rather export the metrics
here than have a separate collector process. We've been looking to export
other cAdvisor metrics (latency and usage data) through Prometheus so this
lines up nicely. It looks like we can register the custom Collector as well
as the other collectors we'd want to use, nice! Looking forward to trying
this out :) We'll certainly be poking you with questions.

cAdvisor is primarily container-agnostic, the API exports all containers
on the machine. We're able to gather stats on Docker and non-Docker
containers simultaneously. We do have some custom code for Docker
containers to get some extra information and export those in a separate
namespace in the API (since it's a very common use).

On a slight side note, the other portion of this is in Heapster
https://github.com/GoogleCloudPlatform/heapster. There we'd like to
collect Prometheus metrics from clients and possibly re-export them to a
Prometheus backend. It seems like there is a library for the extraction
https://github.com/prometheus/client_golang/tree/master/extraction
(awesome!) and a custom Collector as you mentioned may help with
re-exporting.

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/502#issuecomment-73735880.

[beorn7]

@vmarmol As a minimal example for a program that scrapes a Prometheus client and does something with it, look at https://github.com/prometheus/prom2json (which reads protobuf and/or text format and spits out a similarly structured JSON - meant for people who like to use JSON query tool to process Prometheus metrics). Interestingly, it doesn't use the extraction library at all. It stays on the protobuf level, or (for the text format) it uses the text format parser, which spits out the Go protobuf objects, too. The extraction library goes one step further and converts the protobuf objects into the objects used internally by Prometheus (namely model.Samples), which might or might not be what you want.

@vishh Push vs. pull is one of those loaded discussions. See our co-developer Brian Brazil's nice take on it. We'd really like to not mix up the two concepts (as that might easily end up in a 'worst of both worlds' situation). We might have some kind of bulk sample upload at some point, but that would be for backfilling purposes and not for (near) real-time monitoring. The bridge between the push and pull world is the Pushgateway, but it's really meant as a last resort if you really can't expose metrics with a pull semantics at all.

[discordianfish]

@vmarmol Let me know if I can help. I've wrote the prometheus container exporter: https://github.com/docker-infra/container_exporter and would be happy to deprecated it in favor our prometheus support in cadvisor since using libcontainer in a 'engine agnostic' way isn't trivial and the api breaks every other day..

[vmarmol]

We'd certainly be happy to get the help :) My thinking was that we'd implement a custom collector that would get the information from all containers and synthesize it into metrics. The hope was to be able to do most of this by introspection so that we wouldn't have to change the code each time we changed a field in ContainerStats.

WDYT?

[discordianfish]

@vmarmol Yes, that sounds good. Can't say much about the cavisor internals (introspection or not) but from a prometheus perspective implementing a Collector would be the right way (similar to container_exporter).

[vmarmol]

I just did one for Kubelet in Kubernetes and was surprised at how easy it was :)

For cAdvisor, with a reference to the manager you can call SubcontainersInfo:

m.SubcontainersInfo("/", ContainerInfoRequest{NumStats: 1})

That should return the latest stats for all containers on the machine.

[discordianfish]

@vmarmol Ah nice! Ok, I think this all looks pretty straight forward:

• Implement a Collector which holds a reference to the manager to get the stats
• Instantiate the Collector in main.go(?) and register the metrics handler

One thing though: Does SubcontainersInfo() return all containers as a flat list or do I need to somehow recursively traversal all Subcontainers?

[vmarmol]

It will return them all as a flat list. Instantiating in main.go SGTM.

The other interesting question will be labels (for identifying Docker containers), but we can tackle that later.

[discordianfish]

Oh right, the ContainerInfo doens't include the Docker metadata. Yes, we need to solve that somehow, otherwise the metrics won't be very useful.

[vmarmol]

It does include them, but I wonder how we expose them through labels (from my experiments they are nested, but maybe that's not the case). ContainerInfo has a ContainerRef which uses the "docker" Namespace and gives the ID and name as Aliases:

https://github.com/google/cadvisor/blob/master/info/container.go#L61

[discordianfish]

@vmarmol I'm working on a PR now and we can discuss details there. Once thing that I think would be useful is to have the image name and tag as labels. That's something we probably need to add somehow.

[vmarmol]

Given that #545 was merged I believe the work here is complete :)

Feel free to open other issues if there is anything missing.

[beorn7]

\o/ And there was much rejoice...

[discordianfish]

@vmarmol I still think having the container name as label would be useful, probably also https://docs.docker.com/userguide/labels-custom-metadata/
If you think this makes sense, I can open an issue and maybe start work on this if time permits.

[vmarmol]

@discordianfish that SGTM. My only concern is how Prometheus is affected by the explosion of labels. I was under the impression that label values had to be a small set.

[juliusv]

@vmarmol AFAIU the container name wouldn't actually add extra dimensionality, just an extra label to every time series? Because there is always only one container name per container ID?

[juliusv]

And to give an impression of dimensionality thresholds in practice: once a single metric has more than tens of thousands of time series, it becomes unwieldly to work with :) But otherwise, we have Prometheus servers with millions of time series that are working fine. The limit always depends on how many instances you monitor, how many time series per instance, and how big your server is.

[vmarmol]

This is true, and glad to hear it shouldn't affect Prometheus from someone who knows :) thanks for the explanation!

[discordianfish]

Opened #688

[tim-gp]

This is a cool feature, and I will admit to being n00b, but how do I view cpu in prometheus as a percentage based on metrics sent by cAdvisor?

[beorn7]

@timpoultney CPU seconds are counted by container_cpu_user_seconds_total. if you take the rate of that over the desired time window, let's say 5min, you get relative CPU usage. That times 100 is percentage:

rate(container_cpu_user_seconds_total[5m]) * 100

[tim-gp]

Extremely helpful, thank you.

[michaeljs1990]

Super sorry to hijack this thread but it's the only place I could find any information on this. @beorn7 Would the CPU usage you wrote out be for just one core? In other words on a 10 core machine would i need to divide my 10 in order to get the actual CPU usage.

[timstclair]

container_cpu_user_seconds_total is total across all cores (based on this field in the API: https://github.com/google/cadvisor/blob/master/info/v1/container.go#L276)

[mmcelroy1982]

Does container_cpu_user_seconds_total monitor the CPU usage at the VM level, or is this monitoring the CPU use within the containers? Based on the output when I setup this metric, it appears to be monitoring the CPU use at the container level.

Is there a metric that monitors the CPU use of the diego cells?