Kube-state-metrics: Accessing node labels

[andrewklau]

Is there plans to support node labels?

ie. I want to get the cpu requests of all pods and then filter it by nodes that are in the region=example

Kubernetes metrics like container_memory_usage_bytes provide these labels but not the ones provided by kube-state-metrics

[andyxning]

@andrewklau Currently, node supports labels but it is contained in node metrics.

IIUC, you mean that you want to add node labels to pods running on it?

Kubernetes metrics like container_memory_usage_bytes provide these labels but not the ones provided by kube-state-metrics

Can you please give an example about container_memory_usage_bytes.

[andrewklau]

Correct.

Here is an example that I get from container_memory_usage_bytes

container_memory_usage_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="r4.xlarge",beta_kubernetes_io_os="linux",class="scheduled",container_name="POD",exported_name="k8s_POD.4ebadca1_node-exporter-primary-component-8wjw6_ops_a9c85c03-31fa-11e7-a1fe-02b4fa6cfe03_1cb56592",failure_domain_beta_kubernetes_io_region="ap-southeast-2",failure_domain_beta_kubernetes_io_zone="ap-southeast-2a",id="/system.slice/docker-957456e739a9782e3cf4dec6c7051f43195dd6eab865902c86d42ceff34b7b59.scope",image="openshift/origin-pod:v1.5.0",instance="ip-10-0-101-125.ap-southeast-2.compute.internal",instance_type="r4.xlarge",job="kubernetes-nodes",kubernetes_io_hostname="ip-10-0-101-125.ap-southeast-2.compute.internal",name="i-0f864105ac89c3d07",namespace="ops",pod_name="node-exporter-primary-component-8wjw6",provider="aws",node_type="support"}

In this example, I am able to export these metrics to be graphed by the node label node_type

[andyxning]

@brancz Does prometheus supports cross join between different metrics? All these info has been collected in different metrics. I think we do not need to duplicate these info. :)

[brancz]

Yes it does, that's the whole point of the kube_pod_labels metric. The label-set represents the identity of the time-series, therefore when labels change it is a new/different time-series. Therefore we want to keep all labels out of the metric that are not there for identifying the particular time-series. So yes, it is unnecessary to duplicate this information, as we can then based on the pod/container identifiers join the labels onto the metrics for querying or futher information.

[andrewklau]

Is there any Prometheus query examples for joining these labels?

[brancz]

A simple example would be to join the kube_pod_labels metric onto the kube_pod_info:

sum(kube_pod_info) by(pod,namespace) * on(pod, namespace) group_right() kube_pod_labels

[andrewklau]

Thanks!

[vsinha]

I'm not sure I'm understanding this correctly @brancz:

Using the example-app (4x replica) example from the prometheus operator setup, I end up with 2 prometheus services, namely prometheus-k8s.monitoring and prometheus-frontend.default

I can execute the label join query above ( sum(kube_pod_info) by(pod,namespace) * on(pod, namespace) group_right() kube_pod_labels ) on the prometheus-k8s metrics but not on prometheus-frontend as this instance isn't monitoring the other container components other than the example-app instances.

As I understand it, I have to connect grafana to prometheus-frontend as a datasource in order to get at the custom metrics (ie demo_cpu_usage_seconds_total) exposed by example app. But this means I don't have the kube_pod_labels etc available to join on.

Where am I going wrong? Also is this the wrong space to continue this discussion?

[brancz]

You do need to have all the metrics in a single prometheus server to join metrics.

[Asgoret]

A simple example would be to join the kube_pod_labels metric onto the kube_pod_info:

sum(kube_pod_info) by(pod,namespace) * on(pod, namespace) group_right() kube_pod_labels

Can you explain more information case, please? For example, I need information about CPU usage on master nodes in avg 1m. In old cases, this query was fine: sum (rate (container_cpu_usage_seconds_total{id="/",role="infra"}[1m])) / sum (machine_cpu_cores{role="infra"}) * 100 but now it doesn't work because there is no any roles anywhere instead kube_node_labels. And I can't map by instance or node name. because the instance has a port in it, and node name doesn't see in some functions.