Kube-state-metrics: Pod labels in metrics

[abinet]

Currently namespace, pod are default labels provided in the metrics.
I would like to print some additional pod labels in the metrics's output to have an ability to group on later in prometheus / grafana. Is it possible?

[brancz]

You just need to join metrics, for example to get the node label from kube_pod_info onto the resource requests metric of a container i n a pod, you can just do:

kube_pod_container_resource_requests_memory_bytes * on(namespace, pod) group_left(node) kube_pod_info

@andyxning @zouyee @mxinden as this is a request we have on a weekly basis, what do you think of adding a new page in the documentation to add examples like the above? (only for metrics within kube-state-metrics though, mixing metrics from different sources would open a whole other can of worms in terms of support questions by users)

[andyxning]

@brancz Yes, IIRC, we have discussed this in an issue. But that is talked about mix metrics from cAdvisor and kube-state-metrics. I totally agree the opinion about documenting join usage like the above.

[mxinden]

@brancz sounds good to me. As we had good experience with community contributed documentation in https://github.com/coreos/prometheus-operator, how about adding a help-wanted label?

Maybe @abinet you want to give it a shot? In case this fixed your use case.

Otherwise I can also step in.

[andyxning]

sounds good to me. As we had good experience with community contributed documentation in https://github.com/coreos/prometheus-operator, how about adding a help-wanted label?

Is the different promethues application usage documentation under prometheus-operator repo the authoritative one? If then, it's ok to add usage doc there and add a link in kube-state-metrics to point to that doc. If the answer is no, then we need to make a new directory to add this under kube-state-metrics reop.

[brancz]

I think Max was just suggesting to label this issue with help wanted and let the community get started on a documentation page like that :)

He was just mentioning the Prometheus operator as there it is a healthy source of contributions outside of the people that primarily work on it.

[andyxning]

Got it.

[abinet]

thanks guys to pointing on that. sorry for silly question.

[brancz]

Not a silly question at all as I said it's a common request, would you like to start the above mentioned docs page @abinet ? :slightly_smiling_face:

[abinet]

@brancz yes, let me try

[ravvereddy]

@brancz
I would like to configure alerts only for specific namespaces and pods with labels for example app=python-helloworld, app=youtube, datastream=mobile-phone etc..

Can you please provide one sample example for the following metrics:

Namespace = my-front-end-app
pod labels = app=python-helloworld, stream=youtube, datastream=mobile-phone

I am trying something like this but didn't worked as expected

increase(kube_pod_container_status_restarts_total{namespace=~"(my-front-end-app-.|kube-system.|)"}[1h]) > 5

The problem is i dont see the pod labels at all for kube_pod_container_status_restarts_total how can i get them so that i can configure alerts rules only for specific patterns in my case it would be that particular namespace "my-front-end-app" and in that namespaces the pods which has labels (app=python-helloworld, stream=youtube, datastream=mobile-phone)

Thanks in Advance

[glove747]

@ravvereddy It was solved in version 1.9.0-rc

func wrapStatefulSetFunc(f func(v1.StatefulSet) *metric.Family) func(interface{}) *metric.Family {
return func(obj interface{}) *metric.Family {
statefulSet := obj.(v1.StatefulSet)
metricFamily := f(statefulSet)
for _, m := range metricFamily.Metrics {
m.LabelKeys = append(descStatefulSetLabelsDefaultLabels, m.LabelKeys...)
m.LabelValues = append([]string{statefulSet.Namespace, statefulSet.Name}, m.LabelValues...)
}
return metricFamily
}

[finology]

@glove747 I'm using version v1.9.0. I don't think that version satisfies the requirement by @ravvereddy .

[finology]

@brancz
I would like to configure alerts only for specific namespaces and pods with labels for example app=python-helloworld, app=youtube, datastream=mobile-phone etc..

Can you please provide one sample example for the following metrics:

Namespace = my-front-end-app
pod labels = app=python-helloworld, stream=youtube, datastream=mobile-phone

I am trying something like this but didn't worked as expected

increase(kube_pod_container_status_restarts_total{namespace=~"(my-front-end-app-._|kube-system._|)"}[1h]) > 5

The problem is i dont see the pod labels at all for kube_pod_container_status_restarts_total how can i get them so that i can configure alerts rules only for specific patterns in my case it would be that particular namespace "my-front-end-app" and in that namespaces the pods which has labels (app=python-helloworld, stream=youtube, datastream=mobile-phone)

Thanks in Advance

Hello, have you solved your problem?

[brancz]

The strategy is the same as I described in https://github.com/kubernetes/kube-state-metrics/issues/536#issuecomment-420203467 but with the kube_pod_labels metric on the right side of the join.