Kube-state-metrics: job label in kube_job_* metrics conflicts with the job label defined by scrape_config in Prometheus

Created on 5 Jan 2018 · 10Comments · Source: kubernetes/kube-state-metrics

All the kube_job_* metrics have a job label with the name of the Job, however this label is conflicting with the job label assigned by Prometheus per what is defined in the job_name of the scrape_config

Prometheus keeps the original value of the label job in a new label called exported_job, but this is a bit confusing.

I think it would be better to rename the job label in the kube_job_* metrics to job_name

lifecyclrotten

Source

dannyk81

Most helpful comment

Agreed, let's change the label here to job_name.

brancz on 29 May 2018

🎉1 👍1

All 10 comments

Does this also happen if honor_labels is set to true? Technically the kube-state-metrics target still has the job label, and the unique combination of labels still uniquely identify the metrics.

Nonetheless I think I agree, there are scenarios where this could become a problem, so maybe using job_name (or maybe kube_job?) is not a bad idea, as long as we do it consistently.

brancz on 8 Jan 2018

It seems that job label is a special case in Prometheus, so if the source metric has a job label it is renamed to exported_job.

As you said, uniquness of the metrics is prserved, but I would rather we use a different label name and not job.

job_name sounds reasonable to me, but I'm not married to it :)

dannyk81 on 9 Jan 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 10 Apr 2018

ping @brancz 😄

dannyk81 on 13 Apr 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot on 13 May 2018

/remove-lifecycle stale

I'm not really happy with any of the solutions, I'll have to discuss more with the other prometheus devs to see if they have any better ideas.

brancz on 14 May 2018

As a Prometheus user, for me it makes sense to use "job_name" as the label of source metric and "job" as the label of Prometheus job to do it consistently.