Kube-prometheus: mixins dependency error on release-0.4

[yogeek]

Hi,

it seems that a regression has been introduced and that there is a problem in this dependency :
https://github.com/coreos/kube-prometheus/blob/release-0.4/jsonnet/kube-prometheus/jsonnetfile.json#L21
The release-0.4 of kube-prometheus points to the release-0.3 of kubernetes-mixin

As a consequence, after an update of our kube-prometheus installation yesterday, the API Server dashboard does not contain any data anymore (because it is now incoherent with the prometheus rule for API Server which are missing in the 0.3 release).

[dgrisonnet]

Hi @yogeek, before yesterday's changes, we were using kubernetes-mixin release-0.3 first commit.
https://github.com/coreos/kube-prometheus/pull/550/files#diff-f96b17d469de00af43d490ee389b9203

The recent changes consist of updating the release-0.3 dependency and pinning the version properly in jsonnetfile.json to avoid breaking Kubernetes compatibility when updating the kubernetes-mixin dependency as its release-0.4 and master branches requires Kubernetes 1.18+.

If you want to use rules from kubernetes-mixin release-0.4 or master, I would suggest you to use kube-prometheus version compatible with them. Respectively release-0.5 and master.

[yogeek]

Hi @dgrisonnet and thank you for your help

I currently have K8S 1.16.6
So according to the matrix compatibility, I installed kube-prometheus release-0.4.

You are saying that the kube-prometheus 0.4 release is not compatible with mixins 0.4 release ?

So in my case I must have :
K8S 1.16.6
Kube-prometheus 0.4 (pointing to mixins 0.3)
?

Sorry but it is not really intuitive ^^

[dgrisonnet]

You are saying that the kube-prometheus 0.4 release is not compatible with mixins 0.4 release ?

Yes, it is not compatible because of the following PR: https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/408. But kubernetes-mixin release-0.4 seems to be targeting Kubernetes 1.18+ compatibility anyway.

In the end, kube-prometheus and kubernetes-mixin are two standalone projects, we never wanted to reflect kube-prometheus versions with kubernetes-mixin ones.
However, what we generally try to do in this project is to always have master pinned to kubernetes-mixins master branch so that we can pin new kube-prometheus release to the most recent kubernetes-mixin release.

[yogeek]

Just to be sure to understand : with my K8S 1.6.6 cluster, I have to use kube-prometheus release-0.4 and as a consequence the kubernetes-mixins release-0.3. Right ?

The thing is I currently deploy grafana K8S dashboards independantly by calling these commands :

jb init
jb install github.com/kubernetes-monitoring/kubernetes-mixin
jsonnet -J vendor -m dashboards_out -e '(import "../mixins.jsonnet").grafanaDashboards'

and with this, I have errors in the API Server dashboard because the dashboards from the master branch of kubernetes-mixins are not coherent with the apiserver prometheus rules installed with release-0.4 kube-prometheus (the one from release-0.3 mixins)

So in my case, I must change the installation of k8s dashboards to use the release-0.3 mixins like this :

jb install github.com/kubernetes-monitoring/kubernetes-mixin/@release-0.3

Can you confirm please ?

[dgrisonnet]

Just to be sure to understand : with my K8S 1.6.6 cluster, I have to use kube-prometheus release-0.4 and as a consequence the kubernetes-mixins release-0.3. Right ?

That's right.

So in my case, I must change the installation of k8s dashboards to use the release-0.3 mixins like this :

jb install github.com/kubernetes-monitoring/kubernetes-mixin/@release-0.3

Exactly :slightly_smiling_face: . This way your dashboard will be installed with the rules from release-0.3 and thus be compatible with kube-prometheus.

[yogeek]

@dgrisonnet ok thank you very much for your help !

[yogeek]

@dgrisonnet sorry but another question :sweat_smile:
Just tested and my API Server dashboard is still empty because of the job name apparently :

in the mixins release-0.3 dashboard, we have job=kube-apiserver
whereas the job name in Prometheus (deployed from kube-prometheus release-0.4) is job=apiserver
(if I edit the apiserver dashboard and replace job=kube-apiserver by job=apiserver in the requests, it works)

Any idea of an explanation please ?

[dgrisonnet]

No problem, you are very welcome to ask questions :slightly_smiling_face:.

Any idea of an explanation please ?

In your case, you are generating Prometheus and the dashboards separately with two different configurations which lead to some mismatches. The job label values are being set using the kube-prometheus configuration whilst your dashboard targets are generated based on kubernetes-mixin selectors. This means that your dashboard is targeting job=kube-apiserver while the label from kube-prometheus is set to job=apiserver.

In order to fix that, you can follow this kubernetes-mixin customization guide and add the selectors that we are using in kube-prometheus.

[yogeek]

@dgrisonnet Thanks, I did not find where the kube-prometheus configuration was done !
I just customize my mixins and it works well :+1:

But, out of curiosity, as kube-prometheus depends on kubernetes-mixins, why having different job name values in kube-prometheus configuration and kubenetes-mixins selector ?

[dgrisonnet]

But, out of curiosity, as kube-prometheus depends on kubernetes-mixins, why having different job name values in kube-prometheus configuration and kubenetes-mixins selector ?

Unfortunately, I don't know the exact details behind this choice, maybe @brancz can give us some insights about it :slightly_smiling_face:.

[brancz]

There are other users of the kubernetex-mixin, and it started outside of the kube-prometheus project.

[yogeek]

ok thanks for the details @brancz
Thank you both for you help. I think we can close the issue now