Kube-prometheus: Kube prometheus 0.6 is using an incompatible version of kube mixin based on k8s compatibility

[SayakMukhopadhyay]

The release-0.6 branch of kube-prometheus has a dependency on kubernetes-mixin's release-0.5 branch. The 0.5 branch of kubernetes-mixin is said to be incomatible with k8s 1.18 as given here. But the release-0.6 branch of kube-prometheus is said to be compatible with 1.18. according to here.

So, one of the 3 correction should be made

1. kube prom release-0.6 should be made incompatible with k8s 1.18
2. kube prom release-0.6 should be updated to have a dependency on kube mixin release-0.4 branch
3. kube mixin release-0.5 should be made compatible with k8s 1.18

I have not done any tests as I am yet to deploy but since I came across this while going through the docs as a preparation for my deployment, I thought I would create an issue to track this.

[paulfantom]

I believe we should move release-0.6 from 1.18 to 1.19 column in the compatibility matrix.

[SayakMukhopadhyay]

Do you want me to create a couple of PRs to master and release-0.6 with this doc change?

[lilic]

I would want to know what is not compatible in mixin release branch with k8s 1.18 first. As our tests pass, but as said on slack it could be an alerting rule or recording rule' metric that changed. Otherwise sgtm

[SayakMukhopadhyay]

Yeah @lilic , even I think that the best outcome would be if kube-mixin 0.5 is compatible with k8s 1.18. But if that's not possible due to some incompatible alerting or recording rule, I am fine with using the 0.5 release of kube prometheus for now.

[dgrisonnet]

The discussion comes from: https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/480#discussion_r462981529

To quote @s-urbaniak who cut the release on the exact reason:

technically it still works with 1.18. so my suggestion to me would be: despite us knowing it works for 1.18 we only guarantee >=1.19 as we'll only fix 0.5 for 1.19 issues.

I'd say, let's do the same here. wdyt?

[paulfantom]

The compatibility matrix should reflect what we know for sure and what we can guarantee. Currently, we can guarantee* that release-0.6 branch will work with k8s 1.19. For every other k8s version, we can say it works now, but there are no guarantees it will in the future. Considering this I would prefer if we move compatibility of release-0.6 to k8s 1.19 and not invest time into finding what and if 1.18 works.

*Guarantee comes from downstream usage in OpenShift and CI tests.

[m1o1]

I just ran into this since I upgraded to release-0.6 (thought that was the newest I could upgrade to). Some dashboards like for the kubelet are using metrics like kubelet_running_pods (changed from kubelet_running_pod_count in k8s 1.19), which caused some panels to show "N/A". I'll be upgrading to 1.19 soon anyway, but yeah I agree, leaning towards not including k8s versions in the compatibility matrix if they don't have these guarantees (or marking them differently) would be helpful. At least, to whatever extent is possible with these guarantees.

[paulfantom]

Closing as release-0.6 is not compatible with k8s 1.18. If anyone would like to work on making it compatible then we are open to PRs :)