kube-state-metrics breaking release aka 2.0

[brancz]

We have accumulated a number of deprecated metrics and odd behaviors that I believe may justify a 2.0 release. I'd like to take this issue to discuss whether people think this is a good idea and collect what we would potentially like to break should we do a breaking release.

Off the top of my head breaking changes I would like to do:

• [x] remove deprecated metrics https://github.com/kubernetes/kube-state-metrics/issues/975
• [x] rename black-/whitelist to allow/deny-list https://github.com/kubernetes/kube-state-metrics/issues/975
• [x] use same ports in all cases (currently the flag defaults to 80/81, but the dockerfile specifies 8080 and 8081) https://github.com/kubernetes/kube-state-metrics/issues/976
• [x] rename hpa metrics to use full horizontalpodautoscaler nomenclature, to match the rest of the exposed metrics https://github.com/kubernetes/kube-state-metrics/issues/977
• [x] rename --namespace flag to --namespaces https://github.com/kubernetes/kube-state-metrics/issues/978
• [x] remove non-identifying labels from pod metrics (eg. node labels)
• [x] ~consider renaming kube-state-metrics to kubernetes-exporter~ we decided against this one
• [x] kube_secret_metadata_resource_version, kube_configmap_metadata_resource_version and kube_ingress_metadata_resource_version expose the resource version as a string in its set of labels. This value can change often and would therefore create huge cardinality. This should be a number or not existing at all. https://github.com/kubernetes/kube-state-metrics/pull/997
• [x] rename use of "collector" in user facing things (flags, help text, etc.) to "resources", as the collector type pattern does not exist anymore. https://github.com/kubernetes/kube-state-metrics/issues/980
• [x] rename storage class labels reclaimPolicy to reclaim_policy and volumeBindingMode to volume_binding_mode. https://github.com/kubernetes/kube-state-metrics/pull/1107
• [x] require kube_*_labels metrics to have explicit lists of labels to expose passed in via a flag to prevent unnecessary high cardinality https://github.com/kubernetes/kube-state-metrics/issues/1047
• [x] reworking resource metrics to try to adhere as best as possible to Prometheus "sums of series should make sense" rule. https://github.com/kubernetes/kube-state-metrics/pull/1168

I would see a breaking release at least 3 months out there, as I would like to validate the performance optimizations independently first. Further thoughts?

@andyxning @zouyee @mxinden

[andyxning]

Agree with do a breaking release to clean up the kube-state-metrics. Actually performance optimizations is a feature. I am fine with a 2.1 release to add that feature.

rename black-/whitelist to allow/deny-list

I am fine with this change. But it seems that black-/whitelist names are also ok. What is the motivation for moving to allow/deny-list. And this apparently is an additional breaking change.

use same ports in all cases (currently the flag defaults to 80/81, but the dockerfile specifies 8080 and 8081)

Fine with this. I also think if we do this, we need to also deprecated the --telemetry-port and --telemetry-host. But what we do for adding telemetry port and host is mainly to split the metrics for kube-state-metrics itself and the metrics for Kubernetes. Merging them into one port is seems not an optimal option.

[brancz]

The performance improvements I would prefer to release in a 1.x release, and once we’re comfortable everything works correctly, we go ahead and do these breaking changes.

Renaming black/whitelist is for ethical reasons. I know it has been a used term but it doesn’t even describe well what it does, but it seems it’s not contentious :) .

I think keeping the telemetry host and port is fine, just the default port should be something other than 81.

[andyxning]

Renaming black/whitelist is for ethical reasons. I know it has been a used term but it doesn’t even describe well what it does, but it seems it’s not contentious :) .

Seems reasonable to me. allow/deny is more accurate.

I think keeping the telemetry host and port is fine, just the default port should be something other than 81.

Any suggestion about the candidates? 82 or some value else? IMHO, 81 is good enough to do this job. :)

[andyxning]

The performance improvements I would prefer to release in a 1.x release, and once we’re comfortable everything works correctly, we go ahead and do these breaking changes.

Agreed.

[brancz]

Sorry I should have been more clear about why I think we should change ports. Anything lower than 1024 requires root on Linux (or at least have the CAP_NET_ADMIN capability). That’s why default ports of kube-state-metrics should be higher than that. And beyond that, whatever we use should be consistent across the pure binary and the container.

[andyxning]

Understood clearly. @brancz

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[brancz]

/remove-lifecycle stale

[tariq1890]

I would also like to propose that we update the k8s dependencies to 1.13 at the very least. Given that it has the potential of introducing backwards-incompatible/breaking changes, it would be a good candidate for the 2.0 release.

[brancz]

Do we have known incompatibilities? So far we only have the policy to support the past 4 versions of Kubernetes, as long as we stick with that it's not so much a "breaking" release.

[tariq1890]

Nothing off the top my head. The client-go version support matrix is unfortunately not very clear in what they support/do not support. I can experiment with the latest k8s client-go dep and see how that turns out with the previous releases.

By past 4 versions of Kubernetes - Do you mean that with respect to the latest Kubernetes version out there or the latest Kubernetes version depped into kube-state-metrics?

[brancz]

By past 4 versions of Kubernetes - Do you mean that with respect to the latest Kubernetes version out there or the latest Kubernetes version depped into kube-state-metrics?

The later. Forward compatibility _should_ work, but we can't guarantee it due to client-go.

[zuzzas]

@brancz
You've raised a motion about removing kube_[object]_owner metrics with <none> labels for Kubernetes objects that lack any owners.

I'd like to object!

We've got a use case for grouping Pods and other objects that lack any controller. This logic only works on ingestion with recording rules if we have a metric that clearly specifies the lack of owners on an object. We can't solve this riddle with absent(), since it requires clearly specifying an instant vector with the required labels.

Please, do not think ill of me after looking at a screenshot of this abomination:

[dohnto]

I might sound silly, but maybe we can unify the single/plural form of options --namespace/--collectors to be more uniform?

[brancz]

@zuzzas

Please, do not think ill of me after looking at a screenshot of this abomination

Haha, if anything I have more respect! That's a pretty nice recording rule, would you mind contributing that to the kubernetes-mixin?

I think you have a fair point, let's keep it. In theory this could be done with a join as well, but feels too difficult to get right. Thanks for bringing this up!

@dohnto :

I might sound silly, but maybe we can unify the single/plural form of options --namespace/--collectors to be more uniform?

Not at all silly. The --namespace flag internally is already called "namespaces", it is only singular for backward compatibility. I'll add making it plural to the list. Thanks for the suggestion, keep them coming! :slightly_smiling_face: .

[brancz]

@sylr brought to my attention, that for a number of pod metrics, there is the node label. For consistency, these labels should not be on those metrics directly, but instead be joined onto the metrics at query time. Example: https://github.com/kubernetes/kube-state-metrics/blob/aed94850b839d111d15308c97e1db7ada8017c6f/internal/collector/pod.go#L511

[brancz]

Added "consider renaming"

[andyxning]

consider renaming kube-state-metrics to kubernetes-exporter

LGTM.

[brancz]

Added

kube_secret_metadata_resource_version, kube_configmap_metadata_resource_version and kube_ingress_metadata_resource_version expose the resource version as a string in its set of labels. This value can change often and would therefore create huge cardinality. This should be a number or not existing at all.

Thanks to @xieyanker for noticing here: https://github.com/kubernetes/kube-state-metrics/pull/777#discussion_r290661344

[lilic]

I would add to the list, renaming all the leftover user-facing occurrences of collectors to resources, as we recently removed the collectors package. That would also mean renaming collector in options to resource. Overall the --resources=pods flag would become more self-descriptive.

[brancz]

Great suggestion! Added.

[tariq1890]

Let's add the Sharding feature to the list as well.

[brancz]

Sharding isn’t breaking so I feel it can be added in a backward compatible way in 1.x or 2.x. It’s fairly close to being ready I would say though so I’d like to see it go into a 1.x release.

[bboreham]

Came here to +1 removal of high-cardinality kube_configmap_metadata_resource_version.
This one metric occupies 3% of all the data in our service.

I'm intrigued: what does anyone use this metric for, in its current form?

[lilic]

@bboreham you can also just blacklist those metrics for now until they are removed? --metric-blacklist="kube_configmap_metadata_resource_version" for example should work. :)

[tariq1890]

One more to the list: We need to rename labels that follow pascalCase instead of snake_case. We would then be able to use newer versions of Prometheus. https://github.com/kubernetes/kube-state-metrics/pull/929#discussion_r331590172

[bboreham]

I didn’t understand this comment at first, and the link didn’t work for me.

I think you mean “would be able to use a newer version of promtool” (which is used like a linter).

[tariq1890]

@bboreham yes. We now use prometheus as a go mod dependency. Since promtool is bundled in that, we would need to use newer versions of prometheus.

[brancz]

Adding renaming of storage class labels reclaimPolicy to reclaim_policy and volumeBindingMode to volume_binding_mode. (RE: https://github.com/kubernetes/kube-state-metrics/pull/939)

[zouyee]

consider renaming kube-state-metrics to kubernetes-exporter

Is it necessary to maintain a list which used the library and instructions for upgrading？

[brancz]

Not sure what you are referring to. Can you elaborate?

[zouyee]

Because there are other k8s related projects that use our repo, such as charts、kubespray, it may be necessary to provide such a maintenance list to update the information, and provide some impact information about the name modification.
Non-native english 😄

[brancz]

Doesn’t a new major release show that there are breaking changes? Of course we need to properly document these changes.

[lilic]

All coresponding issues from above were created under the https://github.com/kubernetes/kube-state-metrics/labels/v2

Google doc for v2 release that we discussed during kubecon. https://docs.google.com/document/d/1lCvbvOAVFai7ciP_heZrJ_QXLOEaFu8ZBmwVVeCwf54/edit?usp=sharing

[brancz]

One more thing that came up during kubecon: Before we do the v2 release we probably want to do another round of scalability tests. I believe Google volunteered to do this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[olivierlemasle]

/remove-lifecycle stale

[lilic]

@brancz anyone to ping regarding scalability tests? Or should we bring it up in the SIG call maybe? @tariq1890

[brancz]

SIG call sounds good

[brancz]

Looks like we've done everything on the list, let's get a pre-release started then! :)

[QuentinBisson]

Should the kube-state-metrics module path not become module k8s.io/kube-state-metrics/v2 in go.mod as well for v2?

[brancz]

You're right, we forgot about that. @omegas27 do you want to take care of that?

[QuentinBisson]

Sure :+1:

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[lilic]

/remove-lifecycle rotten

We are almost there 🎉

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[lilic]

We finished and release is cut 🎉 Thank you all!!