Kube-state-metrics: Resource Metrics for CRDs

[mrueg]

/kind feature

With the growing use of CRDs in K8s it might be useful to monitor those as well and provide some initial metrics for them.

Idea:
Have kube-state-metrics read a configuration file that allows the user to list further resources.

A list of crds to monitor:

- servicemonitors.monitoring.coreos.com
- certificaterequests.cert-manager.io

could create metrics similar to https://github.com/kubernetes/kube-state-metrics/blob/master/docs/configmap-metrics.md

kube_CRD_NAME_info
kube_CRD_NAME_created
kube_CRD_NAME_metadata_resource_version

[tariq1890]

Looks like this PR would need to be revived: https://github.com/kubernetes/kube-state-metrics/pull/515

I agree that it would be a great addition to kube-state-metrics. I am also wondering if we should take out VPA metrics and have them be monitored as CRDs. VPAs aren't considered as k8s primitive APIs.

/help

[k8s-ci-robot]

@tariq1890:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Looks like this PR would need to be revived: https://github.com/kubernetes/kube-state-metrics/pull/515

I agree that it would be a great addition to kube-state-metrics. I am also wondering if we should take out VPA metrics and have them be monitored as CRDs. VPAs aren't considered as k8s primitive APIs.

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dgrisonnet]

Is anyone currently working on this? I would be interested in implementing this feature.

[lilic]

Hey @dgrisonnet there is no real plan on the roadmap for this. I would prefer native instrumentation for CRs, while I understand its not always possible, but have my doubts on CR instance metrics being added in kube-state-metrics.

In any case I would prefer to wait for release-2.0 to be out before we do this.

[dgrisonnet]

I completely agree with you that CR should not be monitored by kube-state-metrics. However, in my opinion, Custom Resource Definition should be as it is a native Kubernetes resource.
Following what was done in #515 and @mrueg proposition, I would say that adding the following metrics seems useful:

kube_customresourcedefinition_info{group="", name=""}
kube_customresourcedefinition_created{group="", name=""}
kube_customresourcedefinition_metadata_resource_version{group="", name=""}

In any case I would prefer to wait for release-2.0 to be out before we do this.

Sure, no worries. I wasn't intending to put pressure on getting this feature in 2.0.

[lilic]

Sure, no worries. I wasn't intending to put pressure on getting this feature in 2.0.

No pressure at all! :)

Seems interesting. @mrueg @dgrisonnet what value does this bring to you, can you describe some use cases how you would use those metrics in queries, alerts or dashboards, etc.? Thanks! We like to get some use cases so we don't just add things for the sake of adding them but folks don't use it and its just extra series that are produced, hope that makes sense.

[mrueg]

@lilic
As mentioned in the initial issue post, what immediately comes to my mind are prometheus-operator's servicemonitors.
Having those metrics would allow to alert on a bad configuration if the target does not appear in the prometheus instance it should be (which for example can happen easily if you set up prometheus{,-operator} to only select servicemonitors with specific labels and the servicemonitor is missing it).

Another use case are cert-manager's CRDs where one could use the crd metrics to verify that the actual secrets including the certificate and key get created.

[lilic]

As mentioned in the initial issue post, what immediately comes to my mind are prometheus-operator's servicemonitors.
Having those metrics would allow to alert on a bad configuration if the target does not appear in the prometheus instance it should be

Not sure how kube_customresourcedefinition_created{group="monitoring.coreos.com", name="servicemonitors"} 1603447067, which just gives you when the custom resource definition was registered with the API, can give you that info? Can you explain a bit more, not saying its a bad idea just want a bit more details. You planning on combing that with another metrics for example? I think for this you would need CR instance metrics, no?

[brancz]

The prometheus operator already exposes metrics about it's CRs itself, which is how I believe it should be. Certmanager should do the same.

[mrueg]

As mentioned in the initial issue post, what immediately comes to my mind are prometheus-operator's servicemonitors.
Having those metrics would allow to alert on a bad configuration if the target does not appear in the prometheus instance it should be

Not sure how kube_customresourcedefinition_created{group="monitoring.coreos.com", name="servicemonitors"} 1603447067, which just gives you when the custom resource definition was registered with the API, can give you that info? Can you explain a bit more, not saying its a bad idea just want a bit more details. You planning on combing that with another metrics for example? I think for this you would need CR instance metrics, no?

Apologies, should have read more closely. yes I was thinking about instance level metrics.

The generic ones could be interesting to provide info when a CRD was updated (unless the tools using them already provide that info)

[lilic]

Apologies, should have read more closely. yes I was thinking about instance level metrics.

Not sure we would do instance level metrics at this points for custom resources, but if you come up with a mini proposal on the pros and cons we can have a look, google doc is fine or just more detailed description in a new issue as I would like to keep this one for CRD metrics. Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[lilic]

/remove-lifecycle stale

I believe this is still valid. @mrueg do you want me to label this as help wanted or do you want to tackle the proposal part yourself? Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[grzesuav]

Was it implemented ? If not - upvoting ;)

[grzesuav]

Maybe my use-case - I have dashboards showing information about resources related to one of the operators - ConfigMap, Secrets and its leaks information about CR's

[lilic]

@grzesuav this was not implemented, it's up for grabs if you are interested in tackling it?

[grzesuav]

hi @lilic , I can try, I am not familiar with the project though, will try to look at it next weekend and ask more question about feasibility of implementation if that's ok

[lilic]

@grzesuav sounds great! Best would be to come up with just a small proposal here on the issue before you start doing a PR, might be good to sync with @mrueg who had an initial idea and wants this as well, so both requests are answered. Thanks again! 🎉

[grzesuav]

just one thing @lilic , as I noticed kube-state-metrics uses typed API, is it ok to focus on CRD v1 introduced in k8s 1.16 and omit v1beta1 ?

[lilic]

Yes that sounds great! 👍

[grzesuav]

hi, I have question related to implementation, I have some easy draft here - https://github.com/kubernetes/kube-state-metrics/pull/1517.

The challenge I currently have is that typed client for CRD is in k8s.io/apiextensions-apiserver v0.21.0 package, also its interface is bit different :

type func(kubeClient "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset".Interface, ns string) "k8s.io/client-go/tools/cache".ListerWatcher)

than

func(kubeClient "k8s.io/client-go/kubernetes".Interface, ns string) "k8s.io/client-go/tools/cache".ListerWatcher

therefore I have some questions :

1. is there nice way to abstract this ? The problem I see it that whole design here heavily uses typed builder, therefore unsure what would be the best way - create internal client interface and make both clienset interfaces compatible// wrapped by it (but it would require a bit of changes with typing in many places)
2. another idea which I have is to use Unstructured API to get CRD's without using apiextensions dependency - but it will require more extractions and tooling in crd package (however leaving rest of the design untouched)

additional questions :

1. for now I am adding metric for crd objects, I think it would be beneficial to have also particular cr's in, but would keep it as separate PR - to not make it too big
2. making prom labels from kubernetes labels - from what I noticed there is certain allowlist to filter which labels should be included - how does it work ? I noticed my metrics have additional labels (on the Secrets) so it is configurable ? I mean how I can extend list of labels to be attached there (on cluster)

That is all what I have, please let me know whad do you think,

Cheers