Kube-state-metrics: Failed to list *v2alpha1.CronJob: the server could not find the requested resource

[jicki]

kubectl get nodes
NAME          STATUS    ROLES     AGE       VERSION
172.16.1.64   Ready     <none>    20d       v1.8.1
172.16.1.65   Ready     <none>    20d       v1.8.1
172.16.1.66   Ready     <none>    20d       v1.8.1

ld not find the requested resource
E1106 07:09:40.573135       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource
E1106 07:09:41.574269       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource
E1106 07:09:42.575712       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource
E1106 07:09:43.576884       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource
E1106 07:09:44.578064       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource
E1106 07:09:45.581133       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource
E1106 07:09:46.582299       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource
E1106 07:09:47.583628       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource
E1106 07:09:48.584764       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource
E1106 07:09:49.586064       1 reflector.go:205] k8s.io/kube-state-metrics/collectors/cronjob.go:93: Failed to list *v2alpha1.CronJob: the server could not find the requested resource

pls help me `

[zouyee]

running apiserver with --runtime-config=batch/v2alpha1=true

[drampelt]

I'm also having this issue, running kube-state-metrics 1.1.0 on Kubernetes 1.8 on GKE. CronJobs are now enabled for GKE but according to the documentation are under the batch/v1beta1 version, not batch/v2alpha1

[Morriz]

+1

any update on this? I am not able to run kubernetes 1.8 now which is out for a while. Why not make the version configurable?

[brancz]

Yes, we'll be updating the API used by kube-state-metrics for the next release. @andyxning @zouyee could one of you take care of this?

[sorenmat]

Does this mean that it doesn't ship metrics to datadog or in general?

[brancz]

I'm confused as to what this has to do with datadog. The kube-state-metrics project aims to _only_ expose metrics in the Prometheus exposition format (which is supported by multiple monitoring projects other than just Prometheus).

[sorenmat]

Yeah, that was a strange comment. What i meant was:
Does this break the metric collection or can the error just be ignored?

[brancz]

I see :slightly_smiling_face: this only affects the CronJob metrics, everything else continues to work as expected. Each collector has it's own go routine, and if one errors, the remaining ones keep doing their work.

[brancz]

tl;dr unless you want CronJob metrics, this error can be ignored. (you can also turn off the cronjob collector :slightly_smiling_face: )

[f0]

291

[errordeveloper]

@brancz this can be closed now, a patch release would be very handy!

[brancz]

I don't think this should be shipped in a patch release. A minor release should be relatively close though. I would just like to get #302 in. Then I think we have enough actual features together to justify a minor.

[zouyee]

agree with @brancz

[denibertovic]

@brancz How does one turn off the cronjob collector?

[brancz]

You need to specify all collectors except the cronjob one in the —collectors flag.

[denibertovic]

@brancz You mean in kube-state-metrics-deployment.yaml at the bottom of the file as an argument to pod_nanny?

EDIT: err no, you probably mean for the kube-state-metrics container. What is the format of --collector flag? Is it just --collectors="col1,col2" Or do I specify the flag multiple times?

EDIT2: Found the answer to my question here: https://github.com/kubernetes/charts/blob/master/stable/kube-state-metrics/templates/deployment.yaml

[denibertovic]

For some reason when I add this and do kubectl apply again it errors out with:
"a container name must be specified for pod kube-state-metrics-59fb6c5b94-pddz6"

[brancz]

this may have to do with kubectl apply, try kubectl replace

[denibertovic]

I tried that, I also tried deleting the service, pods and deployment first ... and then just applying everything again but no go.

[brancz]

When I use the exact manifest, it works for me. Can you share in more detail what you are trying to do, and the manifests you are using to accomplish this?

[denibertovic]

@brancz This is the change I made:

--- a/kube-state-metrics/kubernetes/kube-state-metrics-deployment.yaml
+++ b/kube-state-metrics/kubernetes/kube-state-metrics-deployment.yaml
@@ -14,6 +14,20 @@ spec:
       containers:
       - name: kube-state-metrics
         image: quay.io/coreos/kube-state-metrics:v1.1.0
+        args:
+          - --collectors=daemonsets
+          - --collectors=deployments
+          - --collectors=limitranges
+          - --collectors=nodes
+          - --collectors=replicasets
+          - --collectors=replicationcontrollers
+          - --collectors=resourcequotas
+          - --collectors=services
+          - --collectors=jobs
+          - --collectors=statefulsets
+          - --collectors=persistentvolumeclaims
+          - --collectors=namespaces
+          - --collectors=horizontalpodautoscalers

The error is still:

Error from server (BadRequest): a container name must be specified for pod kube-state-metrics-7598f64994-l2dp5, choose one of: [kube-state-metrics addon-resizer]

EDIT: I forgot --collectors=pods and I removed namespaces and horizontalpodautoscalers and it works now!!!

[zouyee]

It likes that - --collectors=daemonsets, pods, ...

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[zouyee]

/close