kubectl describe (statefulset|clusterrole|clusterrolebinding|cronjobs) fails in v1.10.0

Created on 11 Apr 2018 · 14Comments · Source: kubernetes/kubectl

What keywords did you search in Kubernetes issues before filing this one? :
kubectl describe statefulset

Is this a BUG REPORT or FEATURE REQUEST?:
BUG REPORT

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-27T00:14:31Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: AWS
OS: macOS Sierra 10.12.6
Kernel: Darwin Kernel Version 16.7.0
Install tools: brew

What happened:

$ kubectl describe statefulset <statefulset> -n <namespace>
Error from server (NotFound): the server could not find the requested resource

kubectl delete statefulset ... also fails with the same error

What you expected to happen:
The kubectl describe|delete statefulset command to succeed

How to reproduce it:
kubectl describe statefulset <statefulset> -n <namespace> on an existing statefulset in the cluster

Anything else we need to know:

kubectl get statefulsets works fine
I can run kubectl describe on other resources successfully such as pods.

It works fine in the previous kubectl version, since comparing the same command with both installations shows it:

$ /usr/local/Cellar/kubernetes-cli/1.9.2/bin/kubectl describe statefulset <statefultset> -n <namespace>
Name:                   statefulset
Namespace:         namespace
CreationTimestamp:  Mon, 09 Apr 2018 18:21:53 +0100
...

$ /usr/local/Cellar/kubernetes-cli/1.10.0/bin/kubectl describe statefulset <statefultset> -n <namespace>
Error from server (NotFound): the server could not find the requested resource

arekubectl kinbug sicli

Source

sacha-cs

👍8 ❤1

Most helpful comment

@rofc do you think kubectl could warn the user we are when this +/- 1 is not respected? Just to say that some command might fail because of the mismatch of the versions.

alecharp on 7 Jun 2018

👍4 ❤1

All 14 comments

same here with v1.8.7

rfum on 21 Apr 2018

This is still an issue with kubectl 1.10.1, as well.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:29:03Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.8", GitCommit:"2f73858c9e6ede659d6828fe5a1862a48034a0fd", GitTreeState:"clean", BuildDate:"2018-02-09T21:23:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl -n monitoring get statefulset elasticsearch-logging
NAME                    DESIRED   CURRENT   AGE
elasticsearch-logging   3         2         28d

$ kubectl -n monitoring describe statefulset elasticsearch-logging
Error from server (NotFound): the server could not find the requested resource

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T20:49:26Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.8", GitCommit:"2f73858c9e6ede659d6828fe5a1862a48034a0fd", GitTreeState:"clean", BuildDate:"2018-02-09T21:23:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl -n monitoring describe statefulset elasticsearch-logging
Name:               elasticsearch-logging
Namespace:          monitoring
CreationTimestamp:  Thu, 29 Mar 2018 17:21:19 -0700
...

ctbarrett on 27 Apr 2018

❤1

Same for _clusterroles_ and _clusterrolebindings_.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-14T06:36:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get clusterroles
NAME                    AGE
cluster-autoscaler      230d
elasticsearch-logging   179d
flannel                 278d
fluentd-es              179d
kops:dns-controller     278d
kube-dns-autoscaler     278d
redis-ha                30m```



md5-fd204a637da5bc5a9d3700301e45e3d9



$ kubectl describe clusterrole redis-ha
Error from server (NotFound): the server could not find the requested resource
rcrisial@revolver:~$ kubectl get clusterrolebindings
NAME                    AGE
cluster-autoscaler      230d
elasticsearch-logging   179d
flannel                 278d
fluentd-es              179d
heapster                278d
kops:dns-controller     278d
kube-dns-autoscaler     278d
kubeadm:node-proxier    278d
kubernetes-dashboard    278d
redis-ha                29m```



md5-fd204a637da5bc5a9d3700301e45e3d9



$ kubectl describe clusterrolebinding redis-ha
Error from server (NotFound): the server could not find the requested resource



md5-fd204a637da5bc5a9d3700301e45e3d9



$ kubectl create clusterrole redis-ha2 --verb=get,list,patch --resource=pods
Error from server (NotFound): the server could not find the requested resource



md5-2baccd3a5bf0eb3e661904e831414a09



$ kubectl create clusterrolebinding redis-ha2 --clusterrole=redis-ha --serviceaccount kube-system:default
clusterrolebinding.rbac.authorization.k8s.io "redis-ha2" created

rofc on 9 May 2018

@sacha-cs can you update issue subject to "_kubectl describe (statefulset|clusterrole|clusterrolebinding|cronjobs) fails in v1.10.0_"? Thank you in advance.

rofc on 14 May 2018

Same behaivoir with _kubectl 1.10.2_ (https://storage.googleapis.com/kubernetes-release/release/stable.txt)

$ ./kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-14T06:36:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ ./kubectl get clusterroles
NAME                    AGE
cluster-autoscaler      236d
elasticsearch-logging   185d
flannel                 284d
fluentd-es              185d
kops:dns-controller     284d
kube-dns-autoscaler     284d
redis-ha                5d



md5-fd204a637da5bc5a9d3700301e45e3d9



$ ./kubectl describe clusterrole redis-ha
Error from server (NotFound): the server could not find the requested resource

rofc on 15 May 2018

❤1

/sig cli
/kind bug

rofc on 18 May 2018

Seems the issue came with v1.10.0-beta.1, the v1.10.0-alpha.3 works fine (at least for me).. FYI:

$ ./kubectl-1.10.0-alpha.3 version
Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.3", GitCommit:"89cbdc0d6ff54dff025113c6c8f15b95abcd721c", GitTreeState:"clean", BuildDate:"2018-02-01T18:36:43Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.9", GitCommit:"19fe91923d584c30bd6db5c5a21e9f0d5f742de8", GitTreeState:"clean", BuildDate:"2017-10-19T16:55:06Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ ./kubectl-1.10.0-alpha.3 get clusterroles
NAME                    AGE
cluster-autoscaler      210d
elasticsearch-logging   188d
flannel                 210d
fluentd-es              188d
kops:dns-controller     210d
kube-dns-autoscaler     210d
redis-ha                8d

$ ./kubectl-1.10.0-alpha.3 describe clusterrole redis-ha
Name:         redis-ha
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  pods       []                 []              [get list patch]

$ ./kubectl-1.10.0-beta.1 version
Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-beta.1", GitCommit:"37555e6d24c2f951c40660ea59a80fa251982005", GitTreeState:"clean", BuildDate:"2018-03-01T06:50:06Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.9", GitCommit:"19fe91923d584c30bd6db5c5a21e9f0d5f742de8", GitTreeState:"clean", BuildDate:"2017-10-19T16:55:06Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ ./kubectl-1.10.0-beta.1 get clusterroles
NAME                    AGE
cluster-autoscaler      210d
elasticsearch-logging   188d
flannel                 210d
fluentd-es              188d
kops:dns-controller     210d
kube-dns-autoscaler     210d
redis-ha                8d

$ ./kubectl-1.10.0-beta.1 describe clusterrole redis-ha
Error from server (NotFound): the server could not find the requested resource

rofc on 18 May 2018

The problem seems to be the +/- 1 version from server..

_kubectl supports +/- 1 version from the server, so kubectl 1.7 or 1.8 would correctly describe the resource_

https://github.com/kubernetes/kubernetes/issues/64042#issuecomment-390365112

rofc on 21 May 2018

👍5

@rofc do you think kubectl could warn the user we are when this +/- 1 is not respected? Just to say that some command might fail because of the mismatch of the versions.

alecharp on 7 Jun 2018

👍4 ❤1

$ kubectl version

Make sure the kubectl version (Client) and Kubectl version of your Kubernetes Cluster (Server version) are same or compatible.

I had issues when the Client version was "1.8.2" and Server version was "1.9.6" . I upgraded my Client version to 1.9.6 and it works fine now. I am able to perform any kubectl operation on the statefulsets without any error.

The below one had Issues while deleting the statefulset.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Upgrading the client version in my mac:
$ curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.9.6/bin/darwin/amd64/kubectl

Make the kubectl binary executable.
$ chmod +x ./kubectl

Move the binary in to your PATH.
$ sudo mv ./kubectl /usr/local/bin/kubectl

$kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

vijayeswari on 18 Jun 2018

😄1 👍1

I'm seeing the same issue but only on one out of three clusters. I initially thought it might be a caching issue but clearing ~/.kube/cache/discover/<cluster> does not seem to help at all. Is there any information that I can provide to help sort this one out?

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-22T05:39:27Z", GoVersion:"go1.9.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

The versions are the same across the three clusters.

alkar on 6 Aug 2018

/area kubectl

seans3 on 26 Sep 2018

All of the error examples seem to be when the client and server diverge more than +/- 1 version, which is not supported. Therefore, I am going to close this issue. Please open another issue if there are errors related to client/server versions within +/- 1 version.

seans3 on 26 Sep 2018

Using the right kubectl client version to match the server version did the trick. Thanks for the help guys.