Kubectl: Missing access right leads to endless error output

Created on 30 Jan 2019 · 10Comments · Source: kubernetes/kubectl

Kubernetes version (use kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6+IKS", GitCommit:"002d263ed027db260968616b951fb46f2bab9bb1", GitTreeState:"clean", BuildDate:"2019-01-09T08:07:22Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: IBM Kubernetes

What happened:

kubectl rollout status deployment is not failing but logging thousend lines of the same error:

check rollout status of deployment
E0130 09:40:22.308805     176 reflector.go:134] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to list *unstructured.Unstructured: deployments.extensions "any-deployment" is forbidden: User "system:serviceaccount:dev:test-kubectl" cannot list deployments.extensions in the namespace "dev"
E0130 09:40:23.312517     176 reflector.go:134] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to list *unstructured.Unstructured: deployments.extensions "any-deployment" is forbidden: User "system:serviceaccount:dev:test-kubectl" cannot list deployments.extensions in the namespace "dev"
E0130 09:40:24.315702     176 reflector.go:134] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to list *unstructured.Unstructured: deployments.extensions "any-deployment" is forbidden: User "system:serviceaccount:dev:test-kubectl" cannot list deployments.extensions in the namespace "dev"
...

What you expected to happen:

It just failing similar to this:

Error from server (Forbidden): deployments.extensions "any-deployment" is forbidden: User "system:serviceaccount:ado-dev:test-kubectl" cannot get deployments.extensions in the namespace "dev"

How to reproduce it (as minimally and precisely as possible):

Create test-kubectl.yml

# Required because of inter namespace communication
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
#  namespace: "dev"
  name: test-kubectl
rules:
- apiGroups: ["", "extensions"]
  resources: ["deployments"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: test-kubectl
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: test-kubectl
subjects:
- kind: ServiceAccount
  name: test-kubectl
  namespace: dev
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-kubectl
imagePullSecrets:
- name: bluemix-default-secret

Install it

kubectl -n=dev create -f test-kubectl.yml

execute kubectl in a pod that is assigned to this service account:

kubectl rollout status deployment any-deployment -n=dev

arekubectl kinbug lifecyclrotten prioritP1 sicli

Source

mdeknowis

👍5

Most helpful comment

@shroudedcode For me, adding the watch verb on deployments suppresses those errors.
In fact, while list and watch are needed for _all_ deployments in the namespace, only get is needed for the specific deployment that is being rolled out. (And patch, if you need to be able to kubectl set image with the same serviceaccount)

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: web
  name: deploy-assets
rules:
- apiGroups: ["extensions"]
  resources: ["deployments"]
  resourceNames: ["assets-prod", "assets-test"]
  verbs: ["get", "patch"]
- apiGroups: ["extensions"]
  resources: ["deployments"]
  verbs: ["list", "watch"]

Feder1co5oave on 3 May 2019

👍5

All 10 comments

not so bad but similar:

With this

# Required because of inter namespace communication
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
#  namespace: "dev"
  name:  test-kubectl
rules:
- apiGroups: ["", "extensions"]
  resources: ["deployments"]
  verbs: ["get", "list"]

you get that output:

check rollout status of deployment
E0130 10:07:48.240771     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
E0130 10:07:49.256351     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
E0130 10:07:50.270648     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
E0130 10:07:51.286169     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
E0130 10:07:52.307516     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
E0130 10:07:53.322151     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
E0130 10:07:54.337076     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
E0130 10:07:55.350552     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
E0130 10:07:56.364719     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
E0130 10:07:57.382483     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "any-deployment" rollout to finish: 1 of 2 updated replicas are available...
E0130 10:07:58.411172     221 reflector.go:251] k8s.io/client-go/tools/watch/informerwatcher.go:110: Failed to watch *unstructured.Unstructured: unknown
deployment "any-deployment" successfully rolled out

mdeknowis on 30 Jan 2019

👍2

@mdeknowis I'm having the same issue:

E0330 12:47:24.329887      36 reflector.go:270] k8s.io/client-go/tools/watch/informerwatcher.go:113: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "backend" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "backend" rollout to finish: 1 out of 3 new replicas have been updated...
E0330 12:47:25.371217      36 reflector.go:270] k8s.io/client-go/tools/watch/informerwatcher.go:113: Failed to watch *unstructured.Unstructured: unknown
Waiting for deployment "backend" rollout to finish: 1 out of 3 new replicas have been updated...

[...]

Waiting for deployment "backend" rollout to finish: 1 old replicas are pending termination...
E0330 12:48:26.882631      36 reflector.go:270] k8s.io/client-go/tools/watch/informerwatcher.go:113: Failed to watch *unstructured.Unstructured: unknown
deployment "backend" successfully rolled out
E0330 12:48:27.924904      36 reflector.go:270] k8s.io/client-go/tools/watch/informerwatcher.go:113: Failed to watch *unstructured.Unstructured: unknown

This is the output I get when running:

kubectl rollout status deployments/backend --watch --timeout 3m

The ClusterRole looks like this:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: deployment
rules:
  - apiGroups: [extensions]
    resources: [deployments]
    verbs: [get, list]

The weird thing is that the command, despite the errors, works perfectly well.

Output of kubectl version

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

shroudedcode on 30 Mar 2019

👍1

/kind bug
/priority P1
/area kubectl
/sig cli

seans3 on 1 Apr 2019

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: web
  name: deploy-assets
rules:
- apiGroups: ["extensions"]
  resources: ["deployments"]
  resourceNames: ["assets-prod", "assets-test"]
  verbs: ["get", "patch"]
- apiGroups: ["extensions"]
  resources: ["deployments"]
  verbs: ["list", "watch"]

Feder1co5oave on 3 May 2019

👍5

Thanks a lot, @Feder1co5oave, it's working for me! The only reasons I didn't add that verb in the first place were that:

I didn't know watch was a separate verb.
The command worked flawlessly without it (except for the errors of course).

This makes me think there are two issues here (at least in my case):

The error message is confusing and simply false since watching works.
It should be corrected by actually making the watching fail and giving more useful information like The "watch" verb is required to watch this object.
The ClusterRole can do more than it should be able to do.
I guess you could call this a :warning: minor security vulnerability and certainly unexpected behavior.

@seans3 Am I interpreting this correctly or is this intended behavior?

shroudedcode on 3 May 2019

👍1

That confusing error message is certainly misleading and also suggests some
kind of bug in kubectl.
Nonetheless, RBAC is _hard_ to work with (security always comes at a cost)
and I've yet to find the perfect workflow for finding out what minimal set
of privileges is required to perform a certain action. Usually kubectl is
more specific about the error, but to troubleshoot this one I had to resort
to k8s audit logs (good luck with those, too).

Feder1co5oave on 3 May 2019

👍2

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 1 Aug 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 31 Aug 2019

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 30 Sep 2019

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.