Autoscaler: Missing RBAC rule for CSINode

[ernoaapa]

Info

Component: auto-scaler
Version: v1.16.0

Problem

In the latest v1.16.0 auto-scaler version, I got RBAC errors about listing CSINodes.

I0911 05:18:18.315693       1 reflector.go:158] Listing and watching *v1beta1.CSINode from k8s.io/client-go/informers/factory.go:134
E0911 05:18:18.317192       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope

I used the RBAC roles from aws examples.

Looks like that the https://github.com/kubernetes/autoscaler/commit/7a4836f92a8cbe1e4559f6fefa52723e3fb5bc3f change didn't add the required RBAC rules to the examples.

Solution

Fixed by adding the following rule to the cluster-autoscaler ClusterRole

# ...
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["watch", "list", "get"]
# ...

[ayosec]

After updating the ClusterRole we get the following error every second:

E0912 03:22:09.848603       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSINode: the server could not find the requested resource

[thanasisk]

After updating the ClusterRole we get the following error every second:

E0912 03:22:09.848603       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSINode: the server could not find the requested resource

Hello, the following link might be of help.
https://kubernetes-csi.github.io/docs/csi-node-object.html
If you are on a version before 1.14, follow the instructions in the link to enable the relevant feature flag and create the relevant CRD

[khteh]

How to enable the --feature-gates=CSINodeInfo=true on AWS EKS?

[losipiuk]

How to enable the --feature-gates=CSINodeInfo=true on AWS EKS?

@Jeffwan ?

[losipiuk]

@khteh I suggest you ask on #sig-autoscaling on k8s slack. That is a good place to ask questions.

[Jeffwan]

@khteh

CSINodeInfo=true is alpha feature on 1.11-1.13. EKS doesn't only enable beta feature gate, so alpha features are not available. If you upgrade to 1.14. You can use it by default. 1.14 is also the latest version EKS supports.

[etwillbefine]

I run into this issue by running autoscaler 1.17.x on kubernetes 1.16.x. Downgraded to helm version 7.0.0 (previous 7.2.2) and using app version 1.16.x

Helm values config for 7.0.0

image:
  tag: v1.16.5
  repository: eu.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler

[andredurao]

Unfortunately I couldn't find a v1.16.5 version as commented.

BTW If you want to check the tags by yourself, this is what I've did:

$ gcloud container images list-tags k8s.gcr.io/cluster-autoscaler

gcloud container images list-tags k8s.gcr.io/cluster-autoscaler | grep 'v1.16'
61610899d162  v1.16.4                          2020-02-13T15:09:01
6b887823b1fd  v1.16.3                          2019-11-28T07:43:54
8d8664ac086f  v1.16.2                          2019-10-16T09:12:01
92390effc355  v1.16.1                          2019-09-23T05:48:21
9034e7500a5b  v1.16.0                          2019-09-09T13:50:26
572ea58dce07  v1.16.0-beta.1                   2019-09-06T12:32:22

[abdennour]

As per AWS doc, the image for EKS must be from this repo :

US: us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.n.n
or
EU: eu.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.n.n
... so on

And for v1.n.n replaces it by the output of kubectl version.. However, v1.16.8 is not available. So I used v1.16.5.

As conclusion, my answer is a details of @etwillbefine answer.
Also using helm-chart 7.0.0 of cluster-autoscaler is the right choice for EKS 1.16