External-storage: [nfs-client] Error receiving resource lock

[Queuecumber]

When starting the latest image of the nfs-client-provisioner, I get

error retrieving resource lock kube-system/softmax.app-nfs: endpoints "softmax.app-nfs" is forbidden: User "system:serviceaccount:storage:nfs-client-provisioner" cannot get endpoints in the namespace "kube-system"

over and over in my logs and it doesn't work. Pretty sure it is because of:

https://github.com/kubernetes-incubator/external-storage/blob/8052cf737caaf2a701f0caad491e08b8ac21573c/lib/controller/controller.go#L651

I don't use the kube-system namespace, do you need to assume kube-system?

[Queuecumber]

As far as I can tell this was actually due to a changing in the RBAC role. However now my volumes arent getting deleted. Will go over the config files again before posting an issue on it

[chz8494]

facing same error here, v2.1.1-k8s1.10 doesn't have this problem, definitely something related to the changes in v3.0

[wongma7]

@chz8494 please try editing your clusterrole so it has these permissions https://github.com/kubernetes-incubator/external-storage/blob/master/nfs-client/deploy/auth/clusterrole.yaml#L19

[chz8494]

@wongma7 thanks for quick reply, adding new permission works!

[geerlingguy]

Was trying to find the file / line @wongma7 referred to in his comment above, but it looks like the file has moved and the above link goes to a 404.

I'm hitting this same issue (with the deploy manifests from master), and it looks like the correct link that should work for the future (pointing to a tag instead of master) is: https://github.com/kubernetes-incubator/external-storage/blob/v5.0.1/nfs-client/deploy/auth/clusterrole.yaml#L18-L20

Specifically, add the following to the ClusterRole:

  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

I'm testing this fix now. It seems like the leader-locking-nfs-client-provisioner _should_ provide this permission, but it looks like it might not be. With the default nfs-client layout in master, I am getting the following error every 6 seconds on the provisioner pod:

E1229 21:02:56.043228       1 leaderelection.go:234] error retrieving resource lock default/fuseim.pri-nfs: endpoints "fuseim.pri-nfs" is forbidden: User "system:serviceaccount:default:nfs-client-provisioner" cannot get resource "endpoints" in API group "" in the namespace "default"

[geerlingguy]

Well, in my case I figured out the problem—the RBAC manifest I was editing locally (and I _thought_ I was applying to the cluster) was not actually being applied. I had left the old NFS RBAC manifest file from the nfs and so the nfs-client provisioner was not being updated.

I reset the cluster, built it again from scratch, and made sure the nfs provisioner's RBAC manifest was not being applied (just the one from nfs-client), and now it's all magically working, yay!

[markus-seidl]

I have the same problem with the current master and the lines geerlingguy pointed to, are missing now. I've added them and it works again.

I'm confused if these lines are still necessary, my local test suggests so.