External-storage: efs-provisioner: error retrieving resource lock - endpoint is forbidden

[richstokes]

Using AWS EFS.

Followed instructions (https://github.com/kubernetes-incubator/external-storage/tree/master/aws/efs), applied RBAC yaml. But getting this in the efs-provisioner logs:

E0816 19:25:11.059420       1 leaderelection.go:252] error retrieving resource lock default/company.io-efs: endpoints "company.io-efs" is forbidden: User "system:serviceaccount:default:default" cannot get resource "endpoints" in API group "" in the namespace "default"

What am I/the docs missing here?

[richstokes]

Got it working by changing the RBAC to:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: efs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-efs-provisioner
subjects:
  - kind: ServiceAccount
    name: efs-provisioner
     # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: efs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
# ---
# kind: Role
# apiVersion: rbac.authorization.k8s.io/v1
# metadata:
#   name: leader-locking-efs-provisioner
# rules:
#   - apiGroups: [""]
#     resources: ["endpoints"]
#     verbs: ["get", "list", "watch", "create", "update", "patch"]
# ---
# kind: RoleBinding
# apiVersion: rbac.authorization.k8s.io/v1
# metadata:
#   name: leader-locking-efs-provisioner
# subjects:
#   - kind: ServiceAccount
#     name: efs-provisioner
#     # replace with namespace where provisioner is deployed
#     namespace: default
# roleRef:
#   kind: Role
#   name: leader-locking-efs-provisioner
#   apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: efs-provisioner

And adding the service account to the manifest.yaml

   spec:
      serviceAccount: efs-provisioner
      containers:
        - name: efs-provisioner
          image: quay.io/external_storage/efs-provisioner:latest

^^ Found this in an old ticket (https://github.com/kubernetes-incubator/external-storage/issues/953), but doesn't look like a fix was ever rolled out. Would be worth integrating these fixes as its quite annoying following docs and example yamls that are out of date and wrong.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[AdapptGru]

/remove-lifecycle stale

[AdapptGru]

Hit this just & @richstokes comment fixed it. The docs can seriously do with some updates.

[wavetylor]

This is likely due to the efs-provisioner's deployment being in the non-default namespace.
If you deploy efs-provisioner into a namespace other than default, you need to also add the namespace to the Role and RoleBinding

[boomshadow]

@wavetylor Thank you so much for posting! That's what I needed. Added the namespace to the Role and RoleBinding fixed my issue!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[nikhita]

Thanks for reporting the issue!

This repo is no longer being maintained and we are in the process of archiving this repo. Please see https://github.com/kubernetes/org/issues/1563 for more details.

If your issue relates to nfs provisioners, please create a new issue in https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner or https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.

Going to close this issue in order to archive this repo. Apologies for the churn and thanks for your patience! :pray: