External-dns: Failed to list services

[yujunz]

I am following the instructions in AWS docs and got the errors below

time="2018-10-23T06:04:30Z" level=info msg="..."
time="2018-10-23T06:04:30Z" level=info msg="Created Kubernetes client https://100.64.0.1:443"
time="2018-10-23T06:04:32Z" level=error msg="services is forbidden: User \"system:serviceaccount:kube-system:external-dns\" cannot list services at the cluster scope"

The related resources are as below

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRole","metadata":{"annotations":{},"name":"external-dns","namespace":""},"rules":[{"apiGroups":[""],"resources":["services"],"verbs":["get","watch","list"]},{"apiGroups":[""],"resources":["pods"],"verbs":["get","watch","list"]},{"apiGroups":["extensions"],"resources":["ingresses"],"verbs":["get","watch","list"]},{"apiGroups":[""],"resources":["nodes"],"verbs":["list"]}]}
  creationTimestamp: 2018-10-23T05:57:03Z
  name: external-dns
  resourceVersion: "1844111"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/external-dns
  uid: 76f19314-d688-11e8-b20e-06c4b7e2ee18
rules:
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
➜  dev-us-west-1 git:(external-dns) ✗ kubectl get clusterrolebinding external-dns -o yaml
Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "external-dns" not found
➜  dev-us-west-1 git:(external-dns) ✗ kubectl get clusterrolebinding external-dns-viewer -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"external-dns-viewer","namespace":""},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"external-dns"},"subjects":[{"kind":"ServiceAccount","name":"external-dns","namespace":"default"}]}
  creationTimestamp: 2018-10-23T05:57:03Z
  name: external-dns-viewer
  resourceVersion: "1844112"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/external-dns-viewer
  uid: 772dcfb9-d688-11e8-b20e-06c4b7e2ee18
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default

[yujunz]

It turns out to be namespace issue. All the manifests should be applied in default namespace since cluster role binding is hard coded.

[dutoitns]

Or just change the namespace reference at the bottom of the ClusterRoleBinding to kube-system.

External-dns failed to change the Route53 record pointing to the ALB on re-provisioning of the ingress. I got the error as indicated in this issue, and after changing the default namespace at the end of the ClusterRoleBinding definition to kube-system it worked.

In other words, there were no longer errors in the logs of External-dns and my Route53 record got updated.

So yeah, imho there's a bug in the example ClusterRoleBinding.