Aws-load-balancer-controller: Can not get ingress controller running

[mattjs]

I followed the kubectl instructions here https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/controller/setup/, but found the following errors after running the command to check the status of the deployment.

kubectl logs -n kube-system $(kubectl get po -n kube-system | egrep -o "alb-ingress[a-zA-Z0-9-]+")

E0715 22:53:07.543100       1 leaderelection.go:270] error retrieving resource lock kube-system/ingress-controller-leader-alb: configmaps "ingress-controller-leader-alb" is forbidden: User "system:serviceaccount:kube-system:alb-ingress" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Am I missing some permissions on my AWS access or did I do something wrong?

[M00nF1sh]

Hi, did you deploy the RBAC rules first?

3. Deploy the RBAC roles manifest


kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.2/docs/examples/rbac-role.yaml

If not, you should first delete the controller YAML and apply the RBAC policies first, and then reapply the controller yaml

[mattjs]

I did add the role, but to confirm 100% I've just deleted the role and the controller and then added them back (role first).

Is the difference maybe the system:serviceaccount:kube-system:alb-ingress I see in the error vs alb-ingress-controller I see in the RBAC rules?

[mattjs]

One thing that may have been wrong was I installed an older version of the manfiest

I had installed
https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.0/docs/examples/alb-ingress-controller.yaml

Instead of the newer version
https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.2/docs/examples/alb-ingress-controller.yaml

Now I have a bit of a different error

Error: AccessDeniedException: User: arn:aws:sts::819588939820:assumed-role/eksctl-hellorevel-nodegroup-ng-09-NodeInstanceRole-DPVDIUFQWDDF/i-0064aaa1c0e738a35 is not authorized to perform: tag:GetResources\n\tstatus code: 400, request id: fe560380-17fb-4cdb-b23b-27ecd9c7b597"  "controller"="alb-ingress-controller" "request"={"Namespace":"revel","Name":"alpha"}

[mattjs]

For more clarity, starting from scratch I ran

k8 apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.2/docs/examples/rbac-role.yaml

Then I ran

k8 apply -f https://gist.githubusercontent.com/mattjs/37945042849a4c8ba26cf10aa8e5b4e1/raw/31b81468d4ef66b60ffb368d6210a984a0068cb0/alb-ingress-controller.yaml

(Which is the modified version of https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.2/docs/examples/alb-ingress-controller.yaml with my cluster name added)

[mattjs]

I closed this because I thought I had fixed the issue, but sadly I had not.

For some additional information I used eksctl to setup the cluster (as you can probably tell)

[mattjs]

OK, I've sorted this out and it was a missing permission.

I modified the node roles by hand and added permissions from this file

https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/master/docs/examples/iam-policy.json

After looking at this related issue https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/924

And now everything is working.

[M00nF1sh]

Hi Matthew,
The ALB Ingress Controller needs IAM permission to talk to AWS Services(https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/controller/config/#aws-api-access)
You need to create an iam policy using https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/master/docs/examples/iam-policy.json, and then attach it to worker nodes.

If you use eksctl, you can use "--alb-ingress-access" when eksctl create cluster/nodegroup to automate it.

BTW, the docs section should be updated to reflect this

[mattjs]

Thanks @M00nF1sh, appreciate the help.