Aws-load-balancer-controller: failed to reconcile LB managed SecurityGroup: failed to reconcile managed LoadBalancer securityGroup: WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity

[a5a5a9]

So I followed the instructions:

https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html#create-service-account-iam-role

https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html

And get the following logs

E0225 23:24:40.746946 1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile LB managed SecurityGroup: failed to reconcile managed LoadBalancer securityGroup: WebIdentityErr: failed to retrieve credentialsncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentityntstatus code:

403, request id: 606ec825-6e74-4086-b11f-51bc11b0713e" "controller"="alb-ingress-controller" "request"={"Namespace"

any ideas how to fix it?

[hjpotter92]

similar issue with my setup. I've tried switching kub2iam to my own AWS keys, still the same logs.

E0226 07:46:38.428745       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile targetGroups due to failed to reconcile targetGroup targets due to Unable to find 80 port on service pose"  "controller"="alb-ingress-controller" "request"={"Namespace":"pose","Name":"pose"}

[a5a5a9]

yes I also tried with kub2iam and no dice

[M00nF1sh]

@hjpotter92 your issue is different. do you have an port 80 on service pose?

@adalandrade what's your trust relationship on your role(which is added as annotation to the alb-ingress-controller's service account)? It needs to allows that SA to assume it

[a5a5a9]

I also

kubectl annotate serviceaccount -n kube-system alb-ingress-controller
eks.amazonaws.com/role-arn=arn:aws:iam::111122223333:role/eks-alb-ingress-controller

[a5a5a9]

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::1111222233338:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/XXXXF62F531065EA1B59E7D562EBEE"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/XXXXXF62F531065EA1B59E7D562EBEE:sub": "system:serviceaccount:kube-system:elb-ingress-controller"
}
}
}
]
}

[M00nF1sh]

@adalandrade from the trust policy, the condition is system:serviceaccount:kube-system:elb-ingress-controller, while the service account is alb-ingress-controller. Not sure how u originally setup this policy, but you need to update it to trust system:serviceaccount:kube-system:alb-ingress-controller instead.

[hjpotter92]

@hjpotter92 your issue is different. do you have an port 80 on service pose?

yes. the following is my manifest for the same:

apiVersion: v1
kind: Service
metadata:
  namespace: pose
  name: pose
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 8010
      protocol: TCP
  selector:
    app: pose

[a5a5a9]

After correcting the typo , I still get this

kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile LB managed SecurityGroup: failed to reconcile managed LoadBalancer securityGroup: UnauthorizedOperation: You are not authorized to perform this operation.ntstatus code: 403, request id: 34748ec5-f9c2-4067-b4dd-64b9fda620cf" "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"prtg-ingress"}

[M00nF1sh]

@hjpotter92 would share your EKS cluster arn? assume you are using EKS

@adalandrade It's a different Error message now, did you role(arn:aws:iam::111122223333:role/eks-alb-ingress-controller) have correct permissions? sample permissions: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/examples/iam-policy.json

[a5a5a9]

I fixed my problem by doing the manual setup of the alb ingress controller instead of using the helm chart.

thanks

[ahrtr]

I see this issue as well, and the workaround solution is to attach following policy

https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.5/docs/examples/iam-policy.json

to the role which is associated with your K8S nodes directly.

[edify42]

I see this issue as well, and the workaround solution is to attach following policy

https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.5/docs/examples/iam-policy.json

to the role which is associated with your K8S nodes directly.

Thanks @ahrtr - I can confirm this is working. We've setup EKS with an OIDC auth URL to allow pods to assume a role instead of using the node's attached role. The alb-ingress ServiceAccount only needs the AWS IAM Role ARN with this policy attached (along with the AssumeRoleWithWebIdentity config).

[h3adache]

Thanks @ahrtr was banging my head on this. They should fix the docs :P

[nitrocode]

If using the terraform-aws-eks module, do the following to attach the above policy to the worker (i.e. node) IAM role.

# get official iam policy for aws alb ingress controller
# Note: change the version to the desire version
data "http" "worker_policy" {
  url = "https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.8/docs/examples/iam-policy.json"

  request_headers = {
    Accept = "application/json"
  }
}

# and attach it
resource "aws_iam_role_policy" "worker_policy" {
  name   = "worker_policy"
  role   = module.my-cluster.worker_iam_role_name
  policy = data.http.worker_policy.body
}

[ameliatqy]

When you installed the controller from https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html, did you use the namespace kube-system (or shown in the instructions) or kubeflow (where the original ALB was deployed right after installation)?