Charts: [incubator/aws-alb-ingress-controller] Fargate and IRSA: Not authorized to perform sts:AssumeRoleWithWebIdentity

[lukaszbudnik]

Describe the bug
When using incubator/aws-alb-ingress-controller helm chart to deploy on EKS with Fargate profile and IRSA the controller is not authorised to to assume role using web entities:

failed to build LoadBalancer configuration due to unable to fetch subnets. Error: WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id:XXX

Version of Helm and Kubernetes:

$ helm version
Client: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-23T14:21:36Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-c0eccc", GitCommit:"c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2", GitTreeState:"clean", BuildDate:"2019-12-22T23:14:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Which chart:

incubator/aws-alb-ingress-controller

What happened:

The aws-alb-ingress-controller did not provision AWS ALB and failed on assuming role with web identity.

What you expected to happen:

aws-alb-ingress-controller to provision AWS ALB.

How to reproduce it (as minimally and precisely as possible):

Create a cluster with fargate profile and enable IAM OIDC provider (used for the assume role with web identity):

eksctl create cluster \
--name YOUR-NAME \
--version 1.14 \
--region ap-northeast-1 \
--external-dns-access \
--alb-ingress-access \
--full-ecr-access \
--fargate

eksctl utils associate-iam-oidc-provider \
    --region ap-northeast-1 \
    --cluster YOUR-NAME \
    --approve

Create policy which allows managing ALB:

aws iam create-policy \
    --policy-name ALBIngressControllerIAMPolicy \
    --policy-document https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.5/docs/examples/iam-policy.json

And create IAM service account (replace aws_account_id with your own):

eksctl create iamserviceaccount \
    --region ap-northeast-1 \
    --name alb-ingress-controller \
    --namespace kube-system \
    --cluster YOUR-NAME \
    --attach-policy-arn arn:aws:iam::{aws_account_id}:policy/ALBIngressControllerIAMPolicy \
    --override-existing-serviceaccounts \
    --approve

eksctl will create a role, copy its ARN and then install helm chart and add service annotation with the role which has permissions to create and manage ALBs:

helm install incubator/aws-alb-ingress-controller --set clusterName=YOUR-NAME --set awsRegion=ap-northeast-1 --set vpcID=vpc-XXX --name ingress-controller --set rbac.serviceAccountAnnotations."eks\.amazonaws\.com/role-arn"='arn:aws:iam::XXX:role/eksctl-YOUR-NAME-addon-iamserviceac-Role1-17JLEPFZYMIWC' --namespace kube-system

And then try to deploy a sample ALB ingress.

when inspecting aws-alb-ingress-controller logs you will find errors like:

E0203 11:49:09.958000       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to unable to fetch subnets. Error: WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: XXX"  "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"XXX-ingress"}

Anything else we need to know:

I can successfully create an ALB controller using a set of kubectl commands on Fargate with IRSA. I'm using the same source image as the helm chart: docker.io/amazon/aws-alb-ingress-controller:v1.1.5.

For reference, here is my working example of AWS EKS with ALB ingress using step by step kubectl: https://github.com/lukaszbudnik/migrator/tree/master/contrib/kubernetes-aws-eks

[jamiegs]

Were you able to figure this issue out? I'm running into it also.

[lukaszbudnik]

@jamiegs well yes, if you do it via a set of kubectl it will work just fine, so if you want to get it running just follow the steps I have here: https://github.com/lukaszbudnik/migrator/tree/master/contrib/kubernetes-aws-eks

if you want to use helm to provision ALB then you have to wait for a fix

[t3-aoki]

eksctl create iamserviceaccount --name alb-ingress-controller --namespace kube-system creates k8s service account named alb-ingress-controller and its role with following assume role policy.
You can see it in Trust Relationships tab in role detail in IAM console.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/oidc.eks.${REGION}.amazonaws.com/id/${CLUSTER_ID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.${REGION}.amazonaws.com/id/${CLUSTER_ID}:aud": "sts.amazonaws.com",
          "oidc.eks.${REGION}.amazonaws.com/id/${CLUSTER_ID}:sub": "system:serviceaccount:kube-system:alb-ingress-controller"
        }
      }
    }
  ]
}

The following condition means it only allows alb-ingress-controller service account in kube-system namespace to sts:AssumeRoleWithWebIdentity action.

          "oidc.eks.ap-northeast-1.amazonaws.com/id/${CLUSTER_ID}:sub": "system:serviceaccount:kube-system:alb-ingress-controller"

On the other hand, helm install incubator/aws-alb-ingress-controller --name ingress-controller --namespace kube-system creates a new service account named ingress-controller. This account is not allowed, even if it has correct eks.amazonaws.com/role-arn annotation.

The workaround is as follow.

First, install alb-ingress-controller with helm.
(The example uses helm v3)

$ helm install incubator/aws-alb-ingress-controller \
  --generate-name --namespace kube-system \
  --set clusterName=${cluster_name} \
  --set awsRegion=${region} \
  --set awsVpcID=${vpc_id}

Please check your service account name.

$ kubectl get serviceaccount -n kube-system | grep alb
aws-alb-ingress-controller-1234567890  1  100s

Next, create IAM ServieAccount by eksctl. You have to set the above service account name to --name option. eksctl automatically updates this service account.

$ eksctl create iamserviceaccount \
    --region ${region} \
    --name aws-alb-ingress-controller-1234567890 \
    --namespace kube-system \
    --cluster ${cluster_name} \
    --attach-policy-arn arn:aws:iam::${aws_account_id}:policy/${policy_id} \
    --override-existing-serviceaccounts \
    --approve

Finally, restart your aws-ingress-controller pod to update pod env vars.

$ kubectl get pod -n kube-system | grep alb
aws-alb-ingress-controller-1234567890-xxxxxxxxx-xxxxx   1/1     Running   0          119s
$ kubectl delete pod -n kube-system aws-alb-ingress-controller-1234567890-xxxxxxxxx-xxxxx

You can see ALB created in EC2 console when the manifests applied.

$ kubectl apply -f deployment.yaml service.yaml ingress.yaml

[t3-aoki]

I think the chart should uses given service account name when rbac.serviceAccountName parameter is given expressly even if rbac.create is true.
It allows us to create RBAC resources (ClusterRole and ClusterRoleBinding) by helm with existing service account created by eksctl.

[t3-aoki]

I think #21474 will fix this issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[lukaszbudnik]

I can now confirm that I can successfully use the helm chart on Fargate with IRSA. Thanks for fixing it! Closing.

[jiangxianliang007]

https://github.com/lukaszbudnik/migrator/tree/master/contrib/kubernetes-aws-eks

this is 404

[lukaszbudnik]

Code was refactored and moved to tutorial folder: https://github.com/lukaszbudnik/migrator/tree/master/tutorials/aws-eks