Aws-load-balancer-controller: Unable to create ALB Ingress controller using Kube2IAM in EKS

[gibsster]

Hi @M00nF1sh,
I have similar issue but in my case I'm using kube2IAM.
For some reason the ALB is not created when creating the ingress for my application.

This is the error I'm getting from the alb-ingress-controller

E0423 17:47:58.856073 1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" "Controller"="alb-ingress-controller" "Request"={"Namespace":"service-task","Name":"service-task-ingress"}

Below is my alb-ingress-controller yaml file

# Application Load Balancer (ALB) Ingress Controller Deployment Manifest.
# This manifest details sensible defaults for deploying an ALB Ingress Controller.
# GitHub: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
  # Namespace the ALB Ingress Controller should run in. Does not impact which
  # namespaces it's able to resolve ingress resource for. For limiting ingress
  # namespace scope, see --watch-namespace.
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alb-ingress-controller
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::xxxxxxxx:role/eks-alb-controller
      creationTimestamp: null
      labels:
        app: alb-ingress-controller
    spec:
      containers:
        - args:
            # Limit the namespace where this ALB Ingress Controller deployment will
            # resolve ingress resources. If left commented, all namespaces are used.
            # - --watch-namespace=your-k8s-namespace

            # Setting the ingress-class flag below ensures that only ingress resources with the
            # annotation kubernetes.io/ingress.class: "alb" are respected by the controller. You may
            # choose any class you'd like for this controller to respect.
            - --ingress-class=alb

            # Name of your cluster. Used when naming resources created
            # by the ALB Ingress Controller, providing distinction between
            # clusters.
            - --cluster-name=dev-eks

            # AWS VPC ID this ingress controller will use to create AWS resources.
            # If unspecified, it will be discovered from ec2metadata.
            # - --aws-vpc-id=vpc-xxxxxx

            # AWS region this ingress controller will operate in. 
            # If unspecified, it will be discovered from ec2metadata.
            # List of regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#vpc_region
            # - --aws-region=us-west-1

            # Enables logging on all outbound requests sent to the AWS API.
            # If logging is desired, set to true.
            # - ---aws-api-debug
            # Maximum number of times to retry the aws calls.
            # defaults to 10.
            # - --aws-max-retries=10
          env:
            # AWS key id for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            #- name: AWS_ACCESS_KEY_ID
            #  value: KEYVALUE

            # AWS key secret for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            #- name: AWS_SECRET_ACCESS_KEY
            #  value: SECRETVALUE
          # Repository location of the ALB Ingress Controller.
          image: 894847497797.dkr.ecr.us-west-2.amazonaws.com/aws-alb-ingress-controller:v1.0.0
          imagePullPolicy: Always
          name: server
          resources: {}
          terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      serviceAccountName: alb-ingress
      serviceAccount: alb-ingress

This is the trust policy of my eks-alb-controller AWS role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxx:role/dev-eks-EksWorkerNodes-JHFBE896CX-NodeInstanceRole-E5UIHIC756KC"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

I attached the same policy you mentioned above to the eks-alb-controller AWS role https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/examples/iam-policy.json

My dev-eks-EksWorkerNodes-JHFBE896CX-NodeInstanceRole-E5UIHIC756KC AWS role has the following policies AmazonEKSWorkerNodePolicy, AmazonEC@ContainerRegistry,ReadOnly, AmazonEKS_CNI_Policy and I created a policy that contains the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": "*"
        }
    ]
}

I'm doing almost exactly the same for S3 and CloudWatch to access them via Kube2IAM and that is working fine which I believe Kube2Iam was deployed correctly. Below is my ingress and service yaml files

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: "service-task-ingress"
  namespace: {{ .Values.namespace }}
  annotations:
    kubernetes.io/ingress.class: alb
  labels:
    app: task-core
spec:
  rules:
    - http:
        paths:
          - path: /*
            backend:
              serviceName: "task-core"
              servicePort: 8080
---
apiVersion: v1
kind: Service
metadata:
  namespace: {{ .Values.namespace }}
  name: task-core
  labels:
    app: task-core
spec:
  type: NodePort
  ports:
  - name: http
    port: 8080
    targetPort: 8080
    protocol: TCP
  selector:
    app: task-core-pod

[M00nF1sh]

Hi. would you share your accountId and clusterName with me? I can help take a look

[gibsster]

I solved my issue I needed to add the following flags in the alb-ingress-controller yaml file - --aws-region= and - --aws-api-debug

[vamsirede]

@gibsster I'm getting the same error as you, Can you please share the syntax and where to add the above two flags you mentioned (- --aws-region= and - --aws-api-debug).

Thanks,

[gibsster]

@vamsirede look at the YAML file I put in this thread... you only need to uncomment those two flags under:

spec:
containers:
- args:

[vamsirede]

Ha ok, Thank You @gibsster