Aws-load-balancer-controller: ALB is not created on AWS

Created on 18 Apr 2019  路  11Comments  路  Source: kubernetes-sigs/aws-load-balancer-controller

Hello, my ALB isn't getting created on AWS.

Here's my config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
spec:
  rules:
    - host: myhost.com
      http:
        paths:
          - path: /
            backend:
              serviceName: myapp
              servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: default
spec:
  ports:
    - port: 80
      targetPort: 8888
      protocol: TCP
  type: NodePort
  selector:
    app: myapp
---
# Application Load Balancer (ALB) Ingress Controller Deployment Manifest.
# This manifest details sensible defaults for deploying an ALB Ingress Controller.
# GitHub: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
  # Namespace the ALB Ingress Controller should run in. Does not impact which
  # namespaces it's able to resolve ingress resource for. For limiting ingress
  # namespace scope, see --watch-namespace.
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alb-ingress-controller
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: alb-ingress-controller
    spec:
      containers:
        - args:
            # Limit the namespace where this ALB Ingress Controller deployment will
            # resolve ingress resources. If left commented, all namespaces are used.
            # - --watch-namespace=your-k8s-namespace

            # Setting the ingress-class flag below ensures that only ingress resources with the
            # annotation kubernetes.io/ingress.class: "alb" are respected by the controller. You may
            # choose any class you'd like for this controller to respect.
            - --ingress-class=alb

            # Name of your cluster. Used when naming resources created
            # by the ALB Ingress Controller, providing distinction between
            # clusters.
            - --cluster-name=seenickcode.com

            # AWS VPC ID this ingress controller will use to create AWS resources.
            # If unspecified, it will be discovered from ec2metadata.
            # - --aws-vpc-id=vpc-xxxxxx

            # AWS region this ingress controller will operate in. 
            # If unspecified, it will be discovered from ec2metadata.
            # List of regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#vpc_region
            # - --aws-region=us-west-1

            # Enables logging on all outbound requests sent to the AWS API.
            # If logging is desired, set to true.
            # - ---aws-api-debug
            # Maximum number of times to retry the aws calls.
            # defaults to 10.
            # - --aws-max-retries=10
          env:
            # AWS key id for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            - name: AWS_ACCESS_KEY_ID
              value: REDACTED

            # AWS key secret for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            - name: AWS_SECRET_ACCESS_KEY
              value: REDACTED
          # Repository location of the ALB Ingress Controller.
          image: docker.io/amazon/aws-alb-ingress-controller:v1.1.0
          imagePullPolicy: Always
          name: server
          resources: {}
          terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      serviceAccountName: alb-ingress
      serviceAccount: alb-ingress
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
rules:
  - apiGroups:
      - ""
      - extensions
    resources:
      - configmaps
      - endpoints
      - events
      - ingresses
      - ingresses/status
      - services
    verbs:
      - create
      - get
      - list
      - update
      - watch
      - patch
  - apiGroups:
      - ""
      - extensions
    resources:
      - nodes
      - pods
      - secrets
      - services
      - namespaces
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: alb-ingress-controller
subjects:
  - kind: ServiceAccount
    name: alb-ingress
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: REDACTED
        imagePullPolicy: Always
        ports:
        - containerPort: 8888

It looks like the ALB is ok though:

kubectl -n kube-system logs -f alb-ingress-controller-974d48f98-49tjn
W0418 16:13:37.262705       1 client_config.go:548] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
-------------------------------------------------------------------------------
AWS ALB Ingress controller
  Release:    v1.1.0
  Build:      git-72962fcb
  Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller.git
-------------------------------------------------------------------------------

I0418 16:13:37.317967       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0418 16:13:37.318440       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0418 16:13:37.318625       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I0418 16:13:37.318949       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0418 16:13:37.319010       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I0418 16:13:37.319302       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0418 16:13:37.319836       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}}
I0418 16:13:37.320318       1 leaderelection.go:187] attempting to acquire leader lease  kube-system/ingress-controller-leader-alb...
I0418 16:13:52.804972       1 leaderelection.go:196] successfully acquired lease kube-system/ingress-controller-leader-alb
I0418 16:13:52.905378       1 :0] kubebuilder/controller "level"=0 "msg"="Starting Controller"  "controller"="alb-ingress-controller"
I0418 16:13:53.005715       1 :0] kubebuilder/controller "level"=0 "msg"="Starting workers"  "controller"="alb-ingress-controller" "worker count"=1
picture seenickcode

Most helpful comment

Hi, the controller lacks permissions to operate on your behalf. (There have been additonal permissions).
The up-to-date complete list of permission required is in https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/examples/iam-policy.json
If you are not using pod based IAM solution like(kube2iam), you need to create an IAM policy that contains the required iam permissions above, and attach it to the role of your worker nodes.

bash version below

export REGION=<your aws region> (e.g. us-west-2)
export NODE_ROLE_NAME=<name-of-your-worker-node-role> (use name instead of arn)
export ALB_POLICY_NAME=alb-ingress-controller
export ALB_POLICY_ARN=$(aws iam create-policy --region=$REGION --policy-name $ALB_POLICY_NAME --policy-document "https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/master/docs/examples/iam-policy.json" --query "Policy.Arn" | sed 's/"//g')
aws iam attach-role-policy --region=$REGION --role-name=$NODE_ROLE_NAME --policy-arn=$ALB_POLICY_ARN
picture M00nF1sh on 18 Apr 2019
👍7

All 11 comments

Hi, would you help share your accountID and clusterName with me ([email protected])?, I can help take a look 馃槃

M00nF1sh picture M00nF1sh on 18 Apr 2019

Thanks for the quick reply. Just emailed it to you @M00nF1sh. Note that I'm not using EKS, as I have created my cluster using kops. Not sure if that matters or not.

seenickcode picture seenickcode on 18 Apr 2019

@seenickcode
I cannot access your kops cluster(no permission to access worker nodes in customer account=.=).
Would you help do a kubectl describe ing/myapp -n default, most cases like this are caused by a bad yaml(e.g. unquoted number/boolean) and lost the kubernetes.io/ingress.class: alb annotation.

M00nF1sh picture M00nF1sh on 18 Apr 2019

Ok. Note the app is 'fcc' here:

kubectl describe ing/fcc -n default
Name:             fcc
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host             Path  Backends
  ----             ----  --------
  seenickcode.com  
                   /   fcc:80 (<none>)
Annotations:
  alb.ingress.kubernetes.io/tags:                    Environment=dev,Team=test
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"alb.ingress.kubernetes.io/scheme":"internet-facing","alb.ingress.kubernetes.io/tags":"Environment=dev,Team=test","kubernetes.io/ingress.class":"alb"},"name":"fcc","namespace":"default"},"spec":{"rules":[{"host":"seenickcode.com","http":{"paths":[{"backend":{"serviceName":"fcc","servicePort":80},"path":"/"}]}}]}}

  kubernetes.io/ingress.class:       alb
  alb.ingress.kubernetes.io/scheme:  internet-facing
Events:                              <none>
seenickcode picture seenickcode on 18 Apr 2019

The ingress configuration looks good to me.
Is that the full log of ALB Ingress Controller? (ending with I0418 16:13:53.005715 1 :0] kubebuilder/controller "level"=0 "msg"="Starting workers" "controller"="alb-ingress-controller" "worker count"=1)

M00nF1sh picture M00nF1sh on 18 Apr 2019

Actually no. That's weird, as when I ran this earlier I didn't see these additional errors.

Here's the continuation of them:

E0418 16:25:38.056300       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="no object matching key \"default/fcc\" in local store"  "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"fcc"}
E0418 16:25:39.178730       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: AccessDeniedException: User: arn:aws:iam::628863141346:user/kops is not authorized to perform: tag:GetResources\n\tstatus code: 400, request id: 9a659b4f-61f6-11e9-9dd7-0dc9efef4ad4"  "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"fcc"}
E0418 16:25:40.205980       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: AccessDeniedException: User: arn:aws:iam::628863141346:user/kops is not authorized to perform: tag:GetResources\n\tstatus code: 400, request id: 9b0202f6-61f6-11e9-9cc1-7d52d613338e"  "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"fcc"}

(the last error keeps repeating)

seenickcode picture seenickcode on 18 Apr 2019

Hi, the controller lacks permissions to operate on your behalf. (There have been additonal permissions).
The up-to-date complete list of permission required is in https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/examples/iam-policy.json
If you are not using pod based IAM solution like(kube2iam), you need to create an IAM policy that contains the required iam permissions above, and attach it to the role of your worker nodes.

bash version below

export REGION=<your aws region> (e.g. us-west-2)
export NODE_ROLE_NAME=<name-of-your-worker-node-role> (use name instead of arn)
export ALB_POLICY_NAME=alb-ingress-controller
export ALB_POLICY_ARN=$(aws iam create-policy --region=$REGION --policy-name $ALB_POLICY_NAME --policy-document "https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/master/docs/examples/iam-policy.json" --query "Policy.Arn" | sed 's/"//g')
aws iam attach-role-policy --region=$REGION --role-name=$NODE_ROLE_NAME --policy-arn=$ALB_POLICY_ARN
M00nF1sh picture M00nF1sh on 18 Apr 2019
👍7

Makes sense. (Note that I'm not using kube2iam due to the learning curve ATM).

So I ran the following as adapted from your snippet above:

export REGION=us-east-1
export NODE_ROLE_NAME=nodes.seenickcode.com
export ALB_POLICY_NAME=alb-ingress-controller
export ALB_POLICY_ARN=$(aws iam create-policy --region=$REGION --policy-name $ALB_POLICY_NAME --policy-document "https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/master/docs/examples/iam-policy.json" --query "Policy.Arn" | sed 's/"//g')
aws iam attach-role-policy --region=$REGION --role-name=$NODE_ROLE_NAME --policy-arn=$ALB_POLICY_ARN

I then deleted the alb-ingress-controller via kubectl delete deploy alb-ingress-controller -n kube-system, re-applied my k8s config as per my original post and I seem to see get the issue:

kubectl -n kube-system logs -f alb-ingress-controller-974d48f98-jpcgc    
W0418 18:36:08.324638       1 client_config.go:548] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
-------------------------------------------------------------------------------
AWS ALB Ingress controller
  Release:    v1.1.0
  Build:      git-72962fcb
  Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller.git
-------------------------------------------------------------------------------

I0418 18:36:08.381597       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0418 18:36:08.386017       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0418 18:36:08.386108       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I0418 18:36:08.386329       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0418 18:36:08.386380       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I0418 18:36:08.386548       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0418 18:36:08.386932       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}}
I0418 18:36:08.387524       1 leaderelection.go:187] attempting to acquire leader lease  kube-system/ingress-controller-leader-alb...
I0418 18:36:24.766468       1 leaderelection.go:196] successfully acquired lease kube-system/ingress-controller-leader-alb
I0418 18:36:24.866865       1 :0] kubebuilder/controller "level"=0 "msg"="Starting Controller"  "controller"="alb-ingress-controller"
I0418 18:36:24.967049       1 :0] kubebuilder/controller "level"=0 "msg"="Starting workers"  "controller"="alb-ingress-controller" "worker count"=1
E0418 18:36:25.009142       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: AccessDeniedException: User: arn:aws:iam::628863141346:user/kops is not authorized to perform: tag:GetResources\n\tstatus code: 400, request id: dedf7e6f-6208-11e9-966c-1d85a4117bf1"  "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"fcc"}
E0418 18:36:26.026824       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: AccessDeniedException: User: arn:aws:iam::628863141346:user/kops is not authorized to perform: tag:GetResources\n\tstatus code: 400, request id: df79c358-6208-11e9-8341-097c52f5be7c"  "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"fcc"}

Note 1: It appears that role for my nodes as set up by kops, nodes.seenickcode.com indeed now correctly has the additional policy per your instructions, including this required permission tag:GetResources as shown in the error above.

Note 2: FWIW, I have resources in a default namespace yet my ALB is in the kube-system namespace.

:-/

seenickcode picture seenickcode on 18 Apr 2019

@seenickcode
Sorry for the late reply, was on a meeting.

I just noticed you manually configured AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the controller yaml. By this way, the controller will use these credentials to talk to AWS, in your case, it's arn:aws:iam::628863141346:user/kops, so you'll need to grant policy to that user.

Instead, this is not secure by include credentials in clear text in yaml. You can just remove the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env from controller yaml, and reply it. After remove it, the controller will automatically use the role of worker node to talk to AWS, which should have the required permissions(granted by the commands i posted above).

M00nF1sh picture M00nF1sh on 18 Apr 2019
🎉3

No worries.

Great, that worked! Thank you @M00nF1sh very much for the prompt help.

seenickcode picture seenickcode on 19 Apr 2019

Hello, could you plz tell me where you got these logs, maybe I have the same issue with you
E0418 16:25:38.056300 1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="no object matching key "default/fcc" in local store" "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"fcc"}
E0418 16:25:39.178730 1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: AccessDeniedException: User: arn:aws:iam::628863141346:user/kops is not authorized to perform: tag:GetResourcesntstatus code: 400, request id: 9a659b4f-61f6-11e9-9dd7-0dc9efef4ad4" "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"fcc"}
E0418 16:25:40.205980 1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: AccessDeniedException: User: arn:aws:iam::628863141346:user/kops is not authorized to perform: tag:GetResourcesntstatus code: 400, request id: 9b0202f6-61f6-11e9-9cc1-7d52d613338e" "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"fcc"}

hardvintage picture hardvintage on 19 May 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

gigi-at-zymergen picture gigi-at-zymergen  路  5Comments
rdubya16 picture rdubya16  路  4Comments
jwickens picture jwickens  路  4Comments
madhu131313 picture madhu131313  路  3Comments
jcderr picture jcderr  路  3Comments