Aws-load-balancer-controller: Added rule to Ingres but controller is not creating target-group

[mlankenau]

I have added a rule in our existing ingress. While the DNS entry is created for the new sub-domain, the target group is not created and not added to the listener. I see no changes in the log of the ingress controller:

AWS ALB Ingress controller
  Release:    v1.1.2
  Build:      git-cc1c5971
  Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller.git
-------------------------------------------------------------------------------

W1104 09:56:36.140785       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1104 09:56:36.180863       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I1104 09:56:36.181158       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I1104 09:56:36.181227       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I1104 09:56:36.181436       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I1104 09:56:36.181516       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"=
I1104 09:56:36.181657       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I1104 09:56:36.181938       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="alb-ingress-controller" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}}
I1104 09:56:36.182139       1 leaderelection.go:205] attempting to acquire leader lease  kube-system/ingress-controller-leader-alb...
I1104 09:56:53.733637       1 leaderelection.go:214] successfully acquired lease kube-system/ingress-controller-leader-alb
I1104 09:56:53.833956       1 :0] kubebuilder/controller "level"=0 "msg"="Starting Controller"  "controller"="alb-ingress-controller"
I1104 09:56:53.934133       1 :0] kubebuilder/controller "level"=0 "msg"="Starting workers"  "controller"="alb-ingress-controller" "worker count"=1

[alejandrox1]

Hi @mlankenau
could you please share more about your configuration? Just to start, could you please share the alb ingress controller version and the Ingress definition for your resource please?
Could you also tell us how you installed the alb controller and in what namespace?

/triage needs-information
/assign

[mlankenau]

The alb ingres controller is docker.io/amazon/aws-alb-ingress-controller:v1.1.2

The deployment file is:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alb-ingress-controller
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: alb-ingress-controller
    spec:
      containers:
        - args:
            - --ingress-class=alb
            - --cluster-name=eksProd
            - --aws-vpc-id=vpc-xxxxxxx
            - --aws-region=eu-central-1
            - --aws-api-debug=true
            - --aws-max-retries=10
            - --feature-gates=waf=false
          env:
            - name: AWS_ACCESS_KEY_ID
              value: XXXXXXXXXXXXXXXXXXXXX
            - name: AWS_SECRET_ACCESS_KEY
              value: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
          image: docker.io/amazon/aws-alb-ingress-controller:v1.1.2
          imagePullPolicy: Always
          name: server
          resources: {}
          terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      serviceAccountName: alb-ingress
      serviceAccount: alb-ingress

The ingres file is (I removes some rules, there are alot):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-price-api-ingres
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/subnets: subnet-xxxxxxxxxxx,subnet-yyyyyyyyyyy,subnet-zzzzzzzzzzzz
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-central-1:xxxxxxxxxxx:certificate/xxxxxxxxxxxxxxxxxxxx
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/load-balancer-attributes:idle_timeout.timeout_seconds: 30
spec:
  rules:
    - host: test.2trde.com
      http:
        paths:
          - backend:
              serviceName: test-seller-service
              servicePort: 5000
    - host: elasticsearch.2trde.com
      http:
        paths:
          - backend:
              serviceName: elasticsearch-service-public
              servicePort: 9201

the alb controller is deployed in kube-system. I installed it following the docs here. The last change was chaning to 1.1.2 cause I had some troubles with 1.1.1. After that I added rules and that worked fine. Today the problems started with adding another rule.

[alejandrox1]

Thank you @mlankenau .
One possible area of trouble may be the fact that the ingress is in a different namespace that the controller (I don't see any namespace field in the Ingress field so im assuming it is in the default namespace).
Afaik, the controller will only detect Ingress resources that are in the same namespace.
So you may want to install another alb ingress controller in the default namespace.

This is something that needs to be fixed. I was hoping to look into it.
But for right now, could you try putting the controller and the ingress in the same namespace?

[mlankenau]

Yes the controller is controller is in kube-system and the ingres in default. But I never changed that and it worked b4. But I can try to change it.

My concern is: I have to avoid to get a new alb. We have an external DNS pointing to the existing DNS and it will be a pain to change. If I move the controller, will it keep the existing alb?

@alejandrox1 Thx for your help!

[alejandrox1]

that interesting. What changes happened from before? When did it stop working having one alb ingress controller watching all the namespaces?

Also, with the configuration as is, the ALB was actually created?

And as an aside, https://github.com/kubernetes-sigs/external-dns makes for a good combination when using this sort of project.That way you can manage dns records declaratively.

[mlankenau]

I have just added rules. I see no changes that could stop it from working.

I just upgraded the version since the initial installation.

I have just seen in the documentation:
# Limit the namespace where this ALB Ingress Controller deployment will
# resolve ingress resources. If left commented, all namespaces are used.
# - --watch-namespace=your-k8s-namespace

So looks like it is supposed to use all namespaces. What do you think? I can try setting it to default

[gregkeys]

any solution for this, I'm experiencing the same issue

[paulalex]

I am also intermittently seeing this issue

[paulalex]

And also where it fails to create a listener on the ALB, but the ALB is consistently created

[sjarking]

• make sure you use at least version 1.1.4 of the alb-ingress-controller
• add annotation alb.ingress.kubernetes.io/target-type: ip to the ingress.yaml of your deployment
• make sure you have private and public subnets that are tagged correctly
  That did the trick for me.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.