Aws-load-balancer-controller: Ingress Group Feature Testing

[M00nF1sh]

I’d like to give you an update about the ingress group feature for ALB Ingress Controller.
It’s still in alpha stage, please don’t use it in production environment.
I’m really appreciate if you can test it and give feedbacks on this feature.

Here is the docker image: docker.io/amazon/aws-alb-ingress-controller:v1.2.0-alpha.1, and code is available here: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/tree/v2 (If you’d like to build it yourself, simple make docker-build is enough 😃).

Here are the steps to install it

1. You need to delete all existing ingresses and previous aws-ingress-controller if you have.
   I’ll provide an patch update to the original aws-ingress-controller so that old aws resources(alb/tg/sg/etc) provisioned can be preserved in this new version, so users can upgrade seamlessly. (In this new version, aws resource are identified based on tags instead of names, so ideally you can bring your own alb/targets etc, but there is no plan to support such case offically).

1. Install this new version using same steps as before, but use image docker.io/amazon/aws-alb-ingress-controller:v1.2.0-alpha.1.

Here are the steps to test it

1. Create ingress resources as you originally do, and attach annotations. Almost all(keep updating :D) old annotations should work as before, and we’ll provision one ALB per Ingress resource.
2. You can add annotation alb.ingress.kubernetes.io/group.name: yourgroupname to your ingresses(even across namespaces) that should be hosted with single ALB.
   
   
   
   1. Simply add that annotation to all your ingresses that should be same group, and apply it will be enough. The groupName shouldn’t contain “/”, and might be limited to [a-z0-9]{1,10} in the future.(based on feedbacks).
   
   
   2. Ingresses can join group(add that annotation), leave group(remove that annotation) and change group(change the groupName). Currently it will works as expected, but we may drop the support for this, since the group should be statically decided by application owners before create ingresses.
   
   
3. You can add annotation alb.ingress.kubernetes.io/group.order: 1-1000 to change the rule ordering between ingresses.
   
   
   
   1. By default the rule order between ingresses are determined by ingress’s namespace/name.
   
   
   2. You can explicitly denote the order using a number between 1-1000(you need to quote numbers in yaml with “”).
   
   
   3. The smaller the order, the rule will be evaluated first. (all ingresses without explicit order setting get order value as 0). And You cannot have two ingress with same explicit order.
   
   

Typical use cases:

1. Share ALB for ingresses across namespaces.
2. Specify different rule for different ports by create two ingress with same group, with different listen-ports settings.
   
   
   
   1. Some settings are aggregated by ports. (e.g. http routing rule /ssl certs are merged together for same port).
   
   
   2. Some settings must be specified on a single ingress. (e.g. ssl policy/inbound cidrs for same port, you still can specify different inbound cidr for different port in different ingress)
   
   

I’ll keep update the image and code, and you can reach me on slack by @M00nF1sh 😃.

[M00nF1sh]

The current approach is built an AWS data model first and deploy it. (the data model is logged in controller logs, so if you encountered any bugs, please send me these logs).
It should only take a few hours to write an cloudformation based deployer, but i hesitate to do it due to some critical cloudformation bugs i found during testing. and it's not possible to maintain backwards compability for existing stack.

[allanyung]

I tried using this but unfortunately it fails with an error about the Security group being invalid

{
    "level": "error",
    "ts": 1555051159.8551583,
    "logger": "kubebuilder.controller",
    "msg": "Reconciler error",
    "controller": "alb-ingress-controller",
    "request": "/SOME_PATH",
    "error": "ValidationError: Security group 'SOME_SECURITY_GROUP' is not valid\n\tstatus code: 400, request id: SOME_ID",
    "stacktrace": "sigs.k8s.io/aws-alb-ingress-controller/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/sigs.k8s.io/aws-alb-ingress-controller/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/aws-alb-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/sigs.k8s.io/aws-alb-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/aws-alb-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/sigs.k8s.io/aws-alb-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nsigs.k8s.io/aws-alb-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/sigs.k8s.io/aws-alb-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nsigs.k8s.io/aws-alb-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/sigs.k8s.io/aws-alb-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nsigs.k8s.io/aws-alb-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/sigs.k8s.io/aws-alb-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"
}

The configs are exactly the same as they were before with the exception that I added the alb.ingress.kubernetes.io/group.name annotation to the ingresses. The logs say that the model was successfully built. I don't want to post it here without anonymising some of the data, but that may render it useless to you. Let me know if it's needed. Cheers

[M00nF1sh]

@allanyung Hi, would you share the securityGroup name to me? It's based on the groupName, and i haven't do validations to it. In the mean while, you can change groupname to [a-z0-9] only.

[allanyung]

Ah, the security group name does contain hyphens. I expect that's why it isn't working. The groups are created in advance by another process. Will hyphens be supported or is it a dealbreaker?

I'm won't be in a position to try this again for about a week, but am happy to try again if it would still be helpful at that time.

[M00nF1sh]

Ah, the security group name does contain hyphens. I expect that's why it isn't working. The groups are created in advance by another process. Will hyphens be supported or is it a dealbreaker?

I'm won't be in a position to try this again for about a week, but am happy to try again if it would still be helpful at that time.

hyphens is fine. It will be super helpful if you can share your group name to me 😄

[allanyung]

I can share the security group name and group name, nothing sensitive there

security group name: app-alb-ingress
group name: apps

Hope it helps!

[M00nF1sh]

@allanyung Just fixed it 😄 Delete & reinstall the controller will solve it 😄

[nisan270390]

@M00nF1sh this is great feature, thanks a lot :)
We've one ALB which serve 5 different services (outside k8s), the routing configured by path filter.
We want to migrate our existing services to k8s step by step, Is there a way can we assign the existing ALB which spin up through cloudformation, and route traffic for /api/customers for example to service powered by k8s and another routes to existing service target groups (cloudformation)?

[M00nF1sh]

@nisan270390 Currently it's not possible, since the controller don't know how to cooperate with existing rules on ALB.
An alternative solution is make EndpointBinding an CRD, so you can manually create targetGroup & add rules, and let the controller manage the targets for that targetGroup to be sync with k8s service. I haven't exposed it as CRD mainly because this feature won't be needed for everyone, and require these ppl to install an extra CRD seems not good.

BTW, have you considered this way to migrate existing service?

[nisan270390]

@M00nF1sh thank you, Migrating From Legacy is very helping for our migration to k8s.

[allanyung]

@M00nF1sh tried this again and it looks like it works great!

One thing I would say is that I wanted one specific ingress group to be ordered last, and don't really care about the ordering of the other groups. The current implementation meant that I had to specify the alb.ingress.kubernetes.io/group.order for all the other groups, whereas the ideal situation would be if I could just specify alb.ingress.kubernetes.io/group.order: 1000 for the one group I want resolved last, and have everything else auto resolve to something < 1000.

Would such a change be reasonable? Thanks!

[M00nF1sh]

@allanyung Yeah, it's make sense. I think the most common use case of explicitly use group.order should be put an catch-all rule at the last. So make the default order to be 0 make sense.
(don't like to make the default order to be like 500, which is hard to use and understand :D)
If it make sense to you, I'll provide a update today.

BTW, Another approach is use the creationTimestamp as default order(nginx controller used that approach), i didn't adopt that cause it's introduced unstable behavior, so hard to implement infrastructure automation.

[allanyung]

@M00nF1sh I think default 0 makes the most sense.

CreationTimestamp sounds like a recipe for disaster :sweat_smile:

[Annegies]

@M00nF1sh Tested the ingress group feature and it works like a charm!
I've tested the grouping of ingresses both inside and cross namespace.

[allanyung]

Hi @M00nF1sh I've encountered something which is probably fine, but looks a bit weird.

We use kustomize to create namespaced versions of our application.

So we have a default that looks like this

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app
  namespace: default
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: app
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/security-groups: app-alb-ingress
    alb.ingress.kubernetes.io/certificate-arn: SOME_CERT
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
  - host: SOME_URL
    http:
      paths:
      - backend:
          serviceName: app
          servicePort: 80

and kustomize additionally generates this resource (different namespace, URL, and SSL cert)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app
  namespace: non-default
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: app
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/security-groups: app-alb-ingress
    alb.ingress.kubernetes.io/certificate-arn: SOME_OTHER_CERT
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
  - host: SOME_OTHER_URL
    http:
      paths:
      - backend:
          serviceName: app
          servicePort: 80

They are merged into a single alb. However, this results in the SSL redirection rule appearing multiple times in the listener rules for port 80. It still works fine, but looks a bit odd. I can probably fix this by moving stuff around in kustomize templates so that the SSL redirect stuff doesn't get defined twice. But I guess this could happen via other means and maybe the ingress group feature could deduplicate occurrences of the SSL redirect rule? No worries if that's too difficult or out of scope, things still work regardless. Thanks!

[M00nF1sh]

@allanyung
Actually I preferer another way to do http->https redirection after ingress group(define different rule for different port in different ingress 🤣 )

(dedupe make sense too, but it might be error prone when we support advanced routing like header/verb/query based, I need to think more about it.)

1. for all our normal HTTPS rule, create them with alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
2. create an single ingress in default namespace(or other), with alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]' and the direction rule.

It's more clean than rely on a trick which omit redirection rule on 443 port.

[Spareo]

I'm trying this out right now and running into an issue. I have one ALB where I use the alb.ingress.kubernetes.io/inbound annotation for specifying a list of allowed CIDR blocks but I keep getting errors like this

"msg":"Reconciler error","controller":"alb-ingress-controller","request":"/MySecurityGroup","error":"InvalidParameterValue: CIDR block 1.2.3.4/32 is malformed

This results in the security group getting created but not having any rules inside of it.

On the stable release version, I don't get this error.

[enmand]

Daniel Enman smiled at you

[allanyung]

@allanyung
Actually I preferer another way to do http->https redirection after ingress group(define different rule for different port in different ingress )

(dedupe make sense too, but it might be error prone when we support advanced routing like header/verb/query based, I need to think more about it.)

1. for all our normal HTTPS rule, create them with alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
2. create an single ingress in default namespace(or other), with alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]' and the direction rule.

It's more clean than rely on a trick which omit redirection rule on 443 port.

@M00nF1sh This is a much nicer solution! I can confirm that this works, and there are no longer duplicates in the port 80 rules. Did you get a chance to cut a release where the default group order is 0 instead of 1000? Thanks :)

[M00nF1sh]

@Spareo
Hi, Thanks for reporting this error. I have fixed the issue, reinstall(delete/apply) the controller will fix it if you have 'imagePullPolicy: Always'. 😄 (Also, IPV6 CIDR should work fine too)

@allanyung
Hi, I have pushed the image to the same tag. reinstall(delete/apply) the controller will fix it if you have 'imagePullPolicy: Always'. The default order is 0, and users can explicitly set it to [1, 1000] 😄

[arminc]

What are the things that need to be done/verified to get this feature to master and released?

[mixman]

An ALB seems to have a 100 rule limit, which I assume means a theoretical 100 service limit (given single path specified).

How does one support over 100 services? Is it possible to have multiple ALB's running for this grouping feature?

[rabbitfang]

@mixman The default rule does not count towards the 100 rule limit, so the theoretical max is 101 services served by a single ALB. It does not appear to be possible to request an increase for that limit, so in order to support more than 100/101 services, you would need to use multiple ALBs (and therefore multiple ingress groups), using host names to partition the ALBs. Since this feature is meant to group multiple ingresses into a single ALB, I doubt it would be able to automatically manage multiple ALBs when you go over the rule limit.

[iverberk]

Hi, we are waiting for this functionality to be merged. Could you give us a status update on the progress and if we can help in any way to get this merged?

[eugenekainara]

Hello, I'm also checked this functionality at my EKS - and it works good for my use case. Now I have just 2 ALB instead of 6. Thanks!
When we can expect this feature to be merged?

[M00nF1sh]

Hi, sorry the late update. I didn't get enough time to thoroughly test all features out.(it's almost rewrite and needs thoroughly testing). I'll submit an PR to merge this by next week.

[EricDube]

@M00nF1sh I ran into a similar issue to the CIDR block malformed but I believe its from using ip-address-type dualstack where staging is my stack.

"msg":"Reconciler error","controller":"alb-ingress-controller","request":"staging/staging-ingress","error":"InvalidParameterValue: CIDR block ::/0 is malformed

It is creating all the target groups and a security group with no inbound rules.

[arminc]

@M00nF1sh any date on the PR? Don't want to pressure but there is a lot of need for it. If I can do something to help let me know.

[M00nF1sh]

@arminc
There is still two pending implementation left in my dev environment to process with this.

1. omit worker node securityGroup. (currently, we are creating an worker node SG per ingress, in the newer version, it will directly use the existing securityGroup on worker node with k8s tag instead).
2. add existing tags on resources to align with the tagging strategy with the newer version. (the newer version uses tags instead of names to identify existing resources, by doing this, it will allow us to preserve ALB/TargetGroup/SG created via old alb controller version when migrate to new alb controller version).
   I'm still working on the two implementations to ensure there are smooth migrations between versions.

[rverma-nikiai]

@M00nF1sh eagerly waiting for this pr to get merged.

[rafaelgaspar]

I've tested it and it's working like it should, only thing is right now it's not respecting --alb-name-prefix.

[runningman84]

I've also tested it. Compared to the traefik ingress the path handling seems to be different.

In traefik I can access all deep links like /wp-login.php in alb I can only access / with this config:

Name:             demo-wordpress
Namespace:        demo
Address:          k8s-wordpress-96977de690-941162455.eu-central-1.elb.amazonaws.com
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                                           Path  Backends
  ----                                           ----  --------
  demo.example.com  
                                                 /   demo-wordpress:http (10.1.40.88:8080)
Annotations:
  alb.ingress.kubernetes.io/group.name:                 wordpress
  alb.ingress.kubernetes.io/target-type:                ip
  traefik.ingress.kubernetes.io/preserve-host:          true
  traefik.ingress.kubernetes.io/redirect-entry-point:   https
  alb.ingress.kubernetes.io/certificate-arn:            arn:aws:acm:eu-central-1:xxxxx:certificate/yyyyy-d094-4e5e-b75d-4427810762a6
  alb.ingress.kubernetes.io/listen-ports:               [{"HTTPS": 443}]
  alb.ingress.kubernetes.io/scheme:                     internet-facing
  ingress.kubernetes.io/ssl-proxy-headers:              X-Forwarded-Proto:https
  kubernetes.io/ingress.class:                          alb
  traefik.ingress.kubernetes.io/frontend-entry-points:  http,https
Events:                                                 <none>

It looks like the alb rules have a fixed path match. I would expect alb to do not create a path rule if there is only a single path like /.

[rifelpet]

You need to use a path of /* rather than /

[runningman84]

which is a pain if you want to easily switch between ingresses.... this issue seems to mention the same problem:
https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/699

[rverma-nikiai]

I do prefer /* over / since it's consistent with alb behaviour. Also gce ingress controller behaves same and various helm charts(e.g. spinnaker) are following the convention. Having this change will break compatibility across exisiting services

[xxthegonzxx]

Let's get this bad boy in soon! I appreciate all the hardwork. You're a rockstar!

[tdmalone]

Just wanted to check if Ingress Groups solves all of these related issues?

• https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/228 Use an existing ALB
• https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/298 Create option to reuse an existing ALB instead of creating a new ALB per Ingress
• https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/688 Feature request: shared ALB across multiple Ingress objects
• https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/724 Share the single ALB

I _think_ it solves all of these mostly-similar requests - except for perhaps the use case of re-using an ALB created _outside_ of the ingress controller, which (from this description in the first post here) sounds like it's unofficially supported if you set the right tags on your existing ALB:

In this new version, aws resource are identified based on tags instead of names, so ideally you can bring your own alb/targets etc, but there is no plan to support such case offically

So yeah, I just wanted to confirm if this feature solves the above cases ^^ and if so, wondering if those issues can be closed to consolidate the discussion here?

[iautom8things]

What is the ETA for this getting merged?

[rverma-nikiai]

@M00nF1sh can you please comment what's blocking on this?

[ronit29]

i am using the group feature but its not working as expected
one of my ingress uses the HTTPS protocol and for another i do not want to use it.
here are the two versions

`apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: internal-ingress
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/certificate-arn: xxxxx
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60
    nginx.ingress.kubernetes.io/use-regex: "true"
    alb.ingress.kubernetes.io/subnets: "xxxx"
    alb.ingress.kubernetes.io/group.name: myapp
    alb.ingress.kubernetes.io/group.order: "2"
  labels:
    app: internal-ingress
spec:
  rules:
    - host: mydomain.com
      http:
        paths:
          - path: /*
            backend:
              serviceName: ssl-redirect
              servicePort: use-annotation
          - path: /*path.xml
            backend:
              serviceName: myservice
              servicePort: 8080

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: internal-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]'
    alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60
    nginx.ingress.kubernetes.io/use-regex: "true"
    alb.ingress.kubernetes.io/subnets: "xxxxxx"
    alb.ingress.kubernetes.io/group.name: myapp
    alb.ingress.kubernetes.io/group.order: "1"
  labels:
    app: internal-ingress
spec:
  rules:
    - host: www.mydomain.com
      http:
        paths:
          - path: /*
            backend:
              serviceName: nodejsservice
              servicePort: 3000
`

the functionality i want is if someone tries to access the domain with www prefix it should redirect to https version of the domain without the prefix in one redirect , which has been configured in the node js service
is the any thing which i can do in the ingress config itself for this.
www.mydomain.com ==> https://mydomain.com

but when i access the www.mysomain.com, it isn't working as expected it still somehow first redirects to the https version first the the nodejs service comes into play.

[JorisAndrade]

@M00nF1sh any news on this? 🙏

[M00nF1sh]

Hi, sorry for late replies. I haven't got enough time in the past months to work on this feature.
(TBH, kind of shame to reply here since didn't do work on this recently to push it forward )

I have created an V2 branch in the official repo to track this work.
Next step plans:

1. release a an official alpha version(v1.2.0-alpha.1) n for v2 branch over this weekend after fix bugs mentioned in this thread.
2. submit PR to master branch for migration and release new version v1.1.3
3. migrate integration tests cases from master branch over the v2 branch
4. address feature parity in V2 branch
5. release official production version of v1.2.0

[JorisAndrade]

Sounds good! Thanks for taking care of this, happy to test the alpha when ready.

[M00nF1sh]

@ronit29
You shouldn't use same name in your Ingresses(i assume it's a typo).
If I understand you correctly, you want to redirect directly from http://www.mydomain.com to https://mydomain.com, and your nodejsservice is doing the job? Then you should check your nodeJS application to see why it isn't happening.
BTW, you don't need an additional service to do the redirect, you can create an annotation based redirect like alb.ingress.kubernetes.io/actions.domain-ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "Host": "mydomain.com", "StatusCode": "HTTP_301"}}' (you can use Host in redirect as well :D)
another BTW, for your case, u don't even need the ingress group feature as well, you can specify the rules in a single ingress, and it's easier to maintain

[mclueppers]

This is what I'm waiting for in order to finish the migration from ECS to K8s. I hope we have an early beta or, why not, even stable release any time soon.

[ronit29]

Type

thnxx...@M00nF1sh i ditn'd knew that Host could be used in the Options

[dahu33]

@M00nF1sh this feature is really lacking in the stable version and blocking many. If you don't have the time to work on it, could someone else in your team or at AWS finishes it?

[kevinmartin]

Testing this works great. Seems like WAF support was removed though. I don't see any support for it anywhere in the new codebase (apart from an unused FeatureGate) nor any annotation support: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/f8556d824c158201ecbda09bf25c4dc2ae610896/pkg/k8s/annotations.go#L12-L54

[bkosick]

I agree this is working very well in Alpha, I'd really like to see it progress... I would humbly like to suggest a few features.

1) The ability to specify "group" features.... IE create a group by name, and set the Cert, HTTP/HTTPS settings, subnets, etc etc to the group. That way when you add a new service and path behind the ALB, you just need 2 or 3 annotations, goup name, path and healthcheck.

2) Have the ability to specify the default group.order, and if you do have a group.order specified for one ingress, auto-increment the rest if left blank. I currently have to specify my default rule as 1000, and because I specified that one, I must specify all the other group/rule orders as well...

[M00nF1sh]

@bkosick There is an ongoing effort in upstream to make IngressClass a standalone resource.
After it happens, we can add annotations(or fields) on that IngressClass to like 1. whether ingress belong to this class should be within a "group", 2. common annotations apply to all ingresses. So that we can have a IngressClass object per Ingress Group :D.

1. You mean provide a setting for default group.order in controller level?

@kevinmartin WAF will be supported, the v2 is not feature complete with current master version but will be(just didn't got enough time to make it complete as of now) :D

[tomhaynes]

@M00nF1sh is it possible to get an estimate on when a stable release will be ready? We're also very keen to start using it

[JorisAndrade]

Hi @M00nF1sh, excited users of the V2 on a testing cluster.
Would it be possible to repackage the v2 to include the aws-sdk update commit? :)

[s2504s]

HI @M00nF1sh
Thank you for the 1.2 version. It looks like very nice tool and I would like to use it for our production. Could you please update the status of this version?
Thanks

[yuhuali1989]

@M00nF1sh Great job! I am wondering how we can reuse an alb from different eks clusters?
I suppose that there should be some annotations to specify alb id.

[oded-dd]

Great feature! Making solutions like nginx ingress redundant! When it will be production ready?

[aholbreich]

Eny plans when this can be used? and if so any tutorial? example?

[yrsurya]

can I know with this feature if I can use a single ALB and ALB controller per EKS cluster and point external-dns to multiple apps?

[rayterrill]

Fingers crossed the radio silence means this is coming at ReInvent... :)

[wjl120]

@allanyung
Hello,My customer want to use group feature on aws EKS ,but this feature version is alpha, so I want to know when the group feature can be release and used in product environment? thank you.

[siqusoft]

Hi, I was able to test this feature in our cluster and it works without any issue so far!! Any update On GA version of this image? we need this for our production workload

Thank you...

[M00nF1sh]

Hi, i still need to more testing and backfill features(like Fargate support) for this to be announce as GA. I have been switched to other projects in the past months, but should be back after reinvent.

[s2504s]

@M00nF1sh
Could you please clarify, have you updated aws-alb-ingress-controller:v1.2.0-alpha.1 with features that you released in the v1.1.4 ?

[ficofer]

Hey,

@M00nF1sh thanks for the work and support, quick question is there any plan in the near future to merge/release a beta of this ?

I see that is base and v1.2.0 is there a plan to rebase to v1.1.4 ?

Thanks!

[Surian]

@M00nF1sh any update on this? :)
This is such a great feature that is really missing today!

[mrballcb]

Can someone give a quick go build guide when you have a fresh checkout of the repo and on the v2 branch? It seems to take more than just go mod init sigs.k8s.io/aws-alb-ingress-controller && go mod vendor to get this to build. The module behavior in the current go version doesn't copy all of the things expected to build the code (deepcopy in pkg/apis/apis.go) nor in the Makefile (controller-tools). (I haven't gotten further than that, so there may be more).

$ make docker-build
<snip>
go vet ./pkg/... ./cmd/...
go run vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go all
stat vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go: no such file or directory
make: *** [manifests] Error 1

Golang's module support only copies things that it builds, not the entirety of the required module. After running go mod vendor, this is the contents of the vendor/sigs.k8s.io/ directory:

$ vdir vendor/sigs.k8s.io/
total 0
drwxr-xr-x   4 tlyons  staff  128 Jan  4 08:34 controller-runtime
drwxr-xr-x  14 tlyons  staff  448 Jan  4 08:34 yaml

What step(s) am I missing?

GOPATH=$HOME/repos/go
This repo os at $HOME/repos/aws-alb-ingress-controller

[anarwal]

@M00nF1sh

I am trying to create two different service and I want to use single Ingress resource and single ALB for them. I have added ingress resource as follows:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: <some-cert>
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-2016-08
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/security-groups: app-alb-ingress
    alb.ingress.kubernetes.io/group.name: app
    alb.ingress.kubernetes.io/listen-ports: '{"HTTPS":443}]'
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /*
            backend:
              serviceName: backend-example
              servicePort: 8080
          - path: /frontend/*
            backend:
              serviceName: frontend-example
              servicePort: 4000

then I have two following service:

apiVersion: v1
kind: Service
metadata:
  name: backend-example
  labels:
    app: backend
spec:
  - port: 8080
    targetPort: 8080
    protocol: TCP
  type: NodePort
  selector:
    app: backend

and

apiVersion: v1
kind: Service
metadata:
  name: frontend-example
  labels:
    app: frontend
spec:
  ports:
  - port: 4000
  selector:
    app: frontend

my health check is still failing on each target group, which is because of path. However I am not sure how to setup individual health check path? As I can't specify alb.ingress.kubernetes.io/healthcheck-path: /ping to ingress-resource because that will use same path on both services. Any suggestions?

[allanyung]

@anarwal you can set the healthcheck annotation on the services instead of the ingress

Based on your example:

apiVersion: v1
kind: Service
metadata:
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: /health
  name: backend-example
  labels:
    app: backend
spec:
  - port: 8080
    targetPort: 8080
    protocol: TCP
  type: NodePort
  selector:
    app: backend

and

apiVersion: v1
kind: Service
metadata:
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: /ping
  name: frontend-example
  labels:
    app: frontend
spec:
  ports:
  - port: 4000
  selector:
    app: frontend

[anarwal]

@allanyung that helped, thanks for tip!

[rverma-jm]

It seems this is taking forever to get merged.

[lancehunt]

@M00nF1sh where can we help on this? Cloned, built & ready to code. 😄

[M00nF1sh]

Can someone give a quick go build guide when you have a fresh checkout of the repo and on the v2 branch? It seems to take more than just go mod init sigs.k8s.io/aws-alb-ingress-controller && go mod vendor to get this to build. The module behavior in the current go version doesn't copy all of the things expected to build the code (deepcopy in pkg/apis/apis.go) nor in the Makefile (controller-tools). (I haven't gotten further than that, so there may be more).

$ make docker-build
<snip>
go vet ./pkg/... ./cmd/...
go run vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go all
stat vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go: no such file or directory
make: *** [manifests] Error 1

Golang's module support only copies things that it builds, not the entirety of the required module. After running go mod vendor, this is the contents of the vendor/sigs.k8s.io/ directory:

$ vdir vendor/sigs.k8s.io/
total 0
drwxr-xr-x   4 tlyons  staff  128 Jan  4 08:34 controller-runtime
drwxr-xr-x  14 tlyons  staff  448 Jan  4 08:34 yaml

What step(s) am I missing?

GOPATH=$HOME/repos/go
This repo os at $HOME/repos/aws-alb-ingress-controller

Hi, you need to

1. clone it to $GOHOME/src/sigs.k8s.io/aws-alb-ingress-controller
2. run dep ensure

[M00nF1sh]

@ficofer @Surian every feature in V1 will be supported in V2 as well.
@lancehunt I think migrate the existing from v1(or create a better) an e2e test framework would be an awesome help 🤣 .

I created an project to track the progress, will work more actively on this. (handing over other internal work to team members :D)

[rfarro82]

@M00nF1sh does this work with the newer versions of AWS api client that support IAM Roles for Service Accounts? When we deployed it to our cluster, the pod is trying to use our worker node instance profile rather than the IAM role arn we’ve associated with the serviceaccount on the pod.

[M00nF1sh]

@romanfarro the one in v2 is not using the latest AWSSDK(so it don't support iam for pods). But it will. I'm currently doing internal code reviews and add UTs/e2e for all these code.

[rverma-jm]

@M00nF1sh Why did we choose to expose the alb as ingress controller at first place, why didn't we exposed it as loadbalancer in kubernetes.

[aalimovs]

@M00nF1sh Why did we choose to expose the alb as ingress controller at first place, why didn't we exposed it as loadbalancer in kubernetes.

because that's what ingress was made for, to live above the service layer (including the loadbalancer type service you mention)

See here https://kubernetes.io/docs/concepts/services-networking/ingress/

[dza89]

Would be nice to have a v1.2.0-alpha.2 with all the features of v.1.1.6 integrated in it.

[ChipWolf]

In the below example, the ALB is deployed as expected for the user-frontend ingress with the correct certificate for *.example.com, and the R53 record is added as per the external-dns annotation.

However, for the admin-frontend and admin-websocket Ingresses; the log states at least one TLS Certificate must be specified for port 443. Only if I specify the certificate manually with the certificate-arn annotation does it work as expected.

It seems certificate-arn must be specified for all albs now or the ingress doesn't deploy.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: user-frontend
  namespace: app
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    external-dns.alpha.kubernetes.io/hostname: app1.example.com
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: user-frontend
          servicePort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: admin-frontend
  namespace: app
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/group.name: app2
    external-dns.alpha.kubernetes.io/hostname: app2.example.com
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: admin-frontend
          servicePort: 4326
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: admin-websocket 
  namespace: app
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":4367}]'
    alb.ingress.kubernetes.io/group.name: app2
    external-dns.alpha.kubernetes.io/hostname: app2.example.com
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: admin-websocket
          servicePort: 4367

[sshishov]

Sorry, maybe offtopic here. I would like to ask, will it support externally created ALB? Can I plug external ALB/NLB to EKS ingress using some annotations?

[ccampo133]

Is there a timeline for this feature to be released? It's been in alpha for over a year now.

[runningman84]

Right now there is an unresolved bug with the alpha 1.2 version which can cause all ingress resources to be unavailable.

https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/1234

It is a catch 22 situation. Either you pay a lot of money for dozens of loadbalancers and accept other problems in the old 1.1 line or you use 1.2 and save a lot of money but risk a downtime.

[M00nF1sh]

@ccampo133 i'll work on this after finished appmesh controller stuff. (likely by the end of this week)
@runningman84 i'm aware of the cause in alpha1.2, it's lacking a node watcher. i'll do a fix and release a new alpha 1.12 version contains the fix. (likely by the end of this week)

[eric-uct]

Hello, have you been able to make some progress on this since your last post? I just ran into this issue myself and am wondering if you've managed to start working on it yet.

[M00nF1sh]

@runningman84 @eric-uct sorry for the delay, i have pushed the fix to https://github.com/kubernetes-sigs/aws-alb-ingress-controller/commit/59708eeb08e221576a8cbed1c161d5e6491562a7, and built a image contains the fix: m00nf1sh/aws-alb-ingress-controller:v1.2.0-alpha.2
you can build your own by check v2 branch at your $GOPATH/src/sigs.k8s.io/ and run "dep ensure && make docker-build"

[siqusoft]

Thanks for pushing the changes! When are you planning to have GA version
and what is your opinion on using this version on production?

On Tue, Jun 2, 2020 at 6:53 PM M00nF1sh [e-mail] wrote:

@runningman84 https://github.com/runningman84 sorry for the delay, i
have pushed the fix to 59708ee
https://github.com/kubernetes-sigs/aws-alb-ingress-controller/commit/59708eeb08e221576a8cbed1c161d5e6491562a7,
and built a image contains the fix:
m00nf1sh/aws-alb-ingress-controller:v1.2.0-alpha.2
you can build your own by check v2 branch at your $GOPATH/src/sigs.k8s.io/
and run "dep ensure && make docker-build"

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/914#issuecomment-637870905,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEIOG2E7N6B6WN2WEACPAQDRUWGHTANCNFSM4HD7AXNQ
.

>

Thank you
Sirak Mulugeta

[runningman84]

@M00nF1sh alpha2 seems to work fine. Is this feature also available in alpha2?

alb.ingress.kubernetes.io/security-groups

Btw. it would be great if we could pass this an arg to the alb ingress controller which would use the settings for all ingresses and ignore all security group annotations. Otherwise if you have a single misbehaving ingress the whole security group stuff would be messed up.

[guitarrapc]

alpha2 works great for me. alb.ingress.kubernetes.io/security-groups also works fine.
I encounter simple restriction that same annotation alb.ingress.kubernetes.io/load-balancer-attributes: should be pass to same group ingress. In my case, I change to same s3 bucket prefix for same group.
Can't wait this feature release to GA.

[runningman84]

It works fine for me too. The only difference I recently discovered is that new loadbalancers are internal instead of internet facing if you don't specify the corresponding attribute.

Now I just need a feature to force the security group setting for all loadbalancers managed by alb ingress.

[evankicks]

Hi @M00nF1sh. I'been using the image aws-alb-ingress-controller:v1.1.6 earlier which has the fixes for issue which involved duplicate listener rules on both ports(80,443). But after trying the aws-alb-ingress-controller:v1.2.0-alpha.1 image to make use of the group feature i again see the duplicate rules on both listeners. can the group feature be implemented on top of v1.1.6 so we can mitigate the duplicate rules?

[zhujik]

When grouping together two ingresses that both have the same set of inbound-cidr defined, then this throws the following error:
"multiple Ingress defined CIDR for port 80"
However, the solution to this is to remove the inbound cidr annotation of the second ingress to join the group, the ALB will have a security group with correct inbound rules set.

As a suggestion: I think alb-ingress-controller could check if all ingresses in the group have the same set of inbound-cidr defined, and if not throw an error

[alexpekurovsky]

Few inputs from my side:

• If alb-ingress-controller manages more than 50 ingresses with the same group name, it leads to error from AWS side about maximum allowed amount of rules per load balancer.
  It worth adding logic to check how many rules/ingresses are already managed and add new ALB with pattern like: k8s-GROUPNAME-ID.
  Also alb-ingress-controller should respect current placement, so even if we have 50 rules in k8s-GROUPNAME-1 and 20 rules in k8s-GROUPNAME-2, when one of the rules in k8s-GROUPNAME-1 will be removed (due to ingress deletion in k8s, it shouldn't move any rules from k8s-GROUPNAME-2 to k8s-GROUPNAME-1, as it will require DNS change and result in issues with DNS cache on client side.
• Could alpha release with grouping feature synced with 1.1.7 features (at least)?

[Adirael]

This looks great!

Am I correct in that this doesn't work yet with Fargate, right?

[dave-meier]

@M00nF1sh -- This feature is working great for me, thank you! Will it be released soon?

[VarroReve]

Mark and waiting for the release.

[rfarro82]

@M00nF1sh was the latest AWSSDK added to the alpha2 so it supports iam for pods?

[dmanchikalapudi]

@M00nF1sh - Any idea when this will be available as a release?

[timmjd]

The absence of this feature really make the ALB unusable in large installations. You will end up in having a single ALB per application - we are currently planing to get rid of the ALB ingress and go back to good old nginx ingress.

[M00nF1sh]

hi, we are actively working on this, and will release a GA version in Q3 2020. :D
which includes follow features.

• IngressGroup described in this thread.
• NLB-IP mode support for K8s service
• new CRD named TargetGroupBinding.

[M00nF1sh]

Hi guys, just want to let you know we have a RC1 version out: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/releases/tag/v2.0.0-rc1
It supports all features existing V1 supports with all new features. :D

[astrived]

we are doing the final phase of testing for the new aws-loadbalancer-controller and the RC image is here https://github.com/kubernetes-sigs/aws-alb-ingress-controller/releases/tag/v2.0.0-rc5. You can find details here https://github.com/kubernetes-sigs/aws-alb-ingress-controller/tree/v2_ga. Stay tuned for What's new coming soon !

[AndrewBoklashko]

@astrived is it possible to update Helm chart at https://github.com/aws/eks-charts/tree/master/stable with the latest RC and publish it to https://aws.github.io/eks-charts ? We would prefer to install it using Helm, but current chart seems to be for rc2.

[astrived]

@AndrewBoklashko since we are very close to finalizing the release we were preferring to put a final version into eks-charts, thanks for your patience !

[runningman84]

Please fix this before going final
https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/1524

[astrived]

We have published the v2.0.0 release at https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v2.0.0 The documents have been updated https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html and https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html and the blog post is here https://aws.amazon.com/blogs/containers/introducing-aws-load-balancer-controller/

[astrived]

@runningman84 1524 lets track that separately @M00nF1sh has some ideas around this issue

[M00nF1sh]

closing this issue as it's finally released 😸