Aws-load-balancer-controller: Support Advanced Request Routing

[Rowern]

The Advanced Request Routing for ALB was just announced: https://aws.amazon.com/fr/blogs/aws/new-advanced-request-routing-for-aws-application-load-balancers/

Do you have an ETA for implementing this ?
One of our use case would be to have the same path defined in our ingress but with different "backend" that would be routed by HTTP method request (POST or GET).
This would be especially useful for people working with Lambdas.

I have no idea if the ingress spec allow for defining twice the same path, if it does not maybe a specific annotation could work.

[M00nF1sh]

Hi, How would you like it be implemented?

Alternative 1 - Use Ingress Group: the routing.condition on ingress will affect all rules inside that ingress, and all ingress belongs to same group(mysvc-group) will be aggragted on same ALB/Listener.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: mysvc-get
annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: mysvc-group
    alb.ingress.kubernetes.io/routing.condition.header:header1=b,header2=d
    alb.ingress.kubernetes.io/routing.condition.verb:GET,HEAD
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /api
        backend:
          serviceName: service-blue
          servicePort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: mysvc-post
annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: mysvc-group
    alb.ingress.kubernetes.io/routing.condition.header:header1=b,header2=d
    alb.ingress.kubernetes.io/routing.condition.verb:POST,PATCH
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /api
        backend:
          serviceName: service-green
          servicePort: 8080

Alternative 2 - Extend the use-annotation:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-foo
annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/actions.api-rule-cfg-get: '{"conditions":["verb": ["get", "head"]]:, "backend": [{"serviceName:service1", "servicePort":80, "weight":80}, {"serviceName:service2", "servicePort":80, "weight":20}]}'
    alb.ingress.kubernetes.io/actions.api-rule-cfg-post: '{"conditions":["verb": ["get", "head"]]:, "backend": [{"serviceName:service1", "servicePort":80, "weight":80}, {"serviceName:service2", "servicePort":80, "weight":20}]}'
spec:
  rules:
  - host: www.example.com
    http:
     paths:
     - path: /api
       backend:
        serviceName: api-rule-cfg-get
        servicePort: use-annotation
    - path: /api
       backend:
        serviceName: api-rule-cfg-post
        servicePort: use-annotation

Alternative 3 - Similar to Alternative 2 but use CRD instead of annotation:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-foo
annotations:
    kubernetes.io/ingress.class: alb
spec:
  rules:
  - host: www.example.com
    http:
     paths:
     - path: /api
       backend:
        serviceName: api-rule-cfg-get
        servicePort: use-crd
    - path: /api
       backend:
        serviceName: api-rule-cfg-post
        servicePort: use-crd
---
apiVersion: ingress.k8s.io/v1alpha1
kind: RouteConfig
metadata:
  name: api-rule-cfg-get
spec:
   # can also embed other rule level settings.
   verb: GET
   query: header1=a
   port: 443
   backend:
   - serviceName: service1
     servicePort: 80
     weight: 80
   - serviceName: service2
     servicePort: 80
     weight: 20
---
apiVersion: ingress.k8s.io/v1alpha1
kind: RouteConfig
metadata:
  name: api-rule-cfg-post
spec:
   # can also embed other rule level settings.
   verb: POST
   query: header1=b
   port: 443
   backend:
   - serviceName: service1
     servicePort: 80
     weight: 80
   - serviceName: service2
     servicePort: 80
     weight: 20

Alternative 4? If you have preferences 😄

[Rowern]

Alternative 3 is maybe the cleanest but doesn't it means you have to "implement" a new object inside kubernetes ?
If you go this way doesn't it means you will have to maintain 2 ways of specifying routing (one by annotation "actions" and one with objects "RouteConfig").

Alternative 2 is the closest to what this controller provides and might be easier to understand for anyone (maybe even easier to implement?).
Maybe the actions annotation could be re-used by adding some more infos:

{
  "Type": "AdvancedRouting",
  "AdvancedRoutingConfig": {
    "Conditions": {
      "Verbs": ["get", "post"]
    },
    "Backend": {
      "Type": "forward",
      "TargetGroupArn": "arn:123:blabha..."
    }
  }
}

With Alternative 2 my use case might look like this (two rules on the /api route to two target groups):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-foo
annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/actions.api-rule-cfg-get: '{"Type": "AdvancedRouting","AdvancedRoutingConfig": {"Conditions": {"Verbs": ["get"]},"Backend": {"Type": "forward","TargetGroupArn": "arn:123:blabha..."}}}'
    alb.ingress.kubernetes.io/actions.api-rule-cfg-post: '{"Type": "AdvancedRouting","AdvancedRoutingConfig": {"Conditions": {"Verbs": ["post"]},"Backend": {"Type": "forward","TargetGroupArn": "arn:456:blabha..."}}}'
spec:
  rules:
  - host: www.example.com
    http:
     paths:
     - path: /api
       backend:
        serviceName: api-rule-cfg-get
        servicePort: use-annotation
    - path: /api
       backend:
        serviceName: api-rule-cfg-post
        servicePort: use-annotation

[rverma-nikiai]

I would prefer alternative 1 as it seems most ci friendly. Also thinking of a blue-green deployment we can safely add new ingress, do canary analysis and drop the previous ingress. Also, I believe alternative one would require a lesser extension so we can expect faster implementation and lesser bugs.

[VincentCouzij]

Including it in the current actions (Alternative 2) would be a great addition for me. Making separate alb files seems confusing when it is the same alb (Alternative 1).

[kannananandan]

Thanks to @M00nF1sh and @Rowern
Ref: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/916

Second alternative seemed to be the easiest with minimal effort.
Combined the suggestions from both and it worked fine

alb.ingress.kubernetes.io/actions.api-rule-cfg-get: '{"Type": "AdvancedRouting","AdvancedRoutingConfig": {"Conditions": {"Verbs": ["get"]},"Backend": {"serviceName:app1", "servicePort":80}}}'
alb.ingress.kubernetes.io/actions.api-rule-cfg-post: '{"Type": "AdvancedRouting","AdvancedRoutingConfig": {"Conditions": {"Verbs": ["post"]},"Backend": {"serviceName:app2", "servicePort":80}}}'
      - path: /api/*
        backend:
          serviceName: api-rule-cfg-post
          servicePort: use-annotation
      - path: /api/*
        backend:
          serviceName: api-rule-cfg-get
          servicePort: use-annotation

One additional step was to manually add the GET and POST methods as additional rules on the AWS side after deploying the Ingress file

[M00nF1sh]

all three option are easy to implement 😄.
Personally i favor alternative2(can add weight based routing later based on it). And we can evolve to use CRD based solution if it works well 😄 .

[marcust]

Not sure which would be my preference, but as long as I can express that I want OPTION to go to the service and GET,POST to be authenticated and then go to the service I'm fine with either..

[brianstorti]

One thing that is nice about alternative 1 (ingress groups) is that we'd also be able to have more granular authentication rules. For example, I could have the path /foo under an ingress with auth configured, and /bar under another ingress that doesn't require authentication, for the same host. I believe that's not supported right now.

What's not so nice is that as the number of ingresses grow (e.g. if I need a new ingress for each service because I want different source ips conditions), I imagine I'd need to keep all the other config annotations (scheme, subnets, target-type, etc.) in sync manually, as they all refer to the same ALB.

As I believe this use case will already be covered after the Ingress Group Feature is released, the alternative 2 seems easier to maintain.

Here's an example using both the ingress grouping and annotations for advanced routing conditions:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: auth-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: mysvc-group
    alb.ingress.kubernetes.io/auth-type: 'oidc'
    alb.ingress.kubernetes.io/auth-on-unauthenticated-request: 'authenticate'
    alb.ingress.kubernetes.io/auth-scope: 'openid'
    alb.ingress.kubernetes.io/auth-session-cookie: 'AWSELBAuthSessionCookie'
    alb.ingress.kubernetes.io/auth-session-timeout: '604800'
    alb.ingress.kubernetes.io/auth-idp-oidc: '{
        "Issuer": "https://accounts.google.com",
        "AuthorizationEndpoint": "https://accounts.google.com/o/oauth2/v2/auth",
        "TokenEndpoint": "https://www.googleapis.com/oauth2/v4/token",
        "UserInfoEndpoint": "https://www.googleapis.com/oauth2/v3/userinfo",
        "SecretName": "alb-oidc-secret"
      }'

spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /foo/*
            backend:
              serviceName: service1
              servicePort: 80

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: no-auth-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: mysvc-group
    alb.ingress.kubernetes.io/actions.from-authorized-sources: '{
      "conditions":[
        "sourceIps": ["177.40.126.35/32", "177.40.126.36/32"]],
        "backend": [
          { "serviceName": "service1", "servicePort": "80" }
        ]
    }'

spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /bar/*
            backend:
              serviceName: from-authorized-sources
              servicePort: use-annotation

          - path: /*
            backend:
              serviceName: service1
              servicePort: 80

This would configure a single ALB where:
example.com/foo/* would require google auth
example.com/bar/* would only allow requests from the whitelisted source IPs
Any other request to example.com/* would be routed to service1

[AdeOpe]

@M00nF1sh any idea when this is going to land?

[nestoras]

Hello @M00nF1sh , do we have any update with this issue ? If you need help, it would be my pleasure to help you!!

[ashish4github]

Is there any planned timeline for the support of advanced routing? This will be very usefull in our multitenant scenario

[akash0710]

Hi any update on this issue and when it will be implemented?

[mr-karan]

+1 waiting on the same. Any timelines? :D
Thanks

[rverma-nikiai]

I assume this project has been abandoned in favor of aws own app-mesh to use routing using envoy. There are no updates on either on shared ingress nor on advanced routing.

[M00nF1sh]

@rverma-nikiai Hi, it's not true. app-mesh is targeting different area than Ingress. I'll be actively back to contribute to this project soon. (was doing other internal tasks)

[rverma-nikiai]

Good to hear that :)

[clhlc]

+1

[tspycher]

+1

[garyfritz]

I'd also need to have the features mentioned #925, specifically "Host header" and "Path" rule. I am running an alb-ingress-controller image from July it keeps deleting rules I manually added to the ALB listener. It is only keeping the rules it created itself.

[esg-dlelis]

+1

[esg-dlelis]

Is there any way currently to restrict access from a specific IP?
I'm using aws-alb-ingress-controller:v1.1.3

[Vadim-Zenin]

How many IPs?
Could Network ACL handle it?

[chendanshuang]

+1

[gxjluck]

+1

[alFReD-NSH]

Alternative 3 looks like the best, but I can live with alternative 2. If it's significantly easier to implement, alternative 2 sounds better and then we can try alternative 3 in the future.

[gsharm2020]

I'm trying to use path-based routing using ALB ingress as following:
[root@ip-10-0-60-180 test-lb]# cat ingresstestlb.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
namespace: "default"
annotations:
kubernetes.io/ingress.class: "alb"
alb.ingress.kubernetes.io/certificate-arn: xxxxx
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/subnets: subnet-1,subnet-2
alb.ingress.kubernetes.io/security-groups: sg-xxxxx
alb.ingress.kubernetes.io/actions.test123: >
{"Type":"forward","TargetGroupArn": "xxxxx"}
spec:
rules:
- host: abcdef.com
http:
paths:
- path: /abcservices/*
backend:
serviceName: test123
servicePort: use-annotation

I'm not able to browse the service in any browser but I'm getting response from curl.

On the other hand, If I simply configure host without any path configured, I am able to browse the services.

My requirement is to use path based routing.

Please advise what I'm missing here.

Thanks

[chandankashyap19]

One thing that is nice about alternative 1 (ingress groups) is that we'd also be able to have more granular authentication rules. For example, I could have the path /foo under an ingress with auth configured, and /bar under another ingress that doesn't require authentication, for the same host. I believe that's not supported right now.

What's not so nice is that as the number of ingresses grow (e.g. if I need a new ingress for each service because I want different source ips conditions), I imagine I'd need to keep all the other config annotations (scheme, subnets, target-type, etc.) in sync manually, as they all refer to the same ALB.

As I believe this use case will already be covered after the Ingress Group Feature is released, the alternative 2 seems easier to maintain.

Here's an example using both the ingress grouping and annotations for advanced routing conditions:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: auth-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: mysvc-group
    alb.ingress.kubernetes.io/auth-type: 'oidc'
    alb.ingress.kubernetes.io/auth-on-unauthenticated-request: 'authenticate'
    alb.ingress.kubernetes.io/auth-scope: 'openid'
    alb.ingress.kubernetes.io/auth-session-cookie: 'AWSELBAuthSessionCookie'
    alb.ingress.kubernetes.io/auth-session-timeout: '604800'
    alb.ingress.kubernetes.io/auth-idp-oidc: '{
        "Issuer": "https://accounts.google.com",
        "AuthorizationEndpoint": "https://accounts.google.com/o/oauth2/v2/auth",
        "TokenEndpoint": "https://www.googleapis.com/oauth2/v4/token",
        "UserInfoEndpoint": "https://www.googleapis.com/oauth2/v3/userinfo",
        "SecretName": "alb-oidc-secret"
      }'

spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /foo/*
            backend:
              serviceName: service1
              servicePort: 80

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: no-auth-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: mysvc-group
    alb.ingress.kubernetes.io/actions.from-authorized-sources: '{
      "conditions":[
        "sourceIps": ["177.40.126.35/32", "177.40.126.36/32"]],
        "backend": [
          { "serviceName": "service1", "servicePort": "80" }
        ]
    }'

spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /bar/*
            backend:
              serviceName: from-authorized-sources
              servicePort: use-annotation

          - path: /*
            backend:
              serviceName: service1
              servicePort: 80

This would configure a single ALB where:
example.com/foo/* would require google auth
example.com/bar/* would only allow requests from the whitelisted source IPs
Any other request to example.com/* would be routed to service1
@brianstorti
Only concern, It is creating 2 load balancer( auth-ingress & no-auth-ingress) with common target group. Is it possible to have a single ALB with both rule implemented.

Manually, it possible not able to get using Ingress.yaml.

[sebsanchez]

Any update on this issue?

[chandankashyap19]

Issue is closed. But still It is not being resolved. Using ingress.yaml, it is still not possible.