Aws-load-balancer-controller: HTTPS only rules

[JakubJecminek]

Hello,
would it be possible to specify that certain rules should be crated only for https listener? Currently all rules are propagated to both http and https listener. And since ingress controller is creating rules in order in witch they are defined in ingress yaml object, you can potentially make mistake by specifying rule some service before actual rule for http -> https redirect:

  "spec": {
    "rules": [
      {
        "host": "test.mydomain.com",
        "http": {
          "paths": [
            {
              "path": "/my-app/*",
              "backend": {
                "serviceName": "my-app",
                "servicePort": 80
              }
            },
            {
        "http": {
          "paths": [
            {
              "path": "/*",
              "backend": {
                "serviceName": "redirect",
                "servicePort": "use-annotation"
              }
            }
          ]
        }
      },
          ]
        }
      }]}

Thus alb rule for redirect will be created as second and my-app can be accessed both through http and https.

Optimal situation would be that if redirect rule is present in configuration then all other rules are created only in https listener. Or there could be some new annotation that would specify if rule is supposed to be applied to http/https listener or both.

With best regards
Jakub

[hluchej]

+1

One problem is the wrong order of redirect rule which is definitely a security risk and error-prone.

The other problem is "a review". A security guy looks at ALB configuration and HTTP listener rules which are supposed to do just redirect. Instead, it shows a ton of rules which are meant only for HTTPS listener - that triggers some worry there.

Feels like the ingress controller does more "magic" than would be necessary. The AWS Console UX considers the listeners as separate while the ingress updates them "as one".

[M00nF1sh]

Hi, the ingress group i'm working on can solve this problem like below. For such cases, you can create two seperate ingress like below:

kind: Ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: my-app
    alb.ingress.kubernetes.io/listen-ports: 'HTTPS:443'
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
    - http:
        paths:
          - path: /https-only
            backend:
              serviceName: echoserver
              servicePort: 443
---
kind: Ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: my-app
     alb.ingress.kubernetes.io/listen-ports: 'HTTP:80'
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
    - http:
        paths:
          - path: /http-only
            backend:
              serviceName: echoserver
              servicePort: 80

The above two ingress will result in a single ALB since they belong to same group: my-app. However, settings on different ingress will still impact rules, so that rules specified in first ingress will be https-only and rules specified in second ingress will be http only.

Would this user experience be fine for such problem? or is there better solutions? It's still under implementation and i'm happy to adjust/change based on early feedback 😄

[hluchej]

Yep, I think that would cover the use case well. Cannot judge if there are other better solutions.

One could add the listener identification into the spec: but that is a breaking change and I do not see a valid justification for that. I like the idea that one can define multiple ingresses running on one ALB - so new services could basically add their rules. So the spec is not centralized.

I guess with multiple ingress definitions for one ALB listener, one could run into an issue of incorrectly ordered rules. Not sure if that needs to be solved in implementation, maybe mentioning such risk in documentation is sufficient so users do not expect a specific order when there are multiple ingresses defined for the same group on the same listener.

[M00nF1sh]

@hluchej
Thanks for the feedback 😄 .
There will be an another annotation like alb.ingress.kubernetes.io/group.priority: 1 to configure rule orders across ingresses, By default, the rule order is determined by the alphabet order of ingresses. Most users won't need to configure such annotation as long as their order related rule are specified within same ingress.

For the listener part, we'll create CRD for alb/listeners/targetGroups. The main ingress controller is just create/update these CRDs 😄 . But i'd rather treat it as implementation details rather than open it for public usages

[plrunner]

Sounds like a very cool feature. We exceeded the max rules limit due to duplication in the HTTP listener, so we are itching to get it available.
Is it in your plan to put it in the next release? Am asking as this issue is not assigned to anyone.
Thank you for this amazing ingress controller

[EricDube]

@M00nF1sh Has there been any success with your group implementation? We just hit this issue and your idea seems to be the perfect answer for our use case.

[M00nF1sh]

@EricDube Hi, there is an draft version availble here: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/914, i'll create an PR to merge it once i got some time

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[tdmalone]

/remove-lifecycle rotten

Because we're close to hitting ALB's 100 rule limits and would love to avoid the duplication between HTTP and HTTPS rules, which currently makes that limit effectively 50 :)

[tomfotherby]

We have reached the ALB 100 rule limit. Currently we have 50 rules, duplicated twice because we use:

alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80, "HTTPS": 443}]'

It would be ideal for us if the HTTP rule could simply redirect to HTTPS so we could use the remaining 99 rules in the HTTPS listener.

[s2504s]

We resolved this issue using this annotation as well:
https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/guide/tasks/ssl_redirect.md
@tomfotherby

[tomfotherby]

@s2504s Thank you for the resource link. I tried it out and although it did setup a redirect on port 80, the rules were still duplicated to both port 80 and 443. Ideally I'd like just a single rule on port 80, the redirect.

(EKS. Kubernetes v1.14.9, Ingress apiVersion: networking.k8s.io/v1beta1, Ingress controller version: docker.io/amazon/aws-alb-ingress-controller:v1.1.5)

[tomfotherby]

I had a stab at skipping unneeded ALB rules that follow a unconditional redirect. Testing so far looks good on our project - it's a big relief to free up 48 rules! I raised a PR if anyone is interested in providing some feedback or raising concerns: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/1162

[tomfotherby]

At https://peopleperhour.com we have been running a ALB Ingress controller customised to skip unneeded rules (PR https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/1162) in Production with success. Our ALB serves 110 requests per second and has 50 ALB rules (it had used up all 100 before the customisation).

If you want to try it, first add the ssl-redirect annotation as the first rule as documented. Then, in your alb-ingress-controller.yaml file, replace:

        image: docker.io/amazon/aws-alb-ingress-controller:v1.1.5

with:

        image: docker.io/peopleperhour/aws-alb-ingress-controller:v1.1.5_2f6bdea9

and kubectl apply the new manifest.

_Disclaimer_: I am not an experienced GoLang coder. Here be Dragons :dragon: . You would be safer to wait until https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/1162 has been reviewed by the professionals.

---

• Docker image: https://hub.docker.com/r/peopleperhour/aws-alb-ingress-controller
• Sourcecode changes: GitHub: https://github.com/PeoplePerHour/aws-alb-ingress-controller - Customised version to skip unneeded ALB rules

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[tomfotherby]

/remove-lifecycle rotten

I fully mean to complete this one day

[crawforde]

Hi, the ingress group i'm working on can solve this problem like below. For such cases, you can create two seperate ingress like below:

kind: Ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: my-app
    alb.ingress.kubernetes.io/listen-ports: 'HTTPS:443'
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
    - http:
        paths:
          - path: /https-only
            backend:
              serviceName: echoserver
              servicePort: 443
---
kind: Ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group: my-app
     alb.ingress.kubernetes.io/listen-ports: 'HTTP:80'
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
    - http:
        paths:
          - path: /http-only
            backend:
              serviceName: echoserver
              servicePort: 80

The above two ingress will result in a single ALB since they belong to same group: my-app. However, settings on different ingress will still impact rules, so that rules specified in first ingress will be https-only and rules specified in second ingress will be http only.

Would this user experience be fine for such problem? or is there better solutions? It's still under implementation and i'm happy to adjust/change based on early feedback 😄

This solution is not working for me. One LB is entirely ignored. I'm also wondering how you're able to get away with not defining a resource name. I get an error using the code that you supplied.

[tdmalone]

Hey all,

Here's a really quick-and-dodgy patch which should enable you to utilise right up to the ALB 100-rule limit by skipping _any_ rule defined for an HTTP listener after the first one:
https://github.com/tdmalone/aws-alb-ingress-controller/commit/c5bee00f6a26e89918dfcf5100c09d1ea381323b

This obviously makes some assumptions and is not going to be suitable for every circumstance, but in initial testing it works quite fine for our use case.

(The above commit is available in a Docker Hub build at tdmalone/aws-alb-ingress-controller:skip-http-rules if you want to try it out, but again, use this at your own risk).

[M00nF1sh]

@crawforde
That was only working for the v1.2.0-alpha1. Now we have a new GA release v2.0.0 which supports the same.

@tdmalone That patch have been available for both v1.1.9 and v2.0.0.
The behavior of v2.0.0 is

1. If you have any redirect rule which hides rules with lower priority, these hidden rules will be omitted.

closing this issue, as it can be either achieved via IngressGroup or a redirect action annotation.