Aws-load-balancer-controller: Feature Request: Denote some ingress rules as externally/manually managed

[gigi-at-zymergen]

We have a routing situation where we have legacy clients expecting to make calls of type subdomain.domain.net/v1/endpoints, and we are actively trying to decompose an old monolith.

Some of those endpoints are now implemented as k8s microservices, and some of them are now implemented as AWS lambdas, as well as continuing to run the old monolith on EC2 instances.

We currently have an ALB where we can manually configure target groups and path based rules to route to the Lambdas and the old ec2 nodes. We also have an ALB that was created by the alb ingress controller that can route traffic to the k8s microservices, but there is no easy way to unify these under one Route53 entry so the clients can continue to call subdomain.domain.net/v1/endpoints and have the proper path based rules applied.

I thought I might be able to hack it by creating an externalName service in k8s, and creating an ingress rule such as :
ExternalName

kind: Service
apiVersion: v1
metadata:
  name: external-name -service
  namespace: development
spec:
  type: ExternalName
  externalName: subdomain-monolith.example.com    

Ingress Rule

  - host: "subdomain.example.com"
    http:
      paths:
      - backend:
          serviceName: k8s-microservice
          servicePort: 80
        path: /v1/k8s-microservice-endpoint/*
  - host: "subdomain.example.com "
    http:
      paths:
      - backend:
          serviceName: external-name -service
        path: /*

But this creates an error on not having defined k8s endpoints.

I thought about creating an explicit endpoints resource and associating a static ip with an NLB and sending traffic to the NLB and then to the lambdas/ec2, but we do SSL termination on the ALB for the Lambdas.

It's starting to look like we'll have to advance our plans to use Envoy/Ambassador/Gloo or something similar as a more complex proxying layer.

However, this would all be easy if there were a way to tell the ALB ingress controller that there are rules and target groups that will be externally managed and not to delete them.

I know this is not currently supported, ie https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/762 . But I wanted to submit a use case where this would come in super handy. Many organizations are in the midst of decomposing much larger legacy apps into microservices and serverless functions and we're probably not the only ones with this situation.

[M00nF1sh]

Hi,
How about extend our existing alb.ingress.kubernetes.io/actions.$action-name annotation

So for your case, you can manually create targetGroup for your backend, let's assume its ARN is legacy-tg-arn.
And then make your ingress looks like below:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: testcase
  name: echoserver
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/actions.legacy-app: '{"Type": "forward", "TargetGroupArn": "legacy-tg-arn"}'
spec:
  rules:
    - http:
        paths:
          - path: /v1/endpoints
            backend:
              serviceName: legacy-app
              servicePort: use-annotation
          - path: /normal-path
            backend:
              serviceName: echoserver
              servicePort: 80

Should be easy to implement

[gigi-at-zymergen]

Thanks for the rapid response! Unfortunately, I'm a Go noob, and still figuring out the source. Just want to confirm my understanding, and probably don't have the Go chops to whip up a PR, but maybe I'll summon the courage for an attempt.

I think the above is your proposal for how to implement this, and want to confirm that it doesn't yet exist. From looking at https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/d0f2a3494859ced79872954b464307b26daefac9/internal/ingress/annotations/action/main.go#L56 it looks like the only currently supported actions types are fixed-response and redirect.

Your suggestion sounds great, and I just wanted to clarify that the work to implement would be to add a new action type : forward with a config of TargetGroupArn. This would assume the specified target group was previously manually created, fail if it didn't exist (?), and add the specified rule to the listener to forward to that target group.

[M00nF1sh]

Yes,
The only needed code change is adding an case "forward", and check TargetGroupArn is not empty 😄 (other stuffs don't need changes). And adding an test case for it.

In the future(or this PR), we'd better add more documentations(e.g. tasks/migrate_legacy_app) on how to use it.

Feel free to try it as your first go commit, and i can help to do that if you prefer 😃

[gigi-at-zymergen]

I have a local implementation with a new forward case on the switch, nil check on TargetGroupArn, an update to the dummy ingress and test case to add a forward annotation, and a new migrate_legacy_apps.md in docs/tasks .

I am in the process of getting my company to sign the contributors agreement and then will follow the Kubernetes contributor guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/README.md to submit a PR. Let me know if there are any github workflow guidelines specific to this sig.

[M00nF1sh]

@gigi-at-zymergen
Thanks so much for contributing this 👍
The general kubernetes controbutor guidelines applies.
You can use make lint and make test to lint & test locally, and make docs-serve to preview the docs website

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[tdmalone]

/remove-lifecycle rotten

[rshutt]

So just to be clear, there should be no expectation that ExternalName services will work in IP mode? They definitely are not working in Instance mode since there are no endpoints right?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tdmalone]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.