Containers-roadmap: [ECS,Fargate] Multiple target groups for a service

[zaitsman]

Summary

I want to be able to route traffic from two different Path rules in my ALB to the same ECS instance and have ECS automatically register targets in both groups.

Description

Pretty much what the summary says. I can manually add targets to the second group, but if the service is restarted, then the second group loses the targets

Expected Behavior

Unable to have 2 targets for the same ECS service

Observed Behavior

N/A

Environment Details

Supporting Log Snippets

[dastbe]

hey @zaitsman, ECS service team member here.

We don't currently support multiple target groups per service today. However, if you're able to use the same port on the same ALB you can re-use the same target group for multiple rules on a single loadbalancer.

[alexzaytsev-newsroomly]

@dastbe I do use that, actually, but my use case is that i want the same container to respond to two different TG's for different ALB's. I understand that is a feature request.
One way to solve it would be if we could get a Cloudwatch alarm each time ECS registers something, we could then lambda those additional registrations.

[adnxn]

One way to solve it would be if we could get a Cloudwatch alarm each time ECS registers something

@alexzaytsev-newsroomly, have you looked at the ECS event stream? There are container instance registration/deregistration that are surfaced through CloudWatch events. Let us know if this captures your specific use case.

[hridyeshpant]

@dastbe does we have Multiple target groups for ECS service support in Fargate mode?

[bataras]

Is multiple target groups per ECS service feature going to happen? On the roadmap?

[DenisBY]

I need this feature too. Our application inside a container listens on 3 different ports for different purposes but I'm able to (automatically) register container only in one target group

[Raizex]

We could also really use this as well. We've reached the maximum number of certificates per load balancer and need a way to add the service to additional load balancers without creating a duplicate service.

[sergedomk]

I could use this as well. We would like to put up a single Fargate ECS service with several containers each providing a different branch of the code running on a different port for our product and QA teams to use.

[mcmar]

This is a blocking issue for me.
I have one primary service and one secondary service. The primary receives traffic for /management route. The primary and secondary both receive traffic for all other routes *.
This pattern is impossible to implement without the ability to register 2 target groups for the primary instance.

This is the pattern currently used by Prisma as described here:
https://github.com/prismagraphql/prisma/issues/2850#issuecomment-409868271

[zymtx5g79k]

I solved an issue by pointing current instances in stack, but it will not works with auto scaling. The feature will be useful.

[mcmar]

@zymtx5g79k Can you clarify what you mean by pointing current instances in stack? I'd love to get a solution to this and I'm not currently using autoscaling. Does your solution work with ECS on Fargate?

[zymtx5g79k]

@mcmar, hello:

EC2 -> Target groups -> Create target group -> Target type (instance) -> Create.
Select just created target group -> Actions -> Register and deregister targets -> Check all Instances that work in your stack -> Add to registered -> Save.

After some time the target group will detect which an instance has your service and will mark it as healthy. Load balancer will redirect traffic on the healthy instance.

_In my case: it slanger service. Port 4567 was assigned in ECS service and port 8080 (websocket) is listen from all instances._

Tip: Target group need some time to correct healthy status after ECS service migration across EC2 instances. You can get connections refused until new healthy instances was found.

_In my case: websocket connections will be repeated. It is regular case for frontend. I use EC2 stack "because I can"! (:_

[aventurella]

I could use this too.

In my use case, we have one legacy application running behind an Application load balancer but rather than rebuild the whole application from scratch, we can go down the route of rebuilding pieces and just forwarding from the first Application Load Balancer to a new target group (listener rules). Unfortunately when using Fargate/ECS it owns the registrations for the Target Group and a target group can only be associated with 1 Load Balancer (which Fargate/ECS owns). So we are left with trying to figure out a way to mirror that Target Group when we get new ECS Registrations.

[m0n5t3r]

another use case would be something that exposes multiple ports (like, say, rabbitmq's 5672 (AMQP) + 1567 (management UI)); since network load balancers seem to force traffic to the target group port instead of keeping the listener port, one needs to have a target group for each port, but only one of them ever gets updated by ECS

[hajdukd]

This is a blocker - we require to have an option to register more than single port for our services.

[aventurella]

@hajdukd I hate to see you blocked by this. Per one of the comments above:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html

Using ECS Event Streams via CloudWatch events and triggering a Lambda definitely is a solution that works. It’s not as convenient if they offered it automatically, but you can add and remove from as many target groups as you want. So for a small bit of extra work (1 Lambda function and some event wiring) you will be unblocked.

[renuez]

Nathan Peck from ECS product team has recommended a few very useful options: https://gist.github.com/nathanpeck/a8537e21627b3a0f3e735eecbe6f6384

[coultn]

Thanks everyone for the feedback on this issue. I wanted to let you know that we on the ECS team are aware of it, and are actively considering solutions. +1's and additional details on use cases are always appreciated and will help inform our work moving forward.

[cnelson]

+1

[davidfti]

+1

[outofcoffee]

@coultn we are trying to model multiple ingress routes to a shared service. In particular, some traffic comes into a load balancer in a private subnet with one set of path routing rules, whilst other traffic comes via a load balancer in a public subnet with a different set of path rules.

Using separate security groups to logically represent the rulesets for the different ingress routes can be somewhat helpful here, but with the default max of 5 SGs per load balancer this can be a squeeze. It also doesn't provide the routing benefits of separate load balancers in public/private subnets.

Isolation at the task level is not required, and it would be inefficient to run multiple service instances, each with multiple task instances for redundancy, simply for the purposes of attaching multiple load balancers.

Essentially the pattern we're looking to emulate is how Pods can have multiple Services pointing to them in Kubernetes. This is one of the main drivers pushing us away from ECS.

[dodwmd]

+1

[senthilkumar-s]

+1

[sampsasaarela]

+1

[broglep-work]

We did a PoC with Cloud Watch Events & Lambda function. You can find the CF stack at https://gist.github.com/broglep-koubachi/0ceed52c509f75a6c26afc43b807a0eb
(it uses an ecs service and its load balancer config as basis to create targets for the twin target group)

[tkgregory]

+1 My use case is a Gitlab task that needs to expose ports 443 and 22. I'm currently using an ALB for the HTTPS, and would need another load balancer and target group for the SSH.

[sandeepboyapati]

We did a PoC with Cloud Watch Events & Lambda function. You can find the CF stack at https://gist.github.com/broglep-koubachi/0ceed52c509f75a6c26afc43b807a0eb
(it uses an ecs service and its load balancer config as basis to create targets for the twin target group)

This works perfectly.....Thanks @broglep-koubachi

[kokuryuha]

+1 as a feature request as well. Currently looking at deployments and realized this was going to be an issue immediately.

[joemoore13]

+1 the case of a single application listening on multiple ports (web and tcp) has multiple cases for many technologies and this makes working with them in ECS more difficult.

[abby-fuller]

moving this over to the containers roadmap since this is an ECS feature request, rather than an agent issue.

[compscigirl]

+1

Is there an estimate when this might be available?

[polarizeme]

Just wanted to chime in with a +1 on this request. HItting this as a roadblock at the moment, which is a huge bummer/frustration.

Usecase is the same as @outofcoffee :

we are trying to model multiple ingress routes to a shared service. In particular, some traffic comes into a load balancer in a private subnet with one set of path routing rules, whilst other traffic comes via a load balancer in a public subnet with a different set of path rules.

[v-g-ustinov]

+1

[jbouse]

I have a use case where I have an ECS TaskDefinition that is launching 2 containers. The first container has 2 ports exposed, one is TCP passthrough and the other can either be HTTPS (ELB) or TLS (NLB) but they point to the same container. The second container exposes a different port that is TCP passthrough as well. The second container is new and only compounds the issue the first one exposed. With just the first container I used a Classic ELB with the two listeners and it works but I have to use static HostPort values for the container PortMappings. This causes an issue when updating the service as the ECS instances can only have 1 container listening on the static host port. If I could use a TargetGroup and NLB then I could use dynamic HostPort mapping and not run into this issue but I can't do this if I can only add 1 entry under LoadBalancers for the ECS Service. It would be extremely helpful if I could provide the ECS Service configuration something along the lines of:

      LoadBalancers:
        - TargetGroupArn: !Ref Service1TG
          ContainerPort: !Ref Container1Port1
          ContainerName: !Ref Container1Name
        - TargetGroupArn: !Ref Service2TG
          ContainerPort: !Ref Container1Port2
          ContainerName: !Ref Container1Name
        - TargetGroupArn: !Ref Service3TG
          ContainerPort: !Ref Container2Port1
          ContainerName: !Ref Container2Name

[jasonnerothin]

Just trying to flip on SSL/TLS: +1

[sanglt]

We did a PoC with Cloud Watch Events & Lambda function. You can find the CF stack at https://gist.github.com/broglep-koubachi/0ceed52c509f75a6c26afc43b807a0eb
(it uses an ecs service and its load balancer config as basis to create targets for the twin target group)

Thanks, It's work perfect for us.

[bbonnefon]

+1
We have micro services, some public and others private, but they should talk together.
It would be very helpfull that public micro services are available on 2 different ALB (public access and private access).

[rtjarvis]

I've encounted this problem today too. Come on AWS - the CFT even supports a list!

[sstelfox]

I'd also like to see this, though for the NLB not the ALB. Our use case is that we are running conditional client certificate authentication on our system, so TLS listeners won't work. This requires handling both HTTP and HTTPS at our service which isn't currently possible without the use of an adapted form of the lambda mentioned earlier in this thread.

[markusrydh]

Yes, we are also needing this. Will try the workarounds.

[dnsymphony]

This is a common requirement for a Spring Boot-based microservice using Actuator, where you want the Actuator endpoints running on a different port available only internally with elevated security and the other endpoints available publicly.

[matandavid]

👍

[talolard]

+1

[angusfz]

Any update after 1 year ?

[polarizeme]

@angusfz added to their kanban board in Nov, no one's shown as assigned yet =/

[derekbedwards]

Just hit this as well trying to host a service that exposes multiple ports. Didn't expect there to be such a fundamental limitation in ECS :(

[apoorvkapil]

I was working on Kong APIM deployment on ECS [Fargate] (https://github.com/Kong/docker-kong/issues/242) and stuck on this. This should be taken as bug rather than a feature as it's blocking many AWS clients moving forward with fargate.
The assumption of exposing a single port from container is really unconvincing.

[krisread]

This is a blocker to us as we have docker containers exposing 2 ports and cannot split this into 2 containers. There seems to be no way for us to get this deployed with terraform.

[coultn]

This is the same issue as https://github.com/aws/containers-roadmap/issues/12

[bkali]

+1 this is a critical roadblock for us, preventing us from using Fargate/ECS. We need service processes (like https://nats.io/ ) that use different ports within the same server process. @coultn , any thoughts on when this feature may be released ? Thank you!

[Zebirdman]

+1 our omnidb service needs dual port exposure. Really surprised this wasnt implemented from the start given that multiple port mappings are available out the box for other orchestrators.

[anshul0915]

+1

[bhkangw]

+1 would love to see progress on this as it's also preventing us from taking advantage of Fargate.

[jk2l]

+1 Hi, in the last 1 year we encountered at least 4 customers that want to use ECS but their containers all require multiple port.

Most common reason is my clients are doing mass migration with lift and shift to move from on-premise VMware into ECS. However this "single port" cause a big issue and the clients are unable to take advantage of it. And unfortunately as AWS advanced partner we would like to promote AWS service instead of adopt 3rd party solution, but to our clients ECS is more like a incomplete solution and it cannot be used at all

[alexandersm]

+1 , our services are listening more than one port and it would be really great to have this feature available in ECS

[vimmis]

+1

[franz-newsadoo]

+1

[herrri]

+1

[coccyx]

+1

[AndreyChugunkovDevPro]

+1

[whereisaaron]

Please show some common courtesy people ✋. No one, including AWS, is going to respect you for posting a '+1' message on Github issue. Anyone but a Github n00b knows you vote-up the original post with a 👍, also known as a '+1' reaction emoji. to show your support. Go ahead and also 👍 any comments above with suggestions or use cases you support, or ❤️ it even! Reaction emoji are available on both the web and mobile clients, so no excuses.

On the other hand, if you have a use case or application that would benefit from this requested feature, or a suggestion on how to implement the change. Do post a comment and share your story.

But please don't waste the time of 50+ people subscribed here by sending everyone a notification with nothing by a '+1' 'me too' post. Instead, go up the top and hit that 👍. I appreciate your support for this issue.

Like and subscribe! 👍 😄

[ashayshub]

@coultn Is this going to be released only for Fargate? That will be a shame really cause I bet most of the users using it are asking it for EC2 as well!

[coultn]

@coultn Is this going to be released only for Fargate? That will be a shame really cause I bet most of the users using it are asking it for EC2 as well!

When this feature is launched it will support ECS on EC2 and ECS on Fargate (commonly known as just 'Fargate').

[mbotmcc]

@coultn will this get released with cloudformation support?

[ledbit]

+1

[coultn]

@coultn will this get released with cloudformation support?

Yes, that is the plan.

[akshayram-wolverine]

Hi everyone! I am on the product management team on AWS Container Services. This feature is now in developer preview.

If you are interested in participating please see here: https://github.com/aws/containers-roadmap/tree/master/preview-programs/ecs-multipletg-service In the interest of your time, the link has examples to set up an ECS service with multiple target groups in both the EC2 and Fargate launch type. Please email me at akshram [at] amazon.com with your AWS account ID(s) and AWS region(s) to get started.

We would love to hear feedback!

[whereisaaron]

Thanks for the update @akshayram-wolverine, I read the examples and both the use cases there focus on using multiple ALBs for one ECS/Fargate service. The misses one common use case, i.e. imagine you have an ECS/Fargate service expose ports 80 and 443, and you want to expose both ports with one ALB or ELBv2. Will this new feature be able to support that? Can we have port 80 and 443 on the same Load Balancer going to the same ECS/Fargate service?

[akshayram-wolverine]

Hey @whereisaaron. Yes your use case should be supported as well! You should just create two target groups one for each port and attach it to your ECS service.

[anshul0915]

Thanks for the update @akshayram-wolverine . can it support the use like if
have one container running in service can we link it to 2 target groups
which will help us to resolve issue to host our application on internal ELB
and external ELB?

On Tue, Jul 9, 2019 at 10:18 AM akshayram-wolverine <
[email protected]> wrote:

Hey @whereisaaron https://github.com/whereisaaron. Yes your use case
should be supported as well! You should just create two target groups one
for each port and attach it to your ECS service.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/containers-roadmap/issues/104?email_source=notifications&email_token=AB4BMPJLGHPELHZ5RI57GC3P6QKBLA5CNFSM4GPJVEQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZPCGUY#issuecomment-509485907,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB4BMPNIGHZRJU5XM4WPZG3P6QKBLANCNFSM4GPJVEQQ
.

--
Regard,
Anshul Agarwal
Mob No:[phone]

[akshayram-wolverine]

Hey @anshul0915 Yes!

[pitkley]

Hi @akshayram-wolverine. These are great news, thank you for implementing this feature-request!

One question I have is whether or not target-groups of instance-type ip are supported with this change? Right now assigning a network load balancer target-group whose instance-type is set to ip, rather than the default instance, fails with a message noting that only instance-mode is supported.

Our use-case is the ability to run TCP-based services within an ECS-cluster, making these services available through a network load balancer, and have those services be reachable _by other services within the same ECS node!_ The last point is crucial: if the request is originating from the same instance the NLB tries to route to, i.e. the source instance-ID and destination instance-ID are identical, the request will not be able to propagate through the NLB (I can't find the documentation for this right now, unfortunately).

Should this be a concious decision or technical limitation, is there a "workaround" or alternative solution to this problem? (I.e. should one use the awsvpc networking in this case?)

(This question might be hijacking this issue, but I was unable to find a more fitting one. Please let me know if I should open a new issue for this!)

[whereisaaron]

@pitkley: Loopback timeout

[pitkley]

@whereisaaron thank you, that is what I was unable to find. If I understand the points made by the documentation correctly, running the containers with the awsvpc networking-mode should fix my issue. I'll test this as soon as I get around to it and report back.

[pmoghaddam]

Will there be a way to distinguish which ALB the request is coming from?

In the case of internal vs. external, if you cannot tell the difference, it'll be challenging to effectively use this.

[coultn]

Launched! https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-ecs-services-now-support-multiple-load-balancer-target-groups/

[GoeranEnvirio]

Is it possilbe to configure a current fargate service this way:
aws ecs update-service --service my-http-service --cli-input-json loadbalancers.json

where I have put the loadbalancers like this is in the loadbalancers.json file:

"loadBalancers":[
{
"targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_1/1234567890123456",
"containerName":"jenkins",
"containerPort":8080
},
{
"targetGroupArn":"arn:aws:elasticloadbalancing:region:123456789012:targetgroup/target_group_name_2/6543210987654321",
"containerName":"jenkins",
"containerPort":50000
}
]

[GoeranEnvirio]

It did not work got:

Error parsing parameter 'cli-input-json': Invalid JSON: No JSON object could be decoded
JSON received: loadbalancers.json

[Istador]

Is it possilbe to configure a current fargate service this way:
aws ecs update-service --service my-http-service --cli-input-json loadbalancers.json

where I have put the loadbalancers like this is in the loadbalancers.json file:

[...]

If I remember correctly, the loadBalancers settings can't be changed once the service is created. Neither with the aws ecs update-service command nor with the aws console.

You should be able to create a new service with aws ecs create-service that is using multiple loadBalancers, and then delete the old one.

It did not work got:

Error parsing parameter 'cli-input-json': Invalid JSON: No JSON object could be decoded
JSON received: loadbalancers.json

The example in the documentation of create-service is using the following syntax with file://:

--cli-input-json file://ecs-simple-service-elb.json

---

BTW another use case that is working thanks to this feature:
Having two target groups for the same ecs service, with one target group being sticky and the other being non-sticky. The public load balancer decides which paths needs to be sticky or not, by forwarding to the designated target group.

[GoeranEnvirio]

@Istador thanks for the info. It's really irritating when it's almost undocumented and you need the feature.

[GoeranEnvirio]

I tried the following:
aws ecs create-service \
--cluster TestDevEC2Cluster \
--service-name ecs-multiport-service-nlb \
--launch-type EC2 \
--cli-input-json file://ecs-multiport-service-nlb.json

An error occurred (InvalidParameterException) when calling the CreateService operation: Not support Classic Load Balancer in multiple load balancers.

I have configured an NLB! So what could the issue be?

If I add a role: "AWSServiceRoleForECS"

It will complain: An error occurred (InvalidParameterException) when calling the CreateService operation: You cannot specify an IAM role for services that require a service linked role.

The file:

{
"serviceName": "ecs-multiport-service-nlb",
"taskDefinition": "test-ec2task:2",
"loadBalancers": [
{
"loadBalancerName": "NewTestNLB",
"containerName": "test",
"containerPort": 80,
"targetGroupArn": "xxxxx"
},
{
"loadBalancerName": "NewTestNLB",
"containerName": "test",
"containerPort": 2022,
"targetGroupArn": "xxxxx"
}
],
"desiredCount": 1
}

So how can I create it without an error?

SUCCEDED!

There was a lot of rabbit holes and hidden ramifications. Probably the reason the examples do not work. The only way to fix it is to read a lot.

[quolpr]

@GoeranEnvirio the issue is that you specified loadBalancerName in loadBalancers. Only targetGroupArn, containerName, containerPort should be specified

The only way to fix it is to read a lot.

But the better way - describe how did you fix your issue, so that will save a lot of time of other guys ;)

[stnor]

I have a use case where I would need a service to use two target groups. One is an NLB which forwards requests (SSL client cert) to one port on the service, and another is used on an ALB.

I'm unable to configure this in the web console. and according to support (Case ID - 6524068251), this is not supported. Please consider this use case in the future, thanks. As it is right now, I need to run two services that does the same thing.

Thanks!

[wedneyyuri]

@stnor it's really strange because your usecase is already supported but you will need to recreate your ECS service.

[stnor]

@wedneyyuri According to AWS Support, a mix of NLB and ALB target groups isn't supported.

[kstro21]

According to the documentation here

The service must use either an Application Load Balancer or Network Load Balancer.

We can't use both for the same service. I would benefit from being able to use both. Please, consider this use case.