Amazon-ecs-agent: Blocker: Stopped containers do not get deregistered from target groups -> serious routing issues

[fabfuel]

If a container (with a dynamic port) stops, the task is correctly recognized as STOPPED (Essential container in task exited), but the container does not get deregistered from the ELB target group.

To reproduce it, just stop a container via docker stop on the ECS instance or just press the task's Stop button in the Management console.

[fabfuel]

Yesterday this turned out to be a serious issue.
It's a show stopper if you use dynamic ports:

I had to restart docker and the ECS agent after installing a monitoring client on a new ECS instance.
Due to the docker restart, all containers were stopped.
The ECS agent then started new tasks on the node. So far so good, BUT:

Docker (or the ECS agent) always seems to start with the same dynamic port and it is then incrementally increasing the dynamic port number for every additional container.

That led to reusing a port which was already in use before the restart.

And because the stopped containers are not removed from a target group, a previously registered target became healthy again. But the problem was: Another application was now using this port!

It also led to the same instance and port being registered in two different target groups.

And as a result, the Application ELB was sending traffic to Application B, although it was meant to be routed to Application A.

I will also contact business support regarding this issue.

Best
Fabian

[samuelkarp]

@fabfuel Thank you for reporting this. We have identified the cause and are working on a fix. We will update this issue with more information once it becomes available.

[joelgerard]

@samuelkarp Any update on this? This is a bug with extremely high severity.

[srv-stack]

+1 Since the unhealthy containers are not unregistered from Application load balancer we've reached the limit of "Number of times a target can be registered per load balancer: 100"

[aaithal]

Hi All,

We have recently deployed a fix that should fix this issue. Please reach out to us if you have any further questions/comments on this. Closing the issue for now.

Thanks,
Anirudh

[shuangwu42]

We have observed the same issue with the latest version 1.14.1. Are you sure it was fixed? Thank you!

[samuelkarp]

@shuangwu42 Please open a new issue if you're experiencing problems.

[kwent]

Same issue with the newly health check introduced on the task definition.

If the health check at the container level (through task definition) fail but the health check at the ELB level is succeeded. Ecs stop the task, target is marked are unhealthy at the ELB level but never unregistered.

[ryanpagel]

Any update on this issue? We're also experiencing the same thing using ECS Fargate 1.2.0. We currently have an accumulation of unregistered targets in a target group.