Containers-roadmap: [ECS] [request]: Daemon service scheduler not working as expected

[henriquesantanati]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

For Daemon services, ignore the state of others tasks/container instances in order to launch a new Daemon task on a new container instance.

Which service(s) is this request for?

ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Working with Daemon tasks, when some container instance is with the status as DRAINING, the new container instances launched is not starting the task until all the DRAINING process finish. But, Daemon tasks should work independently according with this documentation page:

"The daemon service scheduler does not place any tasks on instances that have the DRAINING status. If a container instance transitions to DRAINING, the daemon tasks on it are stopped. It also monitors when new container instances are added to your cluster and adds the daemon tasks to them" - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html#service_scheduler_daemon

That said, for tasks that take a time for finish the DRAINING process, the new container instances take the same amount of time of the DRAINING process to launch a new task.

Are you currently working around this issue?

The workaround depends on the application load and the DRAINING time. Reduce this time is not always possible.

Additional context

It seems that the Daemon service scheduler is working like the replica type logic. It should have the own logic and work independently of the other tasks/container instances.

Also, in order to simulate this, you will need to have a tasks that take a while to be terminated to see the issue.

[pavneeta]

Hi @henriquesantanati thank you for the feedback on this issue . Just so that I know I understand this correctly, the problem that you are facing is that ECS is not scheduling Daemon task on the replacement instance on the cluster until the previous instance is fully DRAINED? Could you please tell us more about your ECS stack - are you using capacity providers with your Daemon Service ? What should be the desired behavior for the Daemon Scheduler ?

[henriquesantanati]

HI @pavneeta , thanks for your reply!

the problem that you are facing is that ECS is not scheduling Daemon task on the replacement instance on the cluster until the previous instance is fully DRAINED? - Yes. Only after the previous instances got DRAINED that the new one receive the task to run.

are you using capacity providers with your Daemon Service ? - No.

What should be the desired behavior for the Daemon Scheduler ? - The Daemon task should be placed into the instances without any dependency between other instances, including the DRAINING status.

Please, let me know if you have more questions regarding this issue.

[pavneeta]

Hey @henriquesantanati we have shipped a fix for this issue across all AWS commercial regions. This removes the dependency between instances for Daemon task replacement.