Containers-roadmap: [ECS] [request]: Ability to update task-placement constraints and strategy of a service that has already been created

[mhumeSF]

Tell us about your request
What do you want us to build?
Ability to update task-placement constraints and strategy of a service that has already been created

Which service(s) is this request for?
ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I have a beautifully running service in production. I notice I've made a typo in the task placement constraint. To correct this and avoid downtime, I have to create a new temporary service with a different name and then delete the old service. Deploy the original service with the original service name with the corrected typo to keep the original service name.

Are you currently working around this issue?
How are you currently solving this problem?
:point_up_2: See the bit above

[CpuID]

Right now we actually delete the service entirely and recreate (in a programmatic fashion), causes an outage of about 30-60sec when we do it. Luckily it's rare, but this is about the only situation that we require service downtime for anymore.

[pavneeta]

@CpuID @mhumeSF Thanks for the report, appreciate the feedback. We wanted to understand the different needs to update/change placement strategies or constraints on existing services.

• What are the main reasons to update placement strategies(PS) and constraints(PC)?
• How often do you need to change the PS and PC?

[CpuID]

@pavneeta see below:

• What are the main reasons to update placement strategies(PS) and constraints(PC)?

Some examples:

• let's say you opt to not use a distinctInstance task placement constraint when deploying initially. a service takes off, increases in desired count and load over time, then you realise you are hitting some kind of load related wall due to running more than 1 instance on a single server. to attempt to add distinctInstance would require a downtime hit, or creating a new service and somehow cutting traffic over to that service

• you deploy a service with a task placement strategy of binpack -> memory, but find out down the track that binpack -> cpu is a better fit with your EC2 provisioned resources. this isn't always clear cut on initial service deploy, but becomes clearer over time as you have data to back this up, and/or if you find you are hitting cluster size "walls" where you have lots of cpu free but very little memory free, and want to resize the task definitions to "fit better".

• How often do you need to change the PS and PC?

Not usually every deploy, but occasionally the need arises and it would be ideal to do it without downtime (currently the only operations I know of requiring service downtime as it stands).

[pavneeta]

@CpuID Thank you for sharing additional feedback. I wanted to double click on __"if you find you are hitting cluster size "walls" where you have lots of cpu free but very little memory free, and want to resize the task definitions to "fit better"."__

So you want to be able to change task size in task definition and then update the service with the new task definition? Could you please shed some more light on what is the desired action?

[CpuID]

@pavneeta I'll rephrase that:

if you find you are hitting cluster size "walls" where you have lots of cpu free but very little memory free, and want to resize the task definitions to "fit better"

Replace with:

if you find you are hitting cluster size "walls" where you have lots of cpu free but very little memory free, and want to keep the task sizing relatively close but change the priority of the binpack algorithm to prefer cpu instead of memory, this may yield a better outcome. being able to test the outcome without downtime also would lower the bar to higher cluster utilizations

:)

[mehric]

Hi @pavneeta, we have many services on ECS and this is an important missing feature for us. We have several older services created with strategies that we are not able to update because we cannot tolerate the downtime. Changing them by recreating the whole services is a real pain, we need the feature to update the placement strategy (maybe via the task definition).

[pavneeta]

@mehric Thank you for your valuable feedback. We are currently scoping the feature to solve this known problem.I wanted to dive a bit deeper into this, If you had a way to update the placement strategy without having to recreate the entire service, the scheduler would still need to move tasks around to satisfy the new strategy - which could potentially lead to disruption in the interim. Would you be open to creating the new service first with the desired placement, having it up and running and then taking down the existing service?

[mehric]

@pavneeta, thanks for your reply.

That is exactly what we are doing currently, but we have a lot oh headache here and there especially with monitoring tools, here is what we are doing: (1) First, we create a temporary service using the same ALB. (3) Drain the old service and delete it. (4) Create a new service with the same name and updated placement strategy. (5) Drain the temporary service and delete it.

As you can see, it is a lot of effort and pain for us to keep the same service name and avoiding disruption, while changing a simple placement strategy. Would really appreciate it if this was simply updateable either on the service or on the task definition.

[maxblaze]

@pavneeta We are also in great need of this. The idea is to update the service with the new placement strategy and then for it to take effect on the next deployment (existing tasks _do not_ need to be shuffled). Recreating services is not an option for us.

[kensaggy]

+1 for @maxblaze ‘s suggestion of having it take effect on next deployment and not reshuffle tasks.

Or not necessarily next deployment. If one of the tasks crashes and the service brings it back up it can use the new placement strategy so it’s more of a “rolling” update

[mauricios]

In my case I need that ECS reshuffle tasks to balance the placement in the cluster without new deployment.

I have clusters with 2 instances for High Availability, when one of them crashes, the tasks are placed in the remaining instance, but when the new instance is registered in the cluster the tasks are not redistributed, leaving the service in a single instance.

[danny-larsen]

We also need this. We want to change it, but at stated, it requires a lot of effort and potentially some downtime. I would also love @maxblaze 's suggestion of just applying on the next deploy. It just seems weird that it can't be changed.

[pavneeta]

Hi Everyone, thank you for sharing your feedback and suggestions. We are currently working on this feature. I wanted to get everyone's thoughts on the CloudFormation behavior around service configuration updates.
Currently, AWS Cloudformation allows you to update your ECS service’s placement strategy and constraints as part of the update stack behavior. However, the current CFN behavior is that of Replacement. AWS CloudFormation recreates the service during an update to placement strategy/constraint, which also generates a new physical ID.

• Is this a problem for you?
• How have you built your deployment tooling and CFN stacks to accommodate for the "replacement" behavior highlighted?

[CpuID]

@pavneeta zero impact for us - as we don't use CloudFormation at all ;)

I'll defer to others to respond there.

[rgarcia]

This is a problem for us. We specify a human-friendly name for our ECS services, and so if there is a change that requires replacement, we have to perform that change over the course of multiple changesets. This ends up being a pretty custom and manual process for us. We recently had to do this to migrate from Fargate to Fargate spot. We would strongly prefer if CloudFormation had fewer "Replacement" behaviors with regard to ECS service properties.

[c-ameron]

@pavneeta
Thanks for the update. I'm currently using this with Terraform, no CloudFormation. So ideally it would be an in-place replacement.

[pavneeta]

Hi everyone,

This feature is now in Preview (Beta) you can use the CLI or SDK to update Placement Strategy and Constraints for existing services on ECS. See the What's New post here.

CloudFormation and Console support to follow..

Please feel free to leave feedback on this thread.

Thanks.

[CpuID]

@pavneeta thanks! can you confirm what conditions exist around using this in a Developer Preview state? do we need our accounts enabled for this, or is it available by default but just treated as beta functionality?

[pavneeta]

@pavneeta thanks! can you confirm what conditions exist around using this in a Developer Preview state? do we need our accounts enabled for this, or is it available by default but just treated as beta functionality?

No, you do not need to get your accounts whitelisted. It is available to all ECS customers across all regions. However this is in beta in terms of functionalities and service terms.

"Updating the task placement strategies and constraints on an Amazon ECS service remains in preview and is a Beta Service as defined by and subject to the Beta Service Participation Service Terms located at https://aws.amazon.com/service-terms ("Beta Terms"). These Beta Terms apply to your participation in this preview."

[CpuID]

@pavneeta great thanks for clarifying :)

[pavneeta]

Hi everyone,

This feature is now in Preview (Beta) you can use the CLI or SDK to update Placement Strategy and Constraints for existing services on ECS. See the What's New post here.

CloudFormation and Console support to follow..

Please feel free to leave feedback on this thread.

Thanks.

We just launched the Console Support for Update Placement Strategies and Constraints.

CloudFormation Support coming soon.

[yerzhan7]

@pavneeta any ETA on CloudFormation Support? It still says Replacement on this page

[mitchlloyd]

Just ran into this surprising limitation with CloudFormation. This required manual intervention in the ECS UI to get a deployment to go through.

[zBart]

While it's great that this can be done through the console now, it would be nice if CloudFormation support for this would also be updated. We can currently change it manually through the console for our existing production setup, but ideally we want to stick to our templates.

[CpuID]

@pavneeta can you please confirm something from the announcement:

https://aws.amazon.com/about-aws/whats-new/2020/03/amazon-ecs-supports-in-preview-updating-placement-strategy-and-constraints-for-existing-ecs-services/

Now, customers can use the UpdateService API to change the placement logic for an existing service by updating the service with the desired placement strategy and constraints. ECS will enforce these changes at the time of the next service deployment.

Does this imply that a single UpdateService API call will take care of this? or do you need to run UpdateService twice and/or do a force deployment (via the forceNewDeployment param) and/or swap out a new task definition revision?

[jeeslo]

There is an issue open on the CloudFormation roadmap to track this effort: https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/651 , please add your interest there to help prioritize it.