Amazon-ecs-agent: Options to set hard cap on CPU for ECS tasks

[kssaril]

Currently there is no options available to set hard cap on CPU for ECS Docker containers

Description

Docker 1.13 added and option --cpus to set a hard cap of the CPU which a docker container can allocate on host machine but AWS ECS task definitions doesn't seems to be supporting that .
ECS doesn't have support for --cpu-quota and cpu-period options too which can be leveraged to get the same results
An option for setting maximum cpu a task can utilize will be desirable in situations where multiple ecs (docker) containers are running on an EC2 instance in below example scenarios

1. A web service task ( java tomcat process ) running on an EC2 instance and servicing requests . During an auto scale event , ECS decides to place another task in the same EC2 instance . Since this new task startup is costly and there is no limit for the CPU it can consume , it will utilize all the available CPU in the EC2 instance which in-turn impact the existing running task which will cause the task to slow down causing a response time spike during auto-scale events .

2) Two or more different types of dockerized applications running on same EC2 instance , one application suddenly started running high on CPU due to some fault ( infinite loop , thread blocks etc.) . This faulty task can utilize all the CPU of the EC2 instance with the current setting and impact other tasks running

Expected Behavior

A hard cap on CPU for newly placed ECS containers will make sure the existing containers are not impacted during startup of new Containers or when another containers becomes faulty .

Observed Behavior

• ECS container startup utilizing all the CPU of the instance which its running causing other tasks to suffer

• Any faulty ECS task can cause other healthy tasks to suffer if it tries to USE all the available CPUs

[petderek]

Hi @kssaril,

Thanks for reaching out to us! We've recently added the 'cpu' field to the task definition at the task level. The task level cpu will function as a hard cap. In order to use this, you will need to be running a container instance with the newest agent release (1.16.0). More info can be found here.

This should address both of your example scenarios. Feel free to reopen this issue if you have additional questions.

[codepushr]

Quick question:
According to my investigations the native docker parameter cpu-shares is only a soft-limit, while cpuset-cpus is a hard limit.
Is the new 'cpu' field on the task level doing a hard or soft limit? Because what bugs me is it looks like a cpu-shares setting, and not a allocation of cores.

Thanks

Source: https://stackoverflow.com/questions/26841846/how-to-allocate-50-cpu-resource-to-docker-container

[samuelkarp]

The task-level CPU field is a hard limit, implemented with CPU quotas (not CPU shares). We do not currently use a cpuset to pin tasks to specific vCPUs.

[kssaril]

@samuelkarp I am looking at the aws documentation here
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html As per the documentation

cpu option maps to --cpu-shares option to docker run

which is infact a soft limit .

Can you please clarify your above comment ? Is there any other options other than cpu which is not in the documentation .

[hyksos]

Hi @kssaril ,

Check the section "container_definitions", there is a "cpu" setting for each container.
Check the section "task_size", there is a "cpu" setting for the whole task.

So the options are both called "cpu" but they are at different level of the task definition, the other answer in this post should taken in that context. "task-level" and "container-level" cpu setting.

[SleeperSmith]

@hyksos
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-cpu

"each task would be guaranteed a minimum ... when needed, and each container could float to higher CPU usage if the other container was not using it"

CloudFormation documentation prob need update.