Containers-roadmap: [ECS] Support use of swap memory

[mhowell-ims]

I have an ECS workload that requires a very large amount of memory. The performance of the workload is secondary, so using swap memory would be the ideal solution.

This issue is related to #124. In that issue, @samuelkarp suggested opening a new issue if swap support is still needed. Since several others added +1 to #124 after it was closed, I'm opening this issue now.

[igorescobar]

This would be fantastic! Personally we are having to pay for larger instances because during deployments we have two containers running (elastic beanstalk stack) at the same time so at this specific window of time we have to use a lot more memory and if you don't have it for those operations your deployment process become a mess. So you basically have to pay for larger instances with more memory just because you will eventually need for deployments. Not good.

[cabbruzzese]

Thanks for the feedback, we'll track this as a feature request. The use case details provided is very helpful.

[juhoojala]

We have a similar use case for using swap memory.

I'm wondering what would happen if I would add swap to the EC2 instance and use memoryReservation for all containers?

(See http://stackoverflow.com/questions/17173972/how-do-you-add-swap-to-an-ec2-instance)

Would the swap memory be available to the containers?

[mhowell-ims]

@juhoojala I tried exactly what you've suggested above, and it didn't work. It appears that when ECS runs your image, it prevents docker from accessing the swap memory. For now we've been forced to abandon ECS and instead install and run docker ourselves on our instances. We'd much rather use ECS, but can't justify the cost of the instances that would have enough RAM for our use case.

[juhoojala]

@mhowell-ims, many thanks for letting me now.

As a workaround, we have chosen to just use more instances, while hoping that AWS will eventually provide a proper solution for this issue. Let's hope its sooner rather than later.

[azban]

as stated in the original pr (https://github.com/aws/amazon-ecs-agent/issues/124#issuecomment-183386891) i don't think it's unreasonable to manage swap on the host instance via user data.

[mhowell-ims]

@azban I tried what I think you're suggesting, setting up swap space on the host instances via user data script, but the containers run via ECS on those hosts were unable to access the swap memory, so it doesn't work.

[rubenhak]

+1 for having ability to use swap memory with ECS

[hblanken]

+1 - swap memory with ECS would be very helpful

[LuisErnestoZamb]

+1

[ElMonty2k]

+1 This would be definitly usefull!

[xavi-]

@azban that method was broken in a recent version of the ECS-agent. Previously the ecs-agent would set the default swappiness to -1, which would cause docker to use the default swappiness on the machine (in our case, 60, as declared in cat /proc/sys/vm/swappiness). More recent versions of the ecs-agent explicitly set swappiness to 0.

[pbecotte]

if the container is using swap- does it use the host swap volume? What happens if the host does not have swap enabled?

[adnxn]

@pbecotte If the host does not have swap enabled, the container will not be able to use swap.

[pbecotte]

Thanks @adnxn - I learned earlier today that what I was seeing was the container running out of memory for the disk cache and suddenly having to read files from disk instead of the cache, not swap.

[nvivo]

I would really like to get an official explanation on why swap is not supported. It simply doesn't make any sense to limit this. Looks like AWS is doing this on purpose to force purchase of more expensive instances even when not needed. Unless swap can be used somethow, looks like I'll have to use something other than ECS.

[nonspecialist]

With at least ecs-agent version 1.16.1, the behaviour of swap allocation is as follows (empirically tested):

For reference, what ECS refers to as "hard" memory limit defines a container's memory setting, and what ECS refers to as the "soft" limit defines a container's memoryReservation setting.

memory (hard) | memoryReservation (soft) | swap amount | comment
----|----|----|----
defined | defined | 2 x hard limit | this is docker default behaviour
undefined | defined | 0 | could be interpreted to mean container cannot be swapped
defined | undefined | 2 x hard limit | this is docker default behaviour
undefined | undefined | n/a | not permitted

so ECS clearly _does_ support "swap" for containers, although there's no way of configuring the amount with any finesse.

An important note is that you do not need to have swap configured on the host system for Docker to swap containers -- it will use the docker-pool storage for this.

Also, this is "docker swap"; it is not visible to processes inside the container, for example via the free -m or swapon -s or other commands which report on the host swap configuration

[Dominik-K]

@nonspecialist Can you point us to a documentation for the docker-pool swap storage, please?

@samuelkarp We want to disable swap at all that the service fails fast if it reaches next to 100% memory usage. Currently it seems to use swap even on Amazon ECS AMI. Maybe because of the docker-pool storage usage.

The only way I've found to disable swap on Docker is: set memory-swap == memory.

[pparth]

We would also like to completely disable swap in ECS. Using swap in production servers is unacceptable for us as performance immediately deteriorates, whatever type of disks we use. Is the above way to disable swap trustworthy then?

[alsmola]

We ran into issues using swappiness with ECS and wrote up a blog post here: https://engineering.clever.com/2018/02/06/swappiness-amazon-ecs/

We were able to get the default swappiness level of 60 in our containers, but it required setting a swap file in userdata and updating the cgroup setting using docker start hooks.

[calvinwyoung]

@alsmola Did you have both a memory hard limit and soft limit defined when you found that ECS was setting swappiness to 0?

If so, how do we reconcile this with @nonspecialist's findings? Is it the case that defining a hard limit and soft limit will enable a swap volume of 2x the hard limit, but set swappiness to 0 such that the swap never gets used?

[alsmola]

@calvinwyoung We use both hard and soft memory limits, but our problem was the not the swap allocation size (we manually created the swap file in userdata, à la https://github.com/aws/amazon-ecs-agent/issues/124#issuecomment-183386891), but the swappiness value itself.

[nithinvenugopal]

@alsmola In my case I just need to change the swap space value. It is taking 2x the hard limit and not the value in host swap. How does this docker start hook work? Is this from docker compose?

[jeffreyctang]

+1 for enabling swap on ECS.

I'm willing/able to manage/configure the swap on the Container Instances.

We have several tasks that have very spiky memory usage. We currently have to provision the task definitions and clusters to be able to serve these occasional spikes. They are over-provisioned 99% of the time. Being able to absorb the spike in swap and let things run a little slowly would reduce our instance requirements we need to account for the memory spikes.

[newhoggy]

+1 for enabling swap on ECS

[fandiks32]

+1 for enabling swap on ECS

[SamuelFolkes]

+1 for enabling swap on ECS

[rizplate]

+1 :anguished:

[pgrzesik]

Currently (v1.20.3) it's back to setting swappiness to 0, +1 for this issue to be resolved

[francesco-cambiaso]

+1

[effyteva]

+1

[j-f-r]

+1

[lohmuller]

You can modify cgroup settings while the process is running. We added a hook on docker-events for the machine container instance, listened for starts, and updated the cgroups setting. The code in our docker-events system container looks like this:

func handleStart(client *docker.Client, id string) {
  swappiness := "60"
  for tries := 0; tries < 10; tries++ { 
    if tries > 0 {
      // handle race condition between docker start and existence of cgroup
      time.Sleep(500 * time.Millisecond)
    }
    container, err := client.InspectContainer(id)
    if err != nil || container == nil || container.HostConfig == nil || container.HostConfig.CgroupParent == "" {
      continue
    }
    swappinessFilePath := path.Join("/cgroup/memory", container.HostConfig.CgroupParent, id, "memory.swappiness")
    if err := ioutil.WriteFile(swappinessFilePath, []byte(swappiness), 0644); err == nil {
      break
    } else {
      if strings.Contains(err.Error(), "no such file") {
        continue
      } else {
        break
      }
    }
  }
}

Hey @alsmola , in your site do you added this in the userdata?
Where I need add this script? And how add "hook" to "docker-events"?
Thanks!

[adnxn]

today the ecs amis don't configure a default swap file - users would have to configure this manually as part of userdata since this feels like something that would be very workload dependent and unique for each use case. the latest version of the agent v1.26.0 will inherit the swappiness value from the host, since it is not explicitly being set per container by the ecs agent, as per docker documentation.

By default, if you do not set --memory-swappiness, the value is inherited from the host machine.

so moving forward - it seems we should be able to enable --memory-swap and --memory-swappiness, but this still means that the host swap has to be configured manually during startup with a userdata script depending on the particular use case. also - looks there are use cases where we just want to disable swap altogether. seems like being able to set --memory-swappiness=0 would allow that.

at this stage - i don't see any obvious blockers to allowing the two flags to be configurable via task definition. we plan to discuss this further and update this issue accordingly with a path forward.

[jpablos11]

+1

[tomi-vanek]

+1

[coultn]

This launched! https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-swap.html

[wwwouter]

Will there ever be Fargate support?

[thomas-maschler]

And support for Batch?

[effyteva]

Any updates on Fargate support for swap?

[sampathkolachana]

+1 for fargate support

[01101100]

+1 for fargate support