The default of no swap seems a bit heavy-handed, for some programs it's really hard to pin-point the usage you'll need, and often 90% of the time they use 10% of that limit until a spike. Maybe an option to disable the limit all together would be nice, not like we want swap at all haha, but for spiky behaviour it's tricky. Let me know what you think!
tj
All 47 comments
I think this really raises two separate issues. Swap support seams like a reasonable thing to add to me.
Personally, I think being able to set an acceptable amount of overcommit and then an oom-kill priority on tasks would be useful for the spiky case and many cases. There's a docker issue for making the priority easy to set and I think without that overcommit is much less valuable.
I'll have to run the above by other people on the team, but if you think that overcommit would help or have other ideas for how to handle that sort of thing, we'd value the feedback.
Best,
Euan
euank
on 26 Jun 2015
Hmm thinking about it more, it's sort of a program design problem on our end and me under-scheduling to make sure we have good utilization on the hosts. A bit of swap would be nicer than crashing, at least that would give us a chance to alert and raise the limit. So far we haven't had any issues actually exhausting memory on the host, just the mis-configurations for containers.
Semi-related but most of the unpredictable ones are more cron-like, we actually use go-cron in a few of them which is partly why it feels like a huge hack to allocate say 3GB to something that uses a tiny fraction of that most of the time. I know we could use start/run-task but if there's any internal debates regarding cron support I'd give that a big +1 :D.
tj
on 26 Jun 2015
+1 for an option to enable swap.
I have been testing a high memory worker service on ECS. On a large instance running a single container with a large memory parameter I saw the Python worker crash:
worker: OSError: [Errno 12] Cannot allocate memory
In this simple case with 1 container per instance, my expectation would be that the container can spike and use swap, and really any amount of the host resources, and not get killed.
But I don't see any way in the APIs to tune the behavior.
nzoschke
on 17 Jul 2015
We worked around this in Convox by monitoring Docker and modifying the cgroups directly after containers start
https://github.com/convox/agent/blob/master/monitor.go#L120-L145
nzoschke
on 6 Aug 2015
+1 for option to enable swap and over-provision memory.
bcwp
on 13 Aug 2015
Cross-referencing this with very similar discussion on the AWS forums: https://forums.aws.amazon.com/thread.jspa?messageID=671642򣾚
My understanding is somewhat limited, but if I understand everything correctly allowing for configuration of the memory-swap parameter would implement the "optional memory limit" feature requested there, yes?
(I'm "ekmallon" on that thread, FWIW)
iameli
on 21 Aug 2015
+1 for option to enable swap
Our use case involves automatically scaling our ECS Cluster and automatically scaling our ECS Services to respond to both the normal daily rise and fall in demand as well as spikes in demand. Some services are CPU heavy and others are memory heavy, so we need to be able to scale up/down on both.
Enabling a fixed amount of swap per service, just like we can with physical memory currently, would be ideal for us. That would allow services to spike into swap and give us enough time to scale up while still serving requests. A fixed amount of swap per service would protect us against rogue containers, without having to deal with the OOM issues involved in allowing overcommit on physical memory.
The advantage of this approach is the ECS Cluster can still be scaled as we know how much swap/CPU/memory is available cluster-wide. Allowing overcommit on physical memory would be great for utilization and cron/batch jobs though...
gavannewell
on 26 Aug 2015
I'd like to add a little more information here since I've investigated this further. Although it's not intentional, right now the default Docker options for swap memory are used. That means that a container does get some amount (in recent Docker versions, default swap is the same as specified memory).
On ECS, your containers will be given swap. docker inspect will show 0, which really means default.
The above only applies if the AMI you're running has swap available, however, and most AMIs / instance types do not have swap by default. There are also kernel configurations that can prevent swap from being used.
Proper swap support should still be modeled and variable per task, so the issue remains, but I wanted to clarify what the current behavior actually is.
Best,
Euan
euank
on 29 Oct 2015
:+1: for allowing specification of docker memory options without ECS imposing further restrictions.
leftclickben
on 15 Nov 2015
:+1:
benweet
on 31 Jan 2016
+1 for memory-swap option. Rabbit MQ container needs it badly.
tuusberg
on 7 Feb 2016
+1
tallavi
on 8 Feb 2016
+1
Ouwen
on 11 Feb 2016
+1. kind of sad that such thing not implemented yet
aldarund
on 12 Feb 2016
Here's one line of userdata for the ECS AMI to make a 5 GB swapfile:
fallocate -l 5G /swapfile && chmod 0600 /swapfile && mkswap /swapfile && swapon /swapfile
(In context: https://github.com/convox/rack/blob/master/api/dist/kernel.json#L644)
I'm not sure if this is enough to make the default docker swap usage that @euank mentioned sufficient.
It would be nice if it is, as I'd love to remove the extra container monitor and cgroup modification hack that I'm using here:
https://github.com/convox/agent/blob/master/containers.go#L173
nzoschke
on 12 Feb 2016
+1
paulrutter
on 3 Mar 2016
+1
Ouwen
on 5 Mar 2016
+1
mixja
on 5 Mar 2016
This is definitely needed. Would love if the ECS group responded with at least an acknowledgement.
ryanwalls
on 15 Mar 2016
+1
JanBednarik
on 23 Mar 2016
+1
devotox
on 15 Apr 2016
+1
ghost
on 15 Apr 2016
+1
edgaredel
on 21 Apr 2016
+1
ooleem
on 1 May 2016
+1
bryceray1121
on 26 May 2016
+1
gabadi
on 8 Jun 2016
+1
theikkila
on 12 Jun 2016
+1
mjaverto
on 16 Jun 2016
The main thing that need this to be ungently fixed is that we ended having to pay for a larger instance just because we need more memory during the build/deployment process. If we could use memory swap we can still be using smaller instances and the build process will use swap memory to complete the process if needed. Win win.
igorescobar
on 8 Jul 2016
Hi all,
We recently added support for memory reservation to allow more flexibility in memory settings (see https://github.com/aws/amazon-ecs-agent/issues/155#issuecomment-232227172 for a discussion of what we built). Reading over all the use-cases in this thread again, it looks like they're all covered by the memory reservation support we added.
I'm going to close this issue for now. If anyone still has a need for swap specifically and isn't covered by either the memory reservation feature or https://github.com/aws/amazon-ecs-agent/issues/124#issuecomment-152307508, please open a new issue specifically focused on swap.
Thanks,
Sam
samuelkarp
on 25 Aug 2016
I think swap VS memory reservation is something different. With swap I explicitly make the choice to trade performance for stability.
PepijnK
on 9 Sep 2016
2nd this, they are two different things. I have a scenario where I don't care if I'm using RAM, I have a background worker that can spike on certain jobs to a lot of memeory, don't care if it's in ram, etc. Don't need it to be fast, and would appreciate not having to pay for the larger instance I don't need 99% of the time.
I think you should support the same options as docker:
https://docs.docker.com/engine/reference/run/#/user-memory-constraints
specifically the memory-swap option.
jamesjnadeau
on 22 Sep 2016
@jamesjnadeau exactly!! What @samuelkarp said didn't solve the problem... at least for what comes to be memory swapping. (i.e: https://github.com/aws/amazon-ecs-agent/issues/124#issuecomment-231325422)
igorescobar
on 22 Sep 2016
Seems that the only thing they need to do is to read the standard $DOCKER_OPTS environment variable inside of the /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh. Right on this line:
But kinda sucks having to overwrite the original file just for adding this and basically lose any further oficial update for it.
igorescobar
on 22 Sep 2016
+1
bs-thomas
on 5 Nov 2016
+1
ivistvanvarga
on 7 Feb 2017
memory-swap will be really useful... +1
albertocubeddu
on 29 Mar 2017
+1
mhowell-ims
on 19 Apr 2017
As suggested by @samuelkarp above, I've opened a new issue #771 specifically for swap support. @PepijnK @jamesjnadeau @igorescobar @bs-thomas @ivistvanvarga @albertocubeddu please contribute your thoughts to #771.
mhowell-ims
on 19 Apr 2017
@nzoschke @bcwp @gavannewell @leftclickben @tuusberg please contribute your thoughts to #771 as well if you can.
mhowell-ims
on 19 Apr 2017
+1 for having ability to use swap memory with ECS
rubenhak
on 1 Oct 2017
+1 for swap
gdrte
on 2 Mar 2018
+1
evgeny-myasishchev
on 19 Apr 2018
+1 for swap
newhoggy
on 10 May 2018
+1 for swap
cat-turner
on 14 Jun 2018
+1
divick
on 4 Mar 2019
+1
jpablos11
on 26 Mar 2019
Related issues
hayajo
路
3Comments
taktakpeops
路
4Comments
MartinMitro
路
3Comments
melo
路
5Comments
leonblueconic
路
3Comments
Most helpful comment
2nd this, they are two different things. I have a scenario where I don't care if I'm using RAM, I have a background worker that can spike on certain jobs to a lot of memeory, don't care if it's in ram, etc. Don't need it to be fast, and would appreciate not having to pay for the larger instance I don't need 99% of the time.
I think you should support the same options as docker:
https://docs.docker.com/engine/reference/run/#/user-memory-constraints
specifically the memory-swap option.