Containers-roadmap: Containers need additional storage

[blaines]

I'm looking in to how to modify the dm.basesize setting. It's not incredibly clear where to push these kinds of configuration settings. Could someone please follow up with a stable place to modify some storage defaults? Thanks!

I'm seeing related settings in /etc/sysconfig/docker-storage-setup, /etc/sysconfig/docker-storage, and other files.

More info about dm.basesize can be found here: https://docs.docker.com/v1.10/engine/reference/commandline/daemon/

[ec2-user@debug ~]$ cat /etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS="--storage-driver devicemapper --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true --storage-opt dm.fs=ext4"
[ec2-user@debug ~]$ cat /etc/sysconfig/docker-storage-setup
# Edit this file to override any configuration options specified in
# /usr/lib/docker-storage-setup/docker-storage-setup.
#
# For more details refer to "man docker-storage-setup"
DEVS=/dev/xvdcz
VG=docker
DATA_SIZE=99%FREE
AUTO_EXTEND_POOL=yes
LV_ERROR_WHEN_FULL=yes
EXTRA_DOCKER_STORAGE_OPTIONS="--storage-opt dm.fs=ext4"
[ec2-user@debug ~]$

For now, I modified OPTIONS in /etc/sysconfig/docker with --storage-opt dm.basesize=20G

Also, is there a way to allocate storage in a task definition? Do you by chance have any suggestions if one container for example needs 100G of storage, when the others may be sufficient with 10G?

[richardpen]

@blaines ECS Optimized Ami reads the information in /etc/sysconfig/docker-storage-setup and write configuration to the file "/etc/sysconfig/docker-storage" which will be used for docker daemon.

You need add the options in "/etc/sysconfig/docker" which docker use to config the storage. As the change requires docker to restart if docker is already running, ECS Optimized AMI start the docker automatically when it start, it may not be safe to restart docker in your normal user data(at which time docker may not have finished initializing). One way you can do is to use the multi-part cloud-init to add the option before docker daemon starts, in which you put your user data like this:

Content-Type: multipart/mixed; boundary="===============BOUNDARY=="
MIME-Version: 1.0

--===============BOUNDARY==
MIME-Version: 1.0
Content-Type: text/x-shellscript; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="standard_userdata.txt"

#!/bin/bash
echo "normal user data script part"


--===============BOUNDARY==
MIME-Version: 1.0
Content-Type: text/cloud-boothook; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="boothook.txt"

#cloud-boothook
echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=20G"' >> /etc/sysconfig/docker

--===============BOUNDARY==--

[blaines]

Excellent thanks for the tip @richardpen! I'd also like to hear about any possibility for customizing the storage quota for individual containers, not sure if it's possible or if there's a place for requests.

[samuelkarp]

@blaines https://github.com/docker/docker/pull/19367 added an option for changing the container rootfs size in Docker 1.12. While we do not currently support this option in ECS, I'd be happy to mark this as a feature request if it is something you'd use.

[rferreira]

+1 for this

[macdiesel]

+1 this would also help us.

[inversion]

+1, this would help us a lot

[jamborta]

+1 this would be very important

[andrew-aladev]

@richardpen. I couldn't understand anything. I've created a "cluster" in you beautiful admin panel. I've selected a great EBS size. Now I need to use regular ssh to tell your docker to use ALL EBS size as dm.basesize. Really?

What is a purpose of this beautiful admin panel? This is just a useless advertisement, right?

[andrew-aladev]

@samuelkarp You have an option "Memory Limits (MB)" in task -> container. Please add something like "EBS Limits (MB)".

[andrew-aladev]

I've fixed this by using "auto scaling" -> "launch configuration" -> "user data"

#!/bin/bash
set -e

echo ECS_CLUSTER=name >> /etc/ecs/ecs.config

echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=99999G"' >> /etc/sysconfig/docker
/etc/init.d/docker restart

This is a terrible workaround.

[blaines]

We're still running on the workaround.

[sergiofigueras]

+1

[fabiangebert]

+1

[pdefreitas]

+1

[macdiesel]

+1 again, still using the workaround for this.

[xennygrimmato]

@samuelkarp Any update on this feature request?

[qpleple]

+1

[djones]

+1

[JustinPullen]

+1

[mattfeury]

+1000

[ibrahima]

One minor note of clarification for those trying to use this workaround: The workaround mentioned in https://github.com/aws/amazon-ecs-agent/issues/549#issuecomment-281113044 is similar but perhaps a little suboptimal compared to approach mentioned in https://github.com/aws/amazon-ecs-agent/issues/549#issuecomment-251517887 , which is also mentioned in the docs - if you do it in a cloud-boothook rather than the regular userdata, you don't need to restart dockerd.

And if you're using terraform, you can use template_cloudinit_config and use that as your userdata rather than just a script - this lets you separate out the cloud-boothook from other regular userdata scripts you may have. Not sure how one would generate this config otherwise.

Just wanted to note this because I at first tried to do it with a regular userdata shell script and that didn't work, because I didn't realize that the cloud-boothook was a different thing.

[khalidhuseynov]

+1

[paulrutter]

+1

[techcto]

+1

[frsechet]

+1

[djones]

Gahh this got me a second time 7 months later. 😞

[djones]

This would all be fixed for me if aws/amazon-ecs-agent#1385 was merged. Then we could up a cluster with the workaround:

#!/bin/bash
set -e

echo ECS_CLUSTER=name >> /etc/ecs/ecs.config

echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=99999G"' >> /etc/sysconfig/docker
/etc/init.d/docker restart

automatically.

[vbarba]

+1

[dhananjaykadam]

What will be file path in case of windows server containers?
above examples mentions path as /etc/sysconfig/docker here
echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=20G"' >> /etc/sysconfig/docker

UPDATE: Got it, under C:\ProgramData\Docker\config\daemon.json