Amazon-ecs-agent: devicemapper leaking

[cjbottaro]

Hi.

Seeing containers fail with this message:

devmapper: Thin Pool has 4247 free data blocks which is less than minimum required 4454 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

Seems to be an issue with devicemapper:
https://github.com/docker/docker/issues/18867
https://github.com/docker/docker/issues/3182

Here's some info about my instance:

$ docker info
...
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: docker-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file:
 Metadata file:
 Data Space Used: 7.308 GB
 Data Space Total: 23.35 GB
 Data Space Available: 16.05 GB
 Metadata Space Used: 4.223 MB
 Metadata Space Total: 25.17 MB
 Metadata Space Available: 20.94 MB
 Thin Pool Minimum Free Space: 2.335 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.93-RHEL7 (2015-01-28)
...

$ sudo vgs
  VG     #PV #LV #SN Attr   VSize  VFree
  docker   1   1   0 wz--n- 22.00g 204.00m

$ sudo lvs
  LV          VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool docker twi-aot--- 21.75g             35.73  16.55

I'm using the latest ECS optimized AMI (the one with the letter g in it).

Is it possible for the next AMI to use the overlay fs and overlay2 driver?

Thanks.

[richardpen]

@cjbottaro DeviceMapper is the default storage driver on ECS Optimized AMI. We choose devicemapper as the default storage driver based on our qualification, because it is more stable than the other alternatives. You can change that by configuring Docker with the overlay/overlay2 storage driver following the instruction here. On ECS Optimized AMI you need to run /sbin/modprobe overlay to load the overlay and change the storage driver from devicemapper to overlay in /etc/sysconfig/docker-storage to make docker use overlay.

Thanks,
Richard

[cjbottaro]

Ok. For posterity, here's the bash script I used to do this. I made an AMI out of the results.

https://gist.github.com/cjbottaro/394bcf05deb83892a6bb2a7f4914e227

The AMI I used as a starting point is ami-a98cb2c3.

[borgstrom]

This PR & gist were very helpful in moving our ECS agents to overlay2 -- thanks!

FWIW, we did this purely in user-data so that we could just keep using the official ECS AMI.

# stop docker so we can convert our storage backend to overlay2
/etc/init.d/docker stop

# cleanup devicemapper
dmsetup remove_all
vgremove --force docker

# setup our volume for overlay2 usage
mkfs -t ext4 -L docker -i 4096 -F /dev/xvdcz
rm -fr /var/lib/docker/*
mkdir /var/lib/docker/overlay2
mount /dev/xvdcz /var/lib/docker/overlay2

# configure docker to use overlay2
echo 'DOCKER_STORAGE_OPTIONS="--storage-driver=overlay2"' > /etc/sysconfig/docker-storage

# restart docker
/etc/init.d/docker start

# set our ECS cluster
echo ECS_CLUSTER=mycluster > /etc/ecs/ecs.config

# re-setup the ecs agent
# it uses an upstart script in /etc/init/ecs.conf, so we use 'start' to control it
rm -rf /var/lib/ecs/data/* /var/cache/ecs/*
start ecs

[jedi4ever]

with the latest ami this still is an issue ; creating / removing files inside a container does not free the files until the containers are removed. So touch > bigfile , rm bigfile makes docker info report an increasing number.