Amazon-ecs-agent: Docker volumes still stored on /dev/xvda, not /dev/xvdcz

[fabfuel]

The disk of one of my docker nodes got full pretty quickly.
Is it correct, that all Docker volumes are still stored in /var/lib/docker/volumes on /dev/xvda instead of the separate Docker volume /dev/xvdcz?

[root@dockerhost ~]# cd /var/lib/docker/
[root@dockerhost docker]# du -h -d1
56K     ./network
4.0K    ./trust
5.8M    ./graph
2.0M    ./containers
4.2G    ./volumes
1.2M    ./devicemapper
4.0K    ./tmp
4.2G    .
[root@dockerhost docker]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  7.7G     0 100% /
devtmpfs        994M  152K  994M   1% /dev
tmpfs          1002M     0 1002M   0% /dev/shm

Best
Fabian

[samuelkarp]

Hi @fabfuel,

Yes, volumes are logs are stored on /dev/xvda while images and containers (which are both layered constructs) are stored on /dev/xvdcz. The two-disk configuration with /dev/xvdcz was picked primarily as a mitigation for performance regressions in Docker 1.9 related to layers (images and containers). The size of both devices should be considered when launching your instances.

Please let us know if you have any additional questions.

Sam

[fabfuel]

Hi @samuelkarp!
Good to know, thanks a lot for the explanation!

[johntiror]

Hi, does this mean that if I want to persist data I'll persist on /dev/xvda? If this is the case why when I create a cluster I'm only asked to configure the size of the bigger volume if the data is stored on the smaller one? I'm sorry if the question seems dumb, but I'm new to ECS/Docker. Maybe I explained here:
http://stackoverflow.com/questions/41961039/elasticsearch-with-docker-how-to-persist-data-with-aws

thanks very much for your help

[tunix]

@samuelkarp I was trying to understand why my containers keep writing to container instance root instead of /dev/xvdcz all day until I saw this ticket. I've an ECS instance (AMI: ami-95f8d2f3, with 8 GB root, 100 GB docker disk) and I'm trying to run a Couchbase docker container on top of that. Couchbase exposes a volume (/opt/couchbase/var) but I neither define any mount point nor volumes in task definition. I expected it to write its data into the container's disk space (/dev/xvdcz) but it seems like /opt/couchbase/var is mounted to host as /dev/xvda1 and all data written into /opt/couchbase/var eats the space on host. Just to be sure, I created an Alpine image with the following command:

docker run -it --rm -v /mnt/foo alpine sh

And I wrote a 2 GB file into /mnt/foo:

dd if=/dev/zero of=/mnt/foo/bar bs=1G count=2

When I check if the ECS host's disk usage, I see that, 2 GB is used from host machine and not from the container.

Is this the intended behaviour? Since that 100 GB is said to be for Docker, I thought of using another container as a volume. I created a volume in task definition without specifying a source path for it but again it uses the space from the root disk of host machine not from /dev/xvdcz.

By the way, I understand that I should have a seperate disk (mounted on host OS and from there to the container) to persist critical data. In this case, however, I want to store all data on the container itself since this is primaly a development environment. I think that, if I don't mount the exposed volume, the data written to the corresponding folder should be stored inside the container and therefore /dev/xvdcz.

[samuelkarp]

Hi @tunix,

This is the expected behavior. /dev/xvdcz is dedicated to layer storage; since volumes are not layers, they're not stored there. You'll see that this is how Docker works on your development system as well; layer storage is separate from volume storage.

It looks like there's currently no way to remove a VOLUME defined in the image (see https://github.com/docker/docker/issues/18286, https://github.com/docker/docker/issues/3639), so you'll either need to change the image you're using or ensure you have enough space for the data outside the layer storage (either on the root filesystem or on another volume that you've mounted in).

Sam

[foretspaisibles]

Where is the docker volume group initialised? The command grep -r vgcreate amazon-ecs-agent did not yield any positives!

[ianseyer]

can this please be clearly documented? this is in no way intuitive.

[bs-thomas]

Hello @samuelkarp,

Sorry if my question is redundant. When using CloudFormation and auto scaling to create my instance, I have 2 volumes by default:

/dev/xvda 8gb (root)
/dev/xvdcz 22gb

So if now, I do a "git clone", may I confirm that all the data will be downloaded and stored in /dev/xvda (8gb) (root)?

I want to know this because that's where most of the storage is needed for my case. I would have to consider increasing the size of /dev/xvda instead.

Thanks in advance for your kind assistance!

[wenlock]

Has Anyone found a way to automate cluster management for a 3rd volume or deploying a larger root volume?

[ianfixes]

@samuelkarp could you elaborate on how to set up an ECS instance with more disk space for /var/lib/docker/volumes?

Just altering the mount points so that /dev/xvdcz gets mounted at /var/lib/docker instead of /var/lib/docker/devicemapper would be a huge improvement, but I'm unclear on how to make that change (and the search terms "volume", "docker", and "mount" are too overloaded for me to find the proper docs).

[josuetex]

@ianfixes https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html