Amazon-ecs-agent: Unable to properly mount volumes on 2016.03.a AMI

[ChrisRut]

The following works just fine on 2015.09.g AMI (ami-33b48a59):

[root@ip-10-204-8-38 ec2-user]# df -h /media/ephemeral0
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdx        74G  8.3G   62G  12% /media/ephemeral0
[root@ip-10-204-8-38 ec2-user]# docker run --rm -v /media/ephemeral0:/testing debian:jessie df -h /testing
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdx        74G  8.3G   62G  12% /testing

But fails on the 2016.03.a AMI (ami-67a3a90d):

[root@ip-10-204-0-128 ec2-user]# df -h /media/ephemeral0
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdx        74G   52M   70G   1% /media/ephemeral0
[root@ip-10-204-0-128 ec2-user]# docker run --rm -v /media/ephemeral0:/testing debian:jessie df -h /testing
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  873M  6.9G  12% /testing

Notice /dev/xvda1 is the Filesystem, this should be /dev/xvdx.

In both cases, the ephemeral volume (/dev/xvdx) is being mounted via user-data on our instances like so:

if [ -e /dev/xvdx ] && ! mountpoint -q /media/ephemeral0; then
    mkfs.ext4 -q /dev/xvdx
    mount /dev/xvdx
fi

What am I missing ?

[ChrisRut]

This appears to be something specific to my configuration as I am not able to replicate this on a clean AMI. I am going to close this issue, and I'll follow up with details of what the root-cause is once I have one.

[ChrisRut]

The Docker service needs to be restarted after mounting a new filesystem, see:

• https://github.com/docker/docker/issues/5489#issuecomment-141438777
• https://github.com/docker/docker/issues/18113

Updated user-data:

# Ensure ephemeral volume is formatted and mounted
if [ -e /dev/xvdx ] && ! mountpoint -q /media/ephemeral0; then
    mkfs.ext4 -q /dev/xvdx
    mount /dev/xvdx
    service docker stop
    # Remove potentially corrupted network kv.db
    # see: https://github.com/docker/docker/issues/18113
    # @TODO: remove this once ECS AMI starts using Docker 1.10+
    rm /var/lib/docker/network/files/local-kv.db
    service docker start
fi