Machine: Default AMI for AWS driver unable to initialize Docker

[dsingh07]

Hi All,

When using the AWS driver for docker-machine, the driver is unable to initialise the docker daemon on the provisioned host and the command exits with the following error:
Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "<IP>:2376": read tcp 30.0.1.38:51238-
Upon examining the logs on the host, this is the error that was found:
ubuntu docker[4174]: time="2016-12-12T14:00:37.336333805Z" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"

This seems to be an issue with the AMI that is used by default (Ubuntu 15.10). When specifying an alternate AMI using the flag (i.e. Ubuntu 14.04 or Ubuntu 16.04), this error does not seem to occur.

Any help or clarification would be great!

[guikcd]

Same issue here but the last rc release (0.9-rc2) is using a newer 16.04 AMI.

The issue with 15.04 is that the docker daemon is not running because of:

Error starting daemon: error initializing graphdriver: driver not supported

Certainly because aufs support is no more present on this AMI.

[abruehl]

Confirmed. It's aufs. I was on the default (15.10 daily build) also tried the 16.04 daily.
Something changed over the weekend.

@dsingh07 I worked around by setting engine-storage-driver=overlay2

https://docs.docker.com/engine/userguide/storagedriver/selectadriver/

root@runner-fd49a367-gitlab-builder-auto-scale-1481590795-d05d96f9:/home/ubuntu# service docker status
● docker.service
Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2016-12-13 01:05:50 UTC; 7s ago
Process: 5192 ExecStart=/usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=amazonec2 --registry-mirror (code=exited, status=1/FAILURE)
Main PID: 5192 (code=exited, status=1/FAILURE)

Dec 13 01:05:49 runner-fd49a367-gitlab-builder-auto-scale-1481590795-d05d96f9 systemd[1]: Started docker.service.
Dec 13 01:05:49 runner-fd49a367-gitlab-builder-auto-scale-1481590795-d05d96f9 docker[5192]: time="2016-12-13T01:05:49.519936693Z" level=info msg="libcontainerd: new containerd process, pid: 5201"
Dec 13 01:05:50 runner-fd49a367-gitlab-builder-auto-scale-1481590795-d05d96f9 docker[5192]: time="2016-12-13T01:05:50.563879188Z" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Dec 13 01:05:50 runner-fd49a367-gitlab-builder-auto-scale-1481590795-d05d96f9 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 01:05:50 runner-fd49a367-gitlab-builder-auto-scale-1481590795-d05d96f9 systemd[1]: docker.service: Unit entered failed state.
Dec 13 01:05:50 runner-fd49a367-gitlab-builder-auto-scale-1481590795-d05d96f9 systemd[1]: docker.service: Failed with result 'exit-code'.

root@runner-fd49a367-gitlab-builder-auto-scale-1481590795-d05d96f9:/home/ubuntu# sudo modprobe aufs
modprobe: FATAL: Module aufs not found.

[fpavageau]

The kernel of the 15.10 image is in the linux-image-4.2.0-18-generic package, so docker-machine tries to install linux-image-extra-4.2.0-18-generic to get the aufs module. However, that version of the package has disappeared from the repository (probably because no release references it anymore):

# apt-cache search linux-image-extra
linux-image-extra-4.2.0-16-generic - Linux kernel extra modules for version 4.2.0 on 64 bit x86 SMP
linux-image-extra-virtual - Transitional package.
linux-image-extra-4.2.0-42-generic - Linux kernel extra modules for version 4.2.0 on 64 bit x86 SMP

The modules matching the running kernel cannot be installed anymore, so no aufs and no docker. The machine would need to have its kernel upgraded to linux-image-4.2.0-42-generic then reboot.

We successfully switched to the AMI for our region as defined in #3686, which is the upgrade to 16.04 in the upcoming v0.9.0.

[guikcd]

If using docker-machine 0.8.2, build e18a919, this can be overridden by:

docker-machine create --driver amazonec2 --amazonec2-region eu-west-1 --amazonec2-instance-type t2.micro --amazonec2-ami ami-21abf052 mydockermachine

If you are in eu-west-1 region for example.

[nathanleclaire]

Hm, this is really unfortunate. cc @thaJeztah -- seems the get.docker.com scripts aren't properly installing AUFS on 16.04 anymore (on AWS at least) ? I'll take a look into seeing if I can reproduce.

[nathanleclaire]

Weirdly, I can't reproduce w/ Machine 0.9.0-rc2. The machine creates fine. I'm using default amazonec2 flag values.

[nathanleclaire]

Ah, I misunderstood the issue. Seems 0.9.0 will fix it. Stay tuned all. Thanks