Moby: [graphdriver] prior storage driver "aufs" failed: invalid argument

[phobologic]

When trying docker 1.7 (from the docker maintained repo) package on Ubuntu 14.04, I now get the following error on boot:

INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: invalid argument
FATA[0000] Error starting daemon: error initializing graphdriver: invalid argument
/var/run/docker.sock is up

This led me to this bug #7321, because we use btrfs for /var/lib/docker. I've since changed it to ext4, but I'm still getting the errors above.

root@ip-10-128-16-91:/etc/init# uname -a
Linux ip-10-128-16-91 3.13.0-55-generic #92-Ubuntu SMP Sun Jun 14 18:32:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

root@ip-10-128-16-91:/etc/init# mount | grep docker
/dev/xvdh on /var/lib/docker type ext4 (rw)

root@ip-10-128-16-91:/etc/init# docker --version
Docker version 1.7.0, build 0baf609

[phobologic]

Environment Details:

Fresh AWS c4.xlarge instance

Couldn't run docker info since the daemon wasn't able to start.

_Steps to Reproduce_:

1. Install docker 1.7 from package repository on Ubuntu 14.04, when it attempts to start it fails.

[isntall]

I've just run into this issue too, on AWS.

[PhE]

Same error here :

INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: driver not supported 
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported 

I am running docker in Linux Mint 17.1.
Docker was automatically updated from 1.6.2 to 1.7.
It worked fine before.

[cpuguy83]

@PhE you have a completely different error message and as such a different issue. Most likely you upgraded your kernel and did not install that kernels extras package, which includes the aufs kernel mods.

[cpuguy83]

@phobologic That error message is coming directly from the kernel. Seems there's some issue with it being able to setup the mounts.
Can you stop docker and check if you have any aufs mounts in /proc/mounts?

[isntall]

My issue came from a kernel upgrade.
I was running

    # apt-get -y update
    # apt-get -y upgrade
    # apt-get -y dist-upgrade
    # apt-get -y install aufs-tools openjdk-7-jre curl wget git vim make php-pear \
php5-dev php5-curl python mc gawk ssh grep sudo htop nmon mysql-client php5-cli \
sqlite3 sysstat sysdig linux-headers-$(uname -r) lxc bsdtar
    # apt-get -y install linux-image-extra-$(uname -r)
    # modprobe aufs
    # apt-get clean

So i think the uname -r was using the current kernel and not the new kernel that the dist-upgrade. So aufs wasn't working properly.

[benguillet]

@cpuguy83 I work with @phobologic , we do have an aufs mount in /proc/mounts:

/dev/disk/by-uuid/82e281c0-5930-4069-b6d7-799dd1e14e66 /var/lib/docker/aufs ext4 rw,relatime,discard,data=ordered 0 0

Any advice?

[cpuguy83]

@benjamin-guillet With docker stopped, call unmount on /var/lib/docker/aufs, then try to start docker again.

[benguillet]

@cpuguy83

root@*:~# stop docker
stop: Unknown instance:
root@*:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 1782524   5897368  24% /
none                   4       0         4   0% /sys/fs/cgroup
udev             3816100      12   3816088   1% /dev
tmpfs             765956     356    765600   1% /run
none                5120       0      5120   0% /run/lock
none             3829776       0   3829776   0% /run/shm
none              102400       0    102400   0% /run/user
/dev/xvdh      104857600     512 102731520   1% /var/lib/docker
root@*:~# umount /var/lib/docker
root@*:~# umount /var/lib/docker/aufs
root@*:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 1782532   5897360  24% /
none                   4       0         4   0% /sys/fs/cgroup
udev             3816100      12   3816088   1% /dev
tmpfs             765956     356    765600   1% /run
none                5120       0      5120   0% /run/lock
none             3829776       0   3829776   0% /run/shm
none              102400       0    102400   0% /run/user
root@*:~# mount /var/lib/docker
root@*:~# start docker
docker start/running, process 25124
root@*:~# docker ps
Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
root@*:~#

Docker starts correctly and works if I umount /var/lib/docker. We use it with an external EBS volume.

[cpuguy83]

@benjamin-guillet Your response is a little ambiguous. Is this resolved?

[phobologic]

@cpuguy83 @benjamin-guillet Sorry - no, it doesn't fix it for us. Our setup includes mounting an EBS volume as /var/lib/docker. For some reason, even after unmounting that, then umounting /var/lib/docker/aufs then remounting /var/lib/docker we see the issues above. I feel like we're missing a step here - should we leave /var/lib/docker mounted and then unmount /var/lib/docker/aufs?

Thanks for all your help with this.

[cpuguy83]

@phobologic Yes, if you are mounting /var/lib/docker elsewhere, definitely just unmount the aufs dir.
Generally Docker would unmount this on exit.

[phobologic]

@cpuguy83 Ok, we'll give that a try and see that it works. Thanks.

[bkeroackdsc]

see https://gist.github.com/parente/025dcb2b9400a12d1a9f

This is another example of Docker having major showstopping issues with one of the most widely-used distros (see #3182 )

[phobologic]

In the end it looks like we had to manually remove /var/lib/docker/aufs after mounting the newly formatted external volume and starting docker1.7. I'm not sure why it would be getting created in the first place, and I'm not entirely sure what is creating it.

We also made sure to unmount /var/lib/docker/aufs before we mounted the new volume. I'm not entirely sure which of our steps finally fixed it for us, but I hope to have some time to narrow it down.

[sleaze]

@phobologic thanks for the info, you just saved some bacon!

[kelindar]

Had the same issue when upgrading to kernel 4.0.x on Ubuntu, where aufs is no longer supported and rm -rf /var/lib/docker/aufs did the trick.

[sdiemert]

@Kelindar thanks! That worked for me.

[Toub]

Thanks @Kelindar

I had @PhE error message ("driver not supported ") on Debian Testing with kernel 4.0, and your solution worked for me.

[phobologic]

So, I had this bite me again today. I was able to dig in some more. Here's some background for what happens on my servers:

1. I build an AMI using ansible that installs most of my non-server specific software. This includes the docker ubuntu package, which comes with startup scripts.
2. When I boot various servers, I check to see if there is a drive mounted @ /dev/xvdh. If there is, that is assumed to be for /var/lib/docker. I then:
3. shut down docker (using the upstart init script)
4. initialize (mkfs) the device
5. mount it as /var/lib/docker
6. Start docker

The issue seems to be that sometimes, when stopping docker, it does not umount not only /var/lib/docker/aufs, but a subdirectory (in this last case: /var/lib/docker/aufs/mnt/ebe69f270843f32a24b4c16343f8c312897d38128bd36df8b25cfe4442a1de3e).

Since I've mounted over /var/lib/docker already with the new device, it doesn't give me the ability to unmount leftover volumes. I was able to manually fix it by doing the following:

1. stop docker (using the init script, though it was already down due to crashing repeatedly due to this bug).
2. umount /var/lib/docker
3. umount /var/lib/docker/aufs/mnt/ebe69f270843f32a24b4c16343f8c312897d38128bd36df8b25cfe4442a1de3e
4. umount /var/lib/docker/aufs
5. mount /var/lib/docker
6. start docker (using the init script)

At this point I'm not sure how I can deal with this in a safe way without having to get manually involved, or writing a script to handle it. My current workflow, in ansible, goes like this:

- service: name=docker state=stopped
  when: new_docker_fs|changed
# Make absolutely sure aufs doesn't exist or docker freaks
# https://github.com/docker/docker/issues/14026
- shell: /bin/umount -t aufs -a -f
  ignore_errors: yes
  when: new_docker_fs|changed
- shell: /bin/umount -f /var/lib/docker/aufs
  ignore_errors: yes
  when: new_docker_fs|changed
- mount: name=/var/lib/docker src=/dev/xvdh fstype=ext4 state=mounted
  when: new_docker_fs|changed
- file: path=/var/lib/docker/aufs state=absent
  ignore_errors: yes
  when: new_docker_fs|changed

The change being that now I call out to the umount command directly and try to force the host to unmount every aufs mount it has. I'm not sure that this will actually work, but I thought I'd at least give an update as to where I am today just in case this bites anyone else, and in case this gives @cpuguy83 any other info that might help.

[stefanteixeira]

Thanks @Kelindar! I also updated the linux kernel on Ubuntu and removing aufs saved me. :)

[agnivade]

+1 to @Kelindar. I also had the same problem after upgrading 4.0. But now all my containers are gone :(

I don't think there is any way to get those back ?

[GuillaumeSeren]

@Kelindar Thank you this was happened on Debian also, and removing '/var/lib/docker/aufs' did work.

[resmo]

can also confirm @Kelindar solution worked for me. Debian stretch.

[sagardixit84]

I had this issue and removing /var/lib/docker/aufs did not help.
I had /var/lib/docker bind mounted to another location, which I had to remove and then docker started successfully

[shivece]

After unwittingly upgrading to 1.10, I got this error starting the docker daemon:

sudo tail -10f /var/log/upstart/docker.log
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: driver not supported
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported

And @Kelindar trick worked for me:

rm -rf /var/lib/docker/aufs

[thaJeztah]

@shivece what platform are you on? How did you install/upgrade docker (using a static binary or from the apt repositories?) Did you also upgrade your system, causing the aufs requirements to no longer be in place?

[CyborTronik]

Had the same issue.
Was resolved after:

   sudo rm -rf /var/lib/docker/aufs
   sudo service docker start

Solution found there: http://log.rowanto.com/broken-docker-in-debian-jessie/

[BentHaase]

CAUTION!
rm -rf /var/lib/docker/aufs also deletes all docker containers...

[zave]

@Kelindar Thank you! I'm on Ubuntu 14.04, 3.13.0-79-generic, and removing '/var/lib/docker/aufs' resolved the issue for me.

[thaJeztah]

@zave Ubuntu 14.04 should have Suppprt for aufs, and is the recommended driver; did you recently update the kernel? If so, you may want to reinstall the Linux-image-extra package

[zave]

I didn't think to do that, since the fix mentioned above took care of the problem, should I do
that in any case?

[thaJeztah]

@zave the "fix" mentioned is basically to get rid of an actual error; the fact that you previously ran docker with the aufs driver, but the driver doesn't work. Removing the aufs directory just nukes your containers and images, so that docker picks the next available driver (which may be vfs in a worst case scenario).

The Linux-image-extra package is tied to your kernel version, so has to be reinstalled after upgrading the kernel; check https://docs.docker.com/engine/installation/linux/ubuntulinux/#prerequisites-by-ubuntu-version on how to install it

[zave]

@thaJeztah Ok, I'll do that. Thank you.

[zave]

@thaJeztah I did it. All is well again. I remember having done that when I installed the Docker Engine. I'd forgotten all about that. Great recommendation. Thanks again.

[schinzelh]

I think i have found a related issue:

I am running

"Docker version 1.10.2, build c3959b1" on
"Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-55-generic x86_64)"

All is fine when running Docker with default setup. But as soon as i change the Docker base storage directory via "-g" parameter in DOCKER_OPTS in /etc/default/docker i get the error

/var/run/docker.sock is up
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: invalid argument
FATA[0000] Error starting daemon: error initializing graphdriver: invalid argument

in my message log. I did not find a way around this yet.

[thaJeztah]

@schinzelh note that /etc/default/docker (and DOCKER_OPTS) is only used for systems running upstart/sysvinit, not systems that use systemd. Have you checked if the options you provided are actually picked up / used by the daemon?

[schinzelh]

@thaJeztah According to https://docs.docker.com/engine/installation/linux/ubuntulinux/#configure-docker-to-start-on-boot Ubuntu 14.04 is using upstart and as I see the error message in

/var/log/upstart/docker.log

i think i am good here. I have used

DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4 -g /srv/docker"

and the directory is picked up by the daemon - but it fails with the error message above while creating

 /srv/docker/aufs

[thaJeztah]

@schinzelh did you perhaps upgrade your system (and kernel)? In that case, check if the linux-image-extra package is installed properly for the kernel https://docs.docker.com/engine/installation/linux/ubuntulinux/#prerequisites-by-ubuntu-version

[zave]

That sounds familiar. That's what solved my problem.

[schinzelh]

@thaJeztah i already did. All packages including linux.image-extra are up to date.

Like i wrote: If Docker is configured to use its default storage location in

/var/lib/docker

all is working 100% perfect. But if i move the location to e.g.

/srv/docker

via the -g parameter the daemon fails to start with

/var/run/docker.sock is up
ERRO[0000] [graphdriver] prior storage driver "aufs" failed: invalid argument
FATA[0000] Error starting daemon: error initializing graphdriver: invalid argument

Would a AMI image be of any use for you to reproduce this issue?

[thaJeztah]

Is there something special with /srv/docker; Is it a regular directory, a mount?

[schinzelh]

/srv is actually a mount of /dev/xvdh

ubuntu@ip-10-73-178-189:~$ lsblk
NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
xvdb  202:16   0   840G  0 disk /mnt
**xvdh  202:112  0    40G  0 disk /srv**
xvda1 202:1    0    25G  0 disk /


ubuntu@ip-10-73-178-189:~$ mount
/dev/xvda1 on / type ext4 (rw,discard)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
/dev/xvdb on /mnt type ext3 (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
**/dev/xvdh on /srv type ext3 (rw)**
/mnt/home on /home type none (rw,bind)

[thaJeztah]

@schinzelh alright, one difference I see is ext3 vs ext4. Just to verify; if you use /var/lib/foobar as graphdir, does it work correctly?

If that works correctly, might be worth checking if things are missing, using the check-config.sh script

[schinzelh]

@thaJeztah Thanks for looking into this. I finally found out that it was caused by user error: The mount was not done via /etc/fstab but via a script executed at server startup in /etc/rc.local

This caused a race condition between docker deamon startup and mount command on server boot, which corrupted the docker storage directory.

Thanks for your help :+1:

[thaJeztah]

@schinzelh oh, that explains things! Good to hear you got it resolved!

[thaJeztah]

Alright, looking at the discussion above, I'm going to close this issue; this issue has been collecting different issues (resulting in the same "error" message). As far as I can see there's no bug involved here.

Feel free to comment here after closing, but please read the discussion above if your error can be the result of one of the things discussed above

[dwu-zingbox]

My experience show docker doesn't like to run under a root directory, meaning /var/lib/docker or what is specified by -g cannot be a root directory of an EBS volume or a LVM of EBS volume. Giving docker one more level of directory in the same volume through -g solved the problem.

I have Docker version 1.10.2, build c3959b1 and ubuntu 14.04, Linux 3.13.0-79

# df
Filesystem                     1K-blocks    Used Available Use% Mounted on
...
/dev/mapper/data-docker         10190136   23144   9626320   1% /data/docker

DOCKER_OPTS="-g /data/docker/docker" works but DOCKER_OPTS="-g /data/docker" doesn't.

[siliconcow]

Yep, weird bug but reproduced it on 14.04 w/ docker 1.6.2

[pmontrasio]

The solution by @thaJeztah worked for me. I upgraded to linux 3.13.0-88-generic and I had to sudo apt-get install linux-image-extra-3.13.0-88-generic. Started docker and it works.

Maybe the error in the log file

[graphdriver] prior storage driver "aufs" failed: driver not supported
Error starting daemon: error initializing graphdriver: driver not supported

should hint to that package. How about adding a line like this?

You could have to upgrade the linux-image-extra to the version matching your kernel.
Example: sudo apt-get install linux-image-extra-3.13.0-88-generic

Better if with the appropriate kernel version and the appropriate package manager according to the distribution.

[jamesfryer]

Same issue, sudo apt-get install linux-image-extra-3.13.0-88-generic fixed it for me.

[jocull]

This happens to me all the time. It seems like it happens after running dist-upgrades in Ubuntu, and I always have to go in and run this:

sudo apt-get install linux-headers-$(uname -r) linux-image-extra-$(uname -r)

[thaJeztah]

@jocull correct; if a dist-upgrade upgrades the kernel, chances are you need to install the correct version of the linux-image-extra package for the updated kernel. I _think_ that installing the linux-image-extra-virtual package would help in that situation (and was recently added to the docs, to match what the install script installs; https://github.com/docker/docker/pull/25614)

[jocull]

I will try that and see if it makes a difference! Thank you!

[MouhamedAymen]

I had the same problem, after updating to 1.12.1 and rebooting.
I solved it by removing /var/lib/docker/overlay (or just renaming it smth else, just in case, you know) and restarting docker. I have no idea why it worked, but glad it did.

[toudidel]

I had the same problem if I wanted to changed my docker images location.
I performed solution from http://stackoverflow.com/a/34731550 and added file /etc/systemd/system/docker.service.d/docker-storage.conf with content:

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --graph="/mnt"

After restarting the system I had the problem. If I delete this file and return to default location (/var/lib/docker) everything is OK.

[mathieufrh]

aufs is not supported by latest kernels, so you should use overlayfs instead.
I just edit the /etc/systemd/system/docker.service.d/docker.conf this way:

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --storage-driver=overlay

Then do:

# systemctl stop docker.service
# systemctl daemon-reload
# systemctl start docker.service

[f3l1x]

I'm running Linux Mint 18 and I've updated from kernel 4.4.0 to 4.8.11. (29.11.2016).

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2016-11-29 13:50:20 CET; 8s ago
     Docs: https://docs.docker.com
 Main PID: 9128 (code=exited, status=1/FAILURE)

Nov 29 13:50:19 XPS systemd[1]: Starting Docker Application Container Engine...
Nov 29 13:50:19 XPS dockerd[9128]: time="2016-11-29T13:50:19.400041011+01:00" level=info msg="libcontainerd: new containerd process, pid: 9135"
Nov 29 13:50:20 XPS dockerd[9128]: time="2016-11-29T13:50:20.406203559+01:00" level=error msg="[graphdriver] prior storage driver \"aufs\" failed: driver not supported"
Nov 29 13:50:20 XPS dockerd[9128]: time="2016-11-29T13:50:20.406360943+01:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Nov 29 13:50:20 XPS systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 13:50:20 XPS systemd[1]: Failed to start Docker Application Container Engine.
Nov 29 13:50:20 XPS systemd[1]: docker.service: Unit entered failed state.
Nov 29 13:50:20 XPS systemd[1]: docker.service: Failed with result 'exit-code'.

---

This helps me out.

 sudo rm -rf /var/lib/docker/aufs
 sudo service docker start

[thaJeztah]

Be aware that sudo rm -rf /var/lib/docker/aufs removes all existing images and containers. Basically it's a "factory reset"

[cpuguy83]

And the real fix is to re-install the aufs kernel module when you upgrade your kernel.

[jocull]

Just to follow up on @thaJeztah's comment, installing the -virtual packages did not correct the issue on my end.

This could be related to using Dokku, or maybe not. I really don't know, but it still breaks every time I run a system upgrade.

[bogste]

Thank you @Kelindar for the answer at https://github.com/docker/docker/issues/14026#issuecomment-128055691

I just upgraded the Kernel to 4.9 and was getting

[graphdriver] prior storage driver aufs failed: driver not supported

Moved the /var/lib/docker/aufs folder to my $HOME_DIR/Backups/docker (just in case) and was able to run service docker stop/start. Now it starts up as it should!

[Vadorequest]

Had the same issue, having an OVH VPS with pre-installed docker, I was getting this annoying error message. Deleting aufs directory did the trick, I was able to start docker service again after that.
I don't even know why I got this bug in the first place, maybe OVH did something with their VPS kernel on the background.

[thaJeztah]

Yes, if they upgraded the kernel and did not install the required kernel modules after that, then aufs would no longer start. Docker does not automatically pick the next available driver in that situation, because then you won't see your existing images and containers, so it deliberately fails to start to give you an option to resolve the situation.

[tahonaPL]

I had this error after kernel update from 4.4.1 to 4.4.129 (Mint 18.1) . I couldn't revive docker-engine so I installed docker-ce and start working:

apt purge docker-engine
rm -Rf /var/lib/docker

and then:

https://coytar.wordpress.com/2017/03/23/docker-ce-on-linux-mint-18-1/

[prhbrt]

 # apt-get -y install aufs-tools openjdk-7-jre curl wget git vim make php-pear \
 php5-dev php5-curl python mc gawk ssh grep sudo htop nmon mysql-client php5-cli \
 sqlite3 sysstat sysdig linux-headers-$(uname -r) lxc bsdtar

What do php, python, mysql, sqlite3 and java have to do with this issue?

[prhbrt]

@Kelindar , thanks, I gave you a thumbs up and down because I think moving is better than deleting.

sudo mv /var/lib/docker/aufs /root/var-lib-docker-aufs-backup`date +%Y-%m-%d`