Boot2docker: error creating overlay mount

[kianmeng]

Dockerfile.txt

Result for most recent released versions.

• v18.09.5 failed
• v18.09.4 failed
• v18.09.3 failed
• v18.09.2 failed
• v18.09.1 failed
• v18.09.0 pass
• v18.06.1-ce pass

When build Docker images with more than 50-plus layers, the build will fail exactly at step 62 as shown below.

Step 62/100 : RUN touch foobar
error creating overlay mount to /mnt/sda1/var/lib/docker/overlay2/83fc9f0b117631bc8a412c13c44db45801ecb056068df501e6c093c7bc893a55-init/merged: no such file or directory
make: *** [Makefile:2: all] Error 1

To reproduce the bug.

docker build --no-cache -t "test-overlay-mount:alpine" -f Dockerfile.txt .

[tianon]

Oh this is bizarre -- my host daemon is 18.09.4 and it works fine there, but I can indeed reproduce in boot2docker (and they're both overlay2). I wonder if there's something in boot2docker's kernel that we need to tweak?

Here's the last few lines of docker.log, in case there's something useful in there:

time="2019-04-23T21:42:04.742912991Z" level=info msg="shim reaped" id=73e12bb0cf1733bc233279a28ed02ff208a2427a71b222b5d8090bc21089f3dd 
time="2019-04-23T21:42:04.753079768Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:04.771233513Z" level=warning msg="73e12bb0cf1733bc233279a28ed02ff208a2427a71b222b5d8090bc21089f3dd cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/73e12bb0cf1733bc233279a28ed02ff208a2427a71b222b5d8090bc21089f3dd/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:04.877586177Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/d8d17303cab57cf78d090d91153aab0bdd3eaf71f5724f65b7899535ed689a8e/shim.sock" debug=false pid=8353 
time="2019-04-23T21:42:05.024312033Z" level=info msg="shim reaped" id=d8d17303cab57cf78d090d91153aab0bdd3eaf71f5724f65b7899535ed689a8e 
time="2019-04-23T21:42:05.034812646Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:05.056345567Z" level=warning msg="d8d17303cab57cf78d090d91153aab0bdd3eaf71f5724f65b7899535ed689a8e cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/d8d17303cab57cf78d090d91153aab0bdd3eaf71f5724f65b7899535ed689a8e/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:05.171429505Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ad370ebc1a5500e3f474304c7f27827cfebac6f56bde5cedafce8bc4ffbf73a7/shim.sock" debug=false pid=8462 
time="2019-04-23T21:42:05.312189027Z" level=info msg="shim reaped" id=ad370ebc1a5500e3f474304c7f27827cfebac6f56bde5cedafce8bc4ffbf73a7 
time="2019-04-23T21:42:05.322298730Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:05.339279397Z" level=warning msg="ad370ebc1a5500e3f474304c7f27827cfebac6f56bde5cedafce8bc4ffbf73a7 cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/ad370ebc1a5500e3f474304c7f27827cfebac6f56bde5cedafce8bc4ffbf73a7/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:05.445777897Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/cf3b2008181bcc4b350cfe5e976bfbd5e6997b0b04a2f9321246e0cdbf010ca0/shim.sock" debug=false pid=8571 
time="2019-04-23T21:42:05.588702464Z" level=info msg="shim reaped" id=cf3b2008181bcc4b350cfe5e976bfbd5e6997b0b04a2f9321246e0cdbf010ca0 
time="2019-04-23T21:42:05.599140762Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:05.619888477Z" level=warning msg="cf3b2008181bcc4b350cfe5e976bfbd5e6997b0b04a2f9321246e0cdbf010ca0 cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/cf3b2008181bcc4b350cfe5e976bfbd5e6997b0b04a2f9321246e0cdbf010ca0/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:05.728919978Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/fae3859564aecef70b0516ed184bb77ed4422aa7185ffc8ba672539a1eb57eb8/shim.sock" debug=false pid=8681 
time="2019-04-23T21:42:05.869297886Z" level=info msg="shim reaped" id=fae3859564aecef70b0516ed184bb77ed4422aa7185ffc8ba672539a1eb57eb8 
time="2019-04-23T21:42:05.879415859Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2019-04-23T21:42:05.895837537Z" level=warning msg="fae3859564aecef70b0516ed184bb77ed4422aa7185ffc8ba672539a1eb57eb8 cleanup: failed to unmount IPC: umount /mnt/sda1/var/lib/docker/containers/fae3859564aecef70b0516ed184bb77ed4422aa7185ffc8ba672539a1eb57eb8/mounts/shm, flags: 0x2: no such file or directory"
time="2019-04-23T21:42:05.963812737Z" level=error msg="error unmounting /mnt/sda1/var/lib/docker/overlay2/74c47a47159d093611f4f37a0a3b2df4d7e7b7a75ed02f3199ef18a517ea65d7-init/merged: invalid argument" storage-driver=overlay2

[tianon]

https://github.com/docker/docker.github.io/blob/b7b1e03f0423c6cbc50ac3e242407be14988ebc6/storage/storagedriver/overlayfs-driver.md#how-the-overlay2-driver-works implies we still have a layer limit, but it's much higher than 100 (and https://github.com/moby/moby/issues/26380 talks about getting up to 122 reliably), so I'm not 100% sure what's happening here.

[tianon]

Nothing in https://github.com/boot2docker/boot2docker/compare/v18.09.0...v18.09.1 jumps out as particularly interesting, so maybe there's something in the kernel between 4.14.79 and 4.14.92 that can help explain what's changed?

[tianon]

Crap, definitely not the kernel -- just tried doing a build of master on 4.14.79 and it still stops at Step 62. Trying dropping the Docker version back down now.

[tianon]

Welp, that works fine, so it's narrowed down to being caused by _something_ in https://github.com/docker/docker-ce/compare/v18.09.0...v18.09.1 :thinking:

[tianon]

Looks like https://github.com/docker/engine/pull/84 / https://github.com/moby/moby/pull/37993 is probably the most likely candidate (especially since the error message in https://github.com/moby/moby/issues/37970 appears to match the one we're getting exactly).

What I'm still fuzzy on is why this would be causing the error to show up when the PR is literally supposed to be the fix for it. :confused:

cc @thaJeztah

(Also not sure why boot2docker is hitting it but my host isn't -- the value of /sys/module/overlay/parameters/index is N on both and they're similar Docker versions.)

[tianon]

Ran https://github.com/moby/moby/blob/2bfa24b39393f3e6e1ec38e1d76004e5a50d109f/contrib/check-config.sh against our current kernel config (for giggles, just in case) and it shows appropriately that we're only missing CONFIG_AUFS_FS and ZFS-related bits, as expected.

[borjator]

We are kind of blocked by this. Any leads I can follow to diagnose and try to fix this issue?