buildkit dies when multiple builds are happening with cache mounts

[daveisfera]

Our builds will frequently fail when multiple builds are happening that use cache mounts. Here's the error output:

#2 [internal] load .dockerignore
#2 DONE 0.0s

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile:
#1 transferring dockerfile: 479B done
#1 DONE 0.4s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.5s

#3 resolve image config for docker.io/docker/dockerfile:experimental
#3 DONE 0.4s

#4 docker-image://docker.io/docker/dockerfile:experimental@sha256:787107d7f...
#4 CACHED
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: frontend grpc server closed unexpectedly 

docker --version: Docker version 19.03.6-ce, build 369ce74
uname -a: Linux ip-___.us-west-2.compute.internal 4.14.173-137.228.amzn2.x86_64 #1 SMP Thu Mar 19 16:50:21 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[tonistiigi]

Please provide reproducer. Might have something more useful in the daemon logs.

[daveisfera]

Have multiple Dockerfiles that are built at the same time. Even something simple like this will make it happen:

# syntax = docker/dockerfile:experimental
FROM node:12.16.2-alpine3.11

RUN mkdir -p /usr/src/app

WORKDIR /usr/src/app

COPY package.json yarn.lock /usr/src/app/
RUN --mount=type=cache,target=/usr/local/share/.cache/yarn \
  yarn install --frozen-lockfile --link-duplicates

COPY . /usr/src/app

CMD ["./my_cmd.js"]

Here's the daemon logs from the minute when this build was running:

Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.194475673Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 systemd: Started Session 22472 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Starting Session 22472 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Started Session 22475 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Starting Session 22475 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Started Session 22470 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Starting Session 22470 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Started Session 22473 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Starting Session 22473 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Started Session 22474 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Starting Session 22474 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Started Session 22476 of user root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Starting Session 22476 of user root.
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.239600278Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.271664253Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 systemd: Removed slice User Slice of root.
Apr 22 23:50:01 ip-172-31-19-202 systemd: Stopping User Slice of root.
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.326422207Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.355288037Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.411010917Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.438537467Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.501407456Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.512906894Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.585573840Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.601737116Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.674449393Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.698042067Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.754967464Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.765533013Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.841606813Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.861328732Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.923336874Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:01 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:01.937540697Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:02 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:02.011004176Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:02 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:02.030430398Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:02 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:02.101620492Z" level=error msg="failed to kill runc t7yvihenzfcwuz050i3xk5ic6: runc did not terminate sucessfully: container \"t7yvihenzfcwuz050i3xk5ic6\" does not exist\n"
Apr 22 23:50:02 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:02.117109639Z" level=error msg="failed to kill runc nwa02qf8fljmk7ynadcolmpkd: runc did not terminate sucessfully: container \"nwa02qf8fljmk7ynadcolmpkd\" does not exist\n"
Apr 22 23:50:02 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:02.521561749Z" level=warning msg="grpc: addrConn.createTransport failed to connect to { 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
Apr 22 23:50:02 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:02.579781878Z" level=info msg="parsed scheme: \"\"" module=grpc
Apr 22 23:50:02 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:02.579841212Z" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
Apr 22 23:50:02 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:02.579876699Z" level=info msg="ccResolverWrapper: sending update to cc: {[{ 0  <nil>}] <nil>}" module=grpc
Apr 22 23:50:02 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:02.579888754Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Apr 22 23:50:13 ip-172-31-19-202 dhclient[3431]: XMT: Solicit on eth0, interval 123080ms.
Apr 22 23:50:42 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:42.046201787Z" level=warning msg="grpc: addrConn.createTransport failed to connect to { 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
Apr 22 23:50:42 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:42.055162882Z" level=warning msg="grpc: addrConn.createTransport failed to connect to { 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
Apr 22 23:50:44 ip-172-31-19-202 dockerd: time="2020-04-22T23:50:44.597299072Z" level=warning msg="grpc: addrConn.createTransport failed to connect to { 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc

[chasain]

I haven't tested this thoroughly but I think I've found a workaround. If you specify the sha hash of the version of dockerfile you use it won't reach out to the registry to check for a new version. For instance when I updated ours to:

#syntax=docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc

we stopped getting the error