Cli: cannot write /tmp/tmp4phcyxn2 because server did not provide an image ID

[AlonMiz]

Description
Build fails with error:
cannot write /tmp/tmp4phcyxn2 because server did not provide an image ID

$ CACHE_FROM=27d861591f4da977a04e94431f1fce214485f293 COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build --progress=plain --build-arg BUILDKIT_INLINE_CACHE=1
Native build is an experimental feature and could change at any time
Building app-frontend
WARNING: Error loading config file: .dockercfg: $HOME is not defined
WARNING: Error loading config file: .dockercfg: $HOME is not defined
#1 [internal] load build definition from Dockerfile
#1 sha256:bfa4b15feec4aff66a56da2fffc697555bf3abd42288cdb47fe7d6e146e43d25
#1 transferring dockerfile: 1.98kB done
#1 DONE 0.5s
#2 [internal] load .dockerignore
#2 sha256:c8a814244b123e69ef5bdbc269631c12cdf20bed9340d85934d92a5b7b8c249f
#2 transferring context: 114B done
#2 DONE 0.4s

....
....
...

#33 158.8 webpack 5.9.0 compiled with 17 warnings in 153155 ms
#33 159.2 Done in 158.35s.
#33 DONE 159.4s
cannot write /tmp/tmp4phcyxn2 because server did not provide an image ID

Steps to reproduce the issue:

1. building images parallel with docker compose

Describe the results you received:
Docker compose fail to build

Describe the results you expected:
either success or provide a meaningful error message

Additional information you deem important (e.g. issue happens only occasionally):
the issue doesn't occur on every build
we use jenkins as our ci
and some of the build are failing. not all, not even most. but we have 10s of failures per day (out of hundreds)
Output of docker version:

WARNING: Error loading config file: .dockercfg: $HOME is not defined
Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:29 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:23 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 17
 Server Version: 20.10.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.4.0-1121-aws
 Operating System: Ubuntu 16.04.7 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 58.97GiB
 Name: ip-172-29-38-181
 ID: 52OS:AMCC:AUGD:EYU4:SQAP:3FUE:2OF7:LJMH:IWMF:VAAB:K7TG:XR3S
 Docker Root Dir: /mnt/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  registry.internal.****.io:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
AWS, ubuntu, running jenkins

[thaJeztah]

error is produced in this part; https://github.com/docker/cli/blob/2f20bf490e2b03d9b5a224768d35bd97325307ce/cli/command/image/build_buildkit.go#L358-L366

the issue doesn't occur on every build we use jenkins as our ci and some of the build are failing. not all, not even most. but we have 10s of failures per day (out of hundreds)

Curious; could it be that it's happening in situations where the cache was fully used, and no new image was built?

[thaJeztah]

Hmm... output of your example shows that at least _some_ part of the build did not use cache;

#33 158.8 webpack 5.9.0 compiled with 17 warnings in 153155 ms
#33 159.2 Done in 158.35s.
#33 DONE 159.4s

So _probably_ that's not it (unless it still produced the same artefact in the end 🤔)

[AlonMiz]

thanks! yea it does some of the build not from the cache (either local or remote)

[thaJeztah]

I asked @tonistiigi to have a look as well (as he may have a better clue what could cause this); it's a USA Holiday today, so I don't expect him to have a look Today at least.

If you have more information that could help debugging (if the failing builds are on a public repo and logs are still available; feel free to post a link)

[tonistiigi]

Please post a reproducer. If you can reproduce without compose then that would be preferable.

[AlonMiz]

it's not reproducible at all. it happens occasionally. and not in our local env. only in our CI
and it's a private repo as well but i can share more logs from the build. (not that they are so meaningful as i shared the important parts)

[tonistiigi]

@AlonMiz You don't happen to be using docker buildx install?

@ulyssessouza FYI if you've seen this in compose or have ideas. Does compose set --load always when invoking builds? Looks like image was never created. This happens on if you don't specify an output on buildx container driver (but there should be a warning then). Or if you specify output other than image.

[ulyssessouza]

@tonistiigi No. I've never seen that.
And docker-compose does not set --load nor use buildx only buildkit.

[tonistiigi]

@ulyssessouza buildx can replace docker build command if it is configured to do so. If compose requires image ID to be returned it should set --output=image. Not sure if it is actually related to this issue though.

[ulyssessouza]

@tonistiigi For the image ID docker-compose passes --iidfile do read it after the build and retrieve the image ID
https://github.com/docker/compose/blob/master/compose/service.py#L1875

[AlonMiz]

we have another insight, worth mentioning
in some cases when this error occurs, it throws the same error in multiple images builds at once, at the end of each build.
so in the same pipeline run, we have several successful builds and some failed builds of different images.

here's an example for such case
build1:

$ CACHE_FROM=$(yarn --silent print-latest-commit) COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build --progress=plain --build-arg BUILDKIT_INLINE_CACHE=1 $npm_package_serviceName
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Building jaeger-manager
WARNING: Error loading config file: .dockercfg: $HOME is not defined
WARNING: Error loading config file: .dockercfg: $HOME is not defined
#1 [internal] load build definition from Dockerfile
#1 sha256:a79fa14da5a0170e9236ac676ec6779d4dcfbcc8e464697067a3bd933d726097
#1 transferring dockerfile: 1.52kB done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:90ae748d3b72e855c0118400bbe684662f169fd6805ae892ae6ded9dace5167f
#2 transferring context: 112B done
#2 DONE 0.1s
#3 [internal] load metadata for registry.internal.***.io:5000/node:15.6.0-slim
#3 sha256:b8b171ed2bb83b615bf3a9008c7266f5f92d83bc1a79a984ac3c6b56ace38cbe
#3 DONE 0.1s
#6 [test  1/18] FROM registry.internal.***.io:5000/node:15.6.0-slim@sha256:01faeeed7eb949dbaa66272c3fa2110433dae7d96cd1b3722c3794c39f579c0f
#6 sha256:b87691b4bde16bb691d2ff64472dd216a5b19944931a0e1e0bb48f7f2c5d080b
#6 DONE 0.0s
#5 importing cache manifest from registry.internal.***.io:5000/jaeger-manager:master-latest
#5 sha256:b120e60b73693a61ffbf45e58ce20f24d8002c3689f2373cf59db45f522be485

.....................

#11 [main  5/11] COPY _isolated_/package-prod.json                          package.json
#11 sha256:d46c554d49bedfea59a4dbb6791329b61d2ee5d6055d23044d079853580f69aa
#11 CACHED
#20 [test  9/16] COPY _isolated_/workspaces  workspaces
#20 sha256:8f21ee22d594ae8e0b7e9afab3c9221a100ec4f71daf72807f99284fa321e962
#20 CACHED
#29 [main 11/11] RUN echo "{ "version": "9f3d85329e38618d79b125da4a49f2fda7fe2cda"}" > workspaces/packages/node-toolbox/src/version.json
#29 sha256:082d69a768d13bec98d465dad9eb42b2f664b6bd6a10a8393845fd5db6c5645f
#29 pulling sha256:92da7bf305a81027a1044dc5ccea512435b230ec14dc06c039f6e6d809ea7d66
#29 pulling sha256:92da7bf305a81027a1044dc5ccea512435b230ec14dc06c039f6e6d809ea7d66 0.5s done
#29 pulling sha256:269da87585a76a6d968186596102d6ce5caf92cbeed486e0a64cd18a07d1307e
#29 pulling sha256:269da87585a76a6d968186596102d6ce5caf92cbeed486e0a64cd18a07d1307e 0.2s done
#29 CACHED
#30 exporting to image
#30 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#30 exporting layers done
#30 writing image sha256:e77f2f3983eb8064dcbe16174ac963a1bf5248fd798063a1e65c341ce7e22de0 done
#30 naming to registry.internal.***.io:5000/jaeger-manager:9f3d85329e38618d79b125da4a49f2fda7fe2cda done
#30 DONE 0.1s
#31 exporting cache
#31 sha256:2700d4ef94dee473593c5c614b55b2dedcca7893909811a8f2b48291a1f581e4
#31 preparing build cache for export
cannot write /tmp/tmpxs87ztq9 because server did not provide an image ID

another image build failure at the same run:

$ CACHE_FROM=$(yarn --silent print-latest-commit) COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build --progress=plain --build-arg BUILDKIT_INLINE_CACHE=1
app-selenium uses an image, skipping
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Building e2e
WARNING: Error loading config file: .dockercfg: $HOME is not defined
WARNING: Error loading config file: .dockercfg: $HOME is not defined
#1 [internal] load build definition from Dockerfile
#1 sha256:1537c48e8ef1e725602c40aa757a5deb5699a3776824cc60b99fffa919398f6f
#1 transferring dockerfile: 1.46kB done
#1 DONE 0.1s
#2 [internal] load .dockerignore
#2 sha256:74b52332e587781e539c061ddb88c87dc4ac8bc93ab480b4000df7fb7efaa561
#2 transferring context: 112B done
#2 DONE 0.0s
#6 [test  1/18] FROM registry.internal.***.io:5000/node:15.6.0-slim@sha256:01faeeed7eb949dbaa66272c3fa2110433dae7d96cd1b3722c3794c39f579c0f
#6 sha256:b87691b4bde16bb691d2ff64472dd216a5b19944931a0e1e0bb48f7f2c5d080b
#6 DONE 0.0s
#3 [internal] load metadata for registry.internal.***.io:5000/node:15.6.0-slim
#3 sha256:b8b171ed2bb83b615bf3a9008c7266f5f92d83bc1a79a984ac3c6b56ace38cbe
#3 DONE 0.0s
#8 [internal] load build context
#8 sha256:e68651c4ea9337abc961a141c9daf6e82d6332f95fcd4d2e27905be7eb06262b
#8 DONE 0.0s
#5 importing cache manifest from registry.internal.***.io:5000/e2e:master-latest
#5 sha256:82132ab0041364bf21d505a58de90cd3fd122de1006e7da7ba70890af125dc69
#5 ...
#4 importing cache manifest from registry.internal.***.io:5000/e2e:9f3d85329e38618d79b125da4a49f2fda7fe2cda
#4 sha256:d9b0aefca28c59197196f892bd1f48c7fb831ca51f09dc272dbc0b1df900772c
#4 DONE 0.3s
#5 importing cache manifest from registry.internal.***.io:5000/e2e:master-latest
#5 sha256:82132ab0041364bf21d505a58de90cd3fd122de1006e7da7ba70890af125dc69
#5 DONE 0.4s
#8 [internal] load build context
#8 sha256:e68651c4ea9337abc961a141c9daf6e82d6332f95fcd4d2e27905be7eb06262b
#8 transferring context: 2.20MB 0.2s done
#8 DONE 0.2s
#10 [compile  4/13] COPY _isolated_/.yarnrc                                         .yarnrc
#10 sha256:a8de93648522abc542c4b7d63b4fdc65458af3bfbb2a7f551aa391ff7629a429
#10 CACHED
#13 [main  7/10] RUN yarn install --prod --frozen-lockfile && yarn cache clean
#13 sha256:8bcbfddec0b782e1b84257d10ef44b28ea822da735d67655349708e87979cfb3
#13 CACHED
#20 [compile 10/13] COPY .eslintignore  .eslintignore
#20 sha256:a4a216d0122930a6dbcb2265daed17c9e136dbcc8083ad32368302344775cb5f
#20 CACHED
#16 [compile  6/13] COPY _isolated_/workspaces-src-less                             workspaces
#16 sha256:59a88ebdff8ec1f37e9be53d855facda99f9c513e6a9815a3c999903d66b8425
#16 CACHED
#19 [compile  9/13] COPY ui-tests                                                   ui-tests
#19 sha256:69749031e71f23ae66a5da06eaed360b3d93e86c6cc2bea066e7948628347b31
#19 CACHED
#22 [compile 12/13] COPY tsconfig.json                                              tsconfig.json
#22 sha256:64f3a30b71a2a8447c98545ec18f5b40090bdba50f1ce69b595e023b45fbf293
#22 CACHED
#11 [main  5/10] COPY _isolated_/package-prod.json                               package.json
#11 sha256:51a702049c639d9e3151ea42a912cfe5bbfa9a0a01e067fa1b84c33e50bd778a
#11 CACHED
#24 [main  9/10] COPY --from=compile /src/dist                                   dist
#24 sha256:bc2b3ce92d6bb21c4a3d5378433049368f13db434ec3f1fe9a0c2f8ad403599e
#24 CACHED
#15 [compile  5/13] COPY _isolated_/package.json                                    package.json
#15 sha256:3c3f95eaac949457a8fea1c7d26ce635805e2f8beee1cff5e91d09aa9cf49ae2
#15 CACHED
#7 [test  2/18] WORKDIR /src
#7 sha256:5549513f33a5ad52ca6e7c5ffbde44172d06faeb3d275514beecf46db03e5db8
#7 CACHED
#14 [main  8/10] COPY _isolated_/workspaces                                      workspaces
#14 sha256:98fd5cc63ae0c7171b19a64ab2579942aa328816e81b50f32788b08fbd08df82
#14 CACHED
#9 [compile  3/13] COPY _isolated_/yarn.lock                                       yarn.lock
#9 sha256:a4c668e6a21d0392b7fd2bbfa164e06561ba60dce7930bfcfb443fc41c1e9397
#9 CACHED
#18 [compile  8/13] COPY _isolated_/workspaces  workspaces
#18 sha256:c2ccd3f37ef15cf4e75fd197ec60d91402aa9c75bb6f92b18bcaf454ed32be70
#18 CACHED
#12 [main  6/10] COPY _isolated_/workspaces-src-less-prod                        workspaces
#12 sha256:c5ae48d73cd06fb94d94ef049061211147bf915b256bec7d2024bb71b5df10af
#12 CACHED
#17 [compile  7/13] RUN yarn install --frozen-lockfile && yarn cache clean
#17 sha256:c4b2cd05ed60ad506acf9682d6e6f98c1c12a6c6e449ddd99cbb03b822850a9a
#17 CACHED
#23 [compile 13/13] RUN yarn lint && yarn compile
#23 sha256:baaaa6e0089de87a56d2a9085397193e6e8ae289211bcb68269e91e25f9291d8
#23 CACHED
#21 [compile 11/13] COPY .eslintrc.js   .eslintrc.js
#21 sha256:231a6f935f445c38b02537f58cbe20d1bd73587199a778f6e3a8c35090f547a7
#21 CACHED
#25 [main 10/10] RUN echo "{ "version": "9f3d85329e38618d79b125da4a49f2fda7fe2cda"}" > version.json
#25 sha256:70b0f0755d3104f43bb9132e8c178693f62000bbf600728b9cd5075e1550f507
#25 pulling sha256:eba798ac6e3530a4bc02df88fc99f6e4a1c721e9665d9742ec852f378f392e6a
#25 pulling sha256:141a3c18f61492441778289f6e7dde820c595b58f553e9f8296b382f17c463ec
#25 pulling sha256:53a19f6c217af491df6b64cafb721a5017cd42e3916c7119795a00abc31de4c7
#25 pulling sha256:141a3c18f61492441778289f6e7dde820c595b58f553e9f8296b382f17c463ec 0.3s done
#25 pulling sha256:53a19f6c217af491df6b64cafb721a5017cd42e3916c7119795a00abc31de4c7 0.3s done
#25 pulling sha256:eba798ac6e3530a4bc02df88fc99f6e4a1c721e9665d9742ec852f378f392e6a 0.4s done
#25 CACHED
#26 exporting to image
#26 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#26 exporting layers done
#26 writing image sha256:ac0373e5d992b47006e849a16101a5de8b426a0fee64eb64b107a6c2622f509e done
#26 naming to registry.internal.***.io:5000/e2e:9f3d85329e38618d79b125da4a49f2fda7fe2cda done
#26 DONE 0.1s
#27 exporting cache
#27 sha256:2700d4ef94dee473593c5c614b55b2dedcca7893909811a8f2b48291a1f581e4
#27 preparing build cache for export
cannot write /tmp/tmp_bc9imzl because server did not provide an image ID

[Madvinking]

any update on this?

[paolomainardi]

We are facing the same issue, right now apparently random, it happens with docker-compose + buildkit