Buildx: [bug] Error: failed to solve: rpc error: code = Unknown desc = content digest sha256:...: not found
Summary
This error occurs randomly, and causes a build to fail when otherwise it would have succeeded. On re-running the exact same build, the error goes away.
Environment
docker-buildx version: standalone binary downloaded from https://github.com/docker/buildx/releases/download/v0.2.2/buildx-v0.2.2.linux-amd64 (sha256 5c31e171440213bb7fffd023611b1aaa7e0498162c54cb708c2a8abe3679717e)
Docker engine version:
Server: Docker Engine - Community
Engine:
Version: 18.09.6
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 481bc77
Built: Sat May 4 01:59:36 2019
OS/Arch: linux/amd64
Experimental: false
How to reproduce
Unfortunately, I have not found a way to reproduce it reliably. However it does not seem to be related to the contents of the Dockerfile, since I have reproduced it (accidentally) with two completely unrelated Dockerfiles.
Based on my understanding of the error message, it appears that --load is required to reproduce.
Here is the output of the most recent occurrence:
+ /home/sh/bin/docker-buildx build --load --build-arg user=sh --build-arg uid=1002 --build-arg docker_guid=999 -t dev -
[+] Building 11.1s (54/54) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 2.30kB 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.4s
=> [1/49] FROM docker.io/library/alpine@sha256:ca1c944a4f8486a153024d9965aafbe24f5723c1d5c02f 0.0s
=> => resolve docker.io/library/alpine@sha256:ca1c944a4f8486a153024d9965aafbe24f5723c1d5c02f4 0.0s
=> CACHED [2/49] RUN apk update 0.0s
=> CACHED [3/49] RUN apk add openssh 0.0s
=> CACHED [4/49] RUN apk add bash 0.0s
=> CACHED [5/49] RUN apk add bind-tools 0.0s
=> CACHED [6/49] RUN apk add curl 0.0s
=> CACHED [7/49] RUN apk add docker 0.0s
=> CACHED [8/49] RUN apk add g++ 0.0s
=> CACHED [9/49] RUN apk add gcc 0.0s
=> CACHED [10/49] RUN apk add git 0.0s
=> CACHED [11/49] RUN apk add git-perl 0.0s
=> CACHED [12/49] RUN apk add make 0.0s
=> CACHED [13/49] RUN apk add python 0.0s
=> CACHED [14/49] RUN apk add openssl-dev 0.0s
=> CACHED [15/49] RUN apk add vim 0.0s
=> CACHED [16/49] RUN apk add py-pip 0.0s
=> CACHED [17/49] RUN apk add file 0.0s
=> CACHED [18/49] RUN apk add groff 0.0s
=> CACHED [19/49] RUN apk add jq 0.0s
=> CACHED [20/49] RUN apk add man 0.0s
=> CACHED [21/49] RUN cd /tmp && git clone https://github.com/AGWA/git-crypt && cd git-crypt 0.0s
=> CACHED [22/49] RUN apk add go 0.0s
=> CACHED [23/49] RUN apk add coreutils 0.0s
=> CACHED [24/49] RUN apk add python2-dev 0.0s
=> CACHED [25/49] RUN apk add python3-dev 0.0s
=> CACHED [26/49] RUN apk add tar 0.0s
=> CACHED [27/49] RUN apk add vim 0.0s
=> CACHED [28/49] RUN apk add rsync 0.0s
=> CACHED [29/49] RUN apk add less 0.0s
=> CACHED [30/49] RUN pip install awscli 0.0s
=> CACHED [31/49] RUN curl --silent --location "https://github.com/weaveworks/eksctl/releases 0.0s
=> CACHED [32/49] RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google- 0.0s
=> CACHED [33/49] RUN curl -L -o /usr/local/bin/kubectl https://storage.googleapis.com/kubern 0.0s
=> CACHED [34/49] RUN curl -L -o /usr/local/bin/kustomize https://github.com/kubernetes-sigs 0.0s
=> CACHED [35/49] RUN apk add ruby 0.0s
=> CACHED [36/49] RUN apk add ruby-dev 0.0s
=> CACHED [37/49] RUN gem install bigdecimal --no-ri --no-rdoc 0.0s
=> CACHED [38/49] RUN gem install kubernetes-deploy --no-ri --no-rdoc 0.0s
=> CACHED [39/49] RUN apk add npm 0.0s
=> CACHED [40/49] RUN npm config set unsafe-perm true 0.0s
=> CACHED [41/49] RUN npm install -g yarn 0.0s
=> CACHED [42/49] RUN npm install -g netlify-cli 0.0s
=> CACHED [43/49] RUN apk add libffi-dev 0.0s
=> CACHED [44/49] RUN pip install docker-compose 0.0s
=> CACHED [45/49] RUN apk add shadow sudo 0.0s
=> CACHED [46/49] RUN echo '%wheel ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers 0.0s
=> CACHED [47/49] RUN useradd -G docker,wheel -m -s /bin/bash -u 1002 sh 0.0s
=> CACHED [48/49] RUN groupmod -o -g 999 docker 0.0s
=> CACHED [49/49] WORKDIR /home/sh 0.0s
=> ERROR exporting to oci image format 10.5s
=> => exporting layers 0.6s
=> => exporting manifest sha256:97c8dedb84183d0d87b6c930b12e1b9396bb81fd9c6587fe2fbb9ae092d30 0.0s
=> => exporting config sha256:ef15b3f6a374fd494a0a586f7b33ac9435953460776e268b20f2eee5f14def6 0.0s
=> => sending tarball 9.6s
=> importing to docker 0.0s
------
> exporting to oci image format:
------
Error: failed to solve: rpc error: code = Unknown desc = content digest sha256:ef15b3f6a374fd494a0a586f7b33ac9435953460776e268b20f2eee5f14def65: not found
Usage:
/home/sh/bin/docker-buildx build [OPTIONS] PATH | URL | - [flags]
Aliases:
build, b
Flags:
--add-host strings Add a custom host-to-IP mapping (host:ip)
--build-arg stringArray Set build-time variables
--cache-from stringArray External cache sources (eg. user/app:cache, type=local,src=path/to/dir)
--cache-to stringArray Cache export destinations (eg. user/app:cache, type=local,dest=path/to/dir)
-f, --file string Name of the Dockerfile (Default is 'PATH/Dockerfile')
-h, --help help for build
--iidfile string Write the image ID to the file
--label stringArray Set metadata for an image
--load Shorthand for --output=type=docker
--network string Set the networking mode for the RUN instructions during build (default "default")
--no-cache Do not use cache when building the image
-o, --output stringArray Output destination (format: type=local,dest=path)
--platform stringArray Set target platform for build
--progress string Set type of progress output (auto, plain, tty). Use plain to show container output (default "auto")
--pull Always attempt to pull a newer version of the image
--push Shorthand for --output=type=registry
--secret stringArray Secret file to expose to the build: id=mysecret,src=/local/secret
--ssh stringArray SSH agent socket or keys to expose to the build (format: default|<id>[=<socket>|<key>[,<key>]])
-t, --tag stringArray Name and optionally a tag in the 'name:tag' format
--target string Set the target build stage to build.
shykes
All 13 comments
I think I see how this could happen if there are parallel builds that end up building the same image and one ends up finishing with this error during load. Is this case possible in your case?
If not then I need to enable debug logs in buildx cause without them it's very hard to guess what actually happened.
tonistiigi
on 8 Jul 2019
With #102 you can create the builder instance with docker buildx create --buildkitd-flags '--debug' and after this happens use docker logs <buildkit_container> to get the debug logs for the daemon.
tonistiigi
on 9 Jul 2019
I think I see how this could happen if there are parallel builds that end up building the same image and one ends up finishing with this error during load. Is this case possible in your case?
Yes, that is very likely. Both environments where the issue occurred match your description. In fact I am also encountering another issue (issue coming soon) which is definitely related to concurrency.
shykes
on 11 Jul 2019
Thank you!
shykes
on 15 Jul 2019
I think this should be reopen.
I'm having this error quite often when I build. Usually, I just re-run the command until it stop failing. (2-3 times in average).
I'm building for 3 platforms in parallel.
Docker version : 19.03.8, build afacb8b
kegi
on 6 May 2020
Same error here:
Using docker Docker version 19.03.8, build afacb8b7f0
On a Raspberry Pi with arm32v7 architecture
MarkCarbonell98
on 13 May 2020
I think this should be reopen.
I'm having this error quite often when I build. Usually, I just re-run the command until it stop failing. (2-3 times in average).
I'm building for 2 platforms in parallel.
Got the same here https://github.com/eclipse/che-dashboard/actions
skabashnyuk
on 20 May 2020
Please open a new issue with reproduction steps if you think there is still a way to hit this.
tonistiigi
on 20 May 2020
According to https://stackoverflow.com/questions/59868448/rpc-error-code-unknown-desc-failed-to-build-llb this is a resource issue. Indeed, adding another processor and a Gig seems to be working for me. Obviously a better error would be appropriate, at the least.
shaunc
on 19 Jul 2020
Just ran into this as well, cross building locally with qemu the docker file was
FROM debian:buster-slim
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV PATH /opt/conda/bin:$PATH
RUN apt-get update --fix-missing && \
apt-get install -y wget bzip2 ca-certificates libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion && \
apt-get clean
COPY setup.sh /setup.sh
RUN /setup.sh
CMD [ "/bin/bash" ]
and the setup file
#!/bin/bash
set -ex
export arch=$(uname -m)
if [ "$arch" == "aarm64" ]; then
arch="arm64";
fi
wget --quiet https://github.com/conda-forge/miniforge/releases/download/4.8.5-1/Miniforge3-4.8.5-1-Linux-${arch}.sh -O ~/miniforge.sh
chmod a+x ~/miniforge.sh
~/miniforge.sh -b -p /opt/conda
/opt/conda/bin/conda clean -tipsy
ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc
echo "conda activate base" >> ~/.bashrc
find /opt/conda/ -follow -type f -name '*.a' -delete
find /opt/conda/ -follow -type f -name '*.js.map' -delete
/opt/conda/bin/conda clean -afy
My build command was: docker buildx build -t holdenk/miniforge3:v0 . --platform linux/amd64,linux/arm64 --push
re-running seems to solve the issue though.
holdenk
on 3 Oct 2020
I am getting the same issue too: It happened for arm64
huguesBouvier
on 13 Feb 2021
Seeing this as well, building for amd64 and arm64 in parallel
`
ocker versio
2 | Client: Docker Engine - Community
3 | Version: 20.10.3
4 | API version: 1.41
5 | Go version: go1.13.15
6 | Git commit: 48d30b5
7 | Built: Fri Jan 29 14:28:23 2021
8 | OS/Arch: linux/amd64
9 | Context: default
10 | Experimental: true
11 | 聽
12 | Server: Docker Engine - Community
13 | Engine:
14 | Version: 20.10.3
15 | API version: 1.41 (minimum version 1.12)
16 | Go version: go1.13.15
17 | Git commit: 46229ca
18 | Built: Fri Jan 29 14:31:57 2021
19 | OS/Arch: linux/amd64
20 | Experimental: true
21 | containerd:
22 | Version: v1.4.3
23 | GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
24 | runc:
25 | Version: 1.0.0-rc92
26 | GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
27 | docker-init:
28 | Version: 0.19.0
29 | GitCommit: de40ad0
30 | + docker buildx version
31 | github.com/docker/buildx v0.5.1 11057da37336192bfc57d81e02359ba7ba848e4a
`
In my case, this is failing as a part of a drone build running inside a kubernetes cluster.
andrewthetechie
on 14 Feb 2021
This issue has been closed. If you are seeing something similar create a new issue with a reproducer and full version infos.
tonistiigi
on 14 Feb 2021
Related issues
iav
路
7Comments
carlosedp
路
12Comments
richarvey
路
13Comments
no1melman
路
4Comments
securisec
路
9Comments
Most helpful comment
I am getting the same issue too: It happened for arm64