Buildkit: Build fails when FROM image isn't cached

Created on 22 Nov 2019 · 16Comments · Source: moby/buildkit

My dockerfile has a first line like:

FROM golang:1.13.4 as binary_tools_context

When building this, I get the following error:

$ docker buildx build .
[+] Building 1.8s (27/176)
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 19.49kB                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/library/golang:1.13.4                                                             0.3s
 => [internal] load metadata for docker.io/library/ubuntu:bionic                                                                   0.0s
 => CACHED [clang_context 1/6] FROM docker.io/library/ubuntu:bionic                                                                0.0s
 => [internal] load build context                                                                                                  0.1s
 => => transferring context: 4.77kB                                                                                                0.0s
 => https://nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz                                                        1.0s
 => CANCELED https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-265.0.0-linux-x86_64.tar.gz               1.0s
 => CANCELED https://nodejs.org/download/release/v12.8.0/node-v12.8.0-linux-x64.tar.gz                                             1.0s
 => CANCELED https://github.com/ncopa/su-exec/archive/v0.2.tar.gz                                                                  0.1s
 => CANCELED https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.gz                                                            0.8s
 => CANCELED https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_am  0.1s
 => CANCELED [ruby_tools_context 2/14] RUN apt-get update && apt-get install -y --no-install-recommends     build-essential     c  1.0s
 => CANCELED https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kubectl                             0.1s
 => CANCELED https://github.com/mikefarah/yq/releases/download/2.4.0/yq_linux_amd64                                                0.1s
 => CANCELED https://download.docker.com/linux/ubuntu/gpg                                                                          0.1s
 => CANCELED https://storage.googleapis.com/kubernetes-helm/helm-v2.16.1-linux-amd64.tar.gz                                        0.1s
 => CANCELED [base_os_context 2/19] RUN apt-get update && apt-get install -y --no-install-recommends     apt-transport-https       0.8s
 => CANCELED [python_context 2/8] RUN apt-get update && apt-get install -y --no-install-recommends     ca-certificates     curl    1.1s
 => CANCELED https://github.com/gohugoio/hugo/releases/download/v0.58.2/hugo_0.58.2_Linux-64bit.tar.gz                             0.1s
 => CANCELED https://github.com/hadolint/hadolint/releases/download/v1.17.2/hadolint-Linux-x86_64                                  0.1s
 => CANCELED https://storage.googleapis.com/shellcheck/shellcheck-v0.7.0.linux.x86_64.tar.xz                                       0.1s
 => CANCELED https://github.com/upx/upx/releases/download/v3.95/upx-3.95-amd64_linux.tar.xz                                        0.1s
 => CANCELED [bazel_context 2/5] RUN apt-get update && apt-get install -y --no-install-recommends     wget     ca-certificates  &  1.2s
 => CANCELED [clang_context 2/6] RUN apt-get update && apt-get install -y --no-install-recommends     xz-utils     wget     ca-ce  1.4s
 => CANCELED https://github.com/protocolbuffers/protobuf/releases/download/v3.9.2/protoc-3.9.2-linux-x86_64.zip                    0.1s
 => ERROR [binary_tools_context 1/77] FROM docker.io/library/golang:1.13.4                                                         0.1s
 => => resolve docker.io/library/golang:1.13.4                                                                                     0.1s
------
 > [internal] load metadata for docker.io/library/golang:1.13.4:
------
------
 > [binary_tools_context 1/77] FROM docker.io/library/golang:1.13.4:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: docker.io/library/golang:1.13.4 not found

If I docker pull docker.io/library/golang:1.13.4 everything works

Any suggestions? Is this user error? I assume not since it works with standard docker build but maybe I missed some docs around this

Source

howardjohn

Most helpful comment

I'm still having issues with this as well, I had to disable buildkit for one of my docker builds. Maybe this should be reopened?

https://github.com/moby/buildkit/issues/1142#issuecomment-696987398

caleb15 on 22 Sep 2020

👍13 👀3

All 16 comments

Is this error persistent or did you just get it once?

tonistiigi on 25 Nov 2019

@tonistiigi

Here is what I did just now:

docker buildx build . -> works (I have the image cached)
docker rmi golang:1.13.4
docker buildx build . -> fails
docker buildx build . -> fails
docker buildx build . -> fails
docker pull golang:1.13.4
docker buildx build . -> passes

So seems to be persistent

howardjohn on 25 Nov 2019

👍2

Please post docker info docker version and dockerd daemon logs (enable debug if needed before).

tonistiigi on 25 Nov 2019

I made a minimal docker file now. Adding more info

$ cat Dockerfile
FROM golang:1.13.4
$ docker buildx build .
[+] Building 0.4s (4/4) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                           0.0s
 => => transferring dockerfile: 31B                                                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                                                0.0s
 => ERROR [internal] load metadata for docker.io/library/golang:1.13.4                                                                                                                                         0.2s
 => ERROR [1/1] FROM docker.io/library/golang:1.13.4                                                                                                                                                           0.2s
 => => resolve docker.io/library/golang:1.13.4                                                                                                                                                                 0.2s
------
 > [internal] load metadata for docker.io/library/golang:1.13.4:
------
------
 > [1/1] FROM docker.io/library/golang:1.13.4:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: docker.io/library/golang:1.13.4 not found
$ docker version
Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf583a
 Built:             Fri Oct 18 15:52:34 2019
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf583a
  Built:            Fri Oct 18 15:51:05 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
$ docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)

Server:
 Containers: 41
  Running: 1
  Paused: 0
  Stopped: 40
 Images: 1172
 Server Version: 19.03.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.67-2rodete2-amd64
 Operating System: Debian GNU/Linux bullseye/sid
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 62.64GiB
 Name: howardjohn.svl.corp.google.com
 ID: A757:6JXI:57IH:OP4C:LWX7:R2OK:ZYYG:GIQK:ARZ4:Q34S:7FUG:A4O5
 Docker Root Dir: /usr/local/google/docker
 Debug Mode: true
  File Descriptors: 50
  Goroutines: 81
  System Time: 2019-11-25T10:30:46.199097216-08:00
  EventsListeners: 0
 Username: howardjohn
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://mirror.gcr.io/
 Live Restore Enabled: false

WARNING: No swap limit support

Might have something to do with

 Registry Mirrors:
  https://mirror.gcr.io/

Will look into getting the logs

howardjohn on 25 Nov 2019

This seems to be caused by the mirror being configured while the mirror doesn't actually contain the image you are trying to use. Note that this is different from the behavior of the mirror mode in the upstream registry where it proxies to the origin registry if no local data is found. The capability to request from other registries if mirrors return errors was only added to containerd very recently and buildkit/docker has not been updated to the new logic yet. If you would like to help, the change would be to move from using Host callback to Hosts in https://github.com/containerd/containerd/blob/394db03f1531bfaa7018ecc470f29930aa30590f/remotes/docker/resolver.go#L90-L112

tonistiigi on 25 Nov 2019

For those still having issues with this, we configured a global env var: DOCKER_BUILDKIT=0 as temporary workaround. Note that this does disable buildkit as a whole, but this was pretty much the only thing which worked for us.

mathieudevos on 5 Aug 2020

👍11 👎1

I'm still having issues with this as well, I had to disable buildkit for one of my docker builds. Maybe this should be reopened?

https://github.com/moby/buildkit/issues/1142#issuecomment-696987398

caleb15 on 22 Sep 2020

👍13 👀3

use docker login command and then provide user login account detail once login success then try again docker build command
It will work.

Lakshpiya on 27 Dec 2020

👎5

This should be fixed in Buildkit v0.7+ / Docker 20.10 and iirc from some trace I saw it was also fixed on gcr side. If you still see this open a new issue with reproduction steps.

tonistiigi on 28 Dec 2020

@Lakshpiya Creating an account / signing in does not help.

I experience the same issue. I have to run an initial build with DOCKER_BUILDKIT=0. Follow-up runs work fine with DOCKER_BUILDKIT=1. Seems to fail downloading the image with buildkit.

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.5.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 22
 Server Version: 20.10.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.128-microsoft-standard
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 12.35GiB
 Name: docker-desktop
 ID: xxxx
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 43
  Goroutines: 50
  System Time: 2021-01-08T16:46:21.9770826Z
  EventsListeners: 4
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

black-snow on 8 Jan 2021

👍4

Same here. I got this issue after pruning and was unable to recover. Only DOCKER_BUILDKIT=0 docker build works.

moltar on 22 Mar 2021

👍5 👀1

This issue seems to be returned. Same here.

felipecrs on 26 Mar 2021

👍5

Same here.
Docker version 20.10.5, build 55c4c88 and had to use the DOCKER_BUILDKIT=0.

StefanoMantero on 27 Mar 2021

👍1

I'm using the latest version of buildx, but I don't know which version of buildkit is being used. Is there any way to check the version of buildkit being used?

felipecrs on 27 Mar 2021

@felipecrs Running docker info provides the version number for BuildKit for Docker in the current environment. Look for the line containing buildx.