Buildkit: [Dockerfile] BuildKit does not respect Docker Registry Mirrors defined in daemon.json
Hi,
I have been trying out the BuildKit feature built into Docker 18.06, but building with DOCKER_BUILDKIT=1 seems to cause my builds to fail since I am behind a firewall and need to access the docker hub registry through a local mirror.
/etc/docker/daemon.json:
{
"registry-mirrors": ["https://my-mirror.company.com"],
"experimental": true
}
Dockerfile:
FROM golang:alpine
RUN echo "test"
docker build -t test .
$ docker build .
[+] Building 0.6s (3/4)
=> local://context (.dockerignore) 0.3s
=> => transferring context: 2B 0.0s
=> local://dockerfile (Dockerfile) 0.2s
=> => transferring dockerfile: 96B 0.0s
=> ERROR docker-image://docker.io/library/golang:alpine 0.1s
=> => resolve docker.io/library/golang:alpine 0.1s
------
> docker-image://docker.io/library/golang:alpine:
------
failed to do request: Head https://registry-1.docker.io/v2/library/golang/manifests/alpine: unexpected EOF
tangtony
👍13
All 12 comments
cc @dmcgowan
tonistiigi
on 5 Sep 2018
I'm also seeing this issue. insecure-registries seems to be ignored as well.
adamsteinbok
on 6 Sep 2018
Is this supposed to be fixed in 18.09? Because I just updated to 18.09 and I'm getting the following error when I try to build now with a registry mirror:
#3 [internal] load metadata for docker.io/library/alpine:latest
#3 digest: sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6
#3 name: "[internal] load metadata for docker.io/library/alpine:latest"
#3 started: 2018-11-11 20:47:20.993963345 +0000 UTC
#3 completed: 2018-11-11 20:47:20.994574285 +0000 UTC
#3 duration: 610.94碌s
#3 error: "parse https://my-mirror.company.com%2F/v2/library/alpine/manifests/latest: invalid URL escape \"%2F\""
#4 [internal] load metadata for docker.io/library/golang:alpine
#4 digest: sha256:299327d28eff710219f2e24597cfa9b226e8b1b0dc90f9e2122573004cfe837f
#4 name: "[internal] load metadata for docker.io/library/golang:alpine"
#4 started: 2018-11-11 20:47:20.994231588 +0000 UTC
#4 completed: 2018-11-11 20:47:20.994427308 +0000 UTC
#4 duration: 195.72碌s
#4 error: "parse https://my-mirror.company.com%2F/v2/library/golang/manifests/alpine: invalid URL escape \"%2F\""
Edit: The registry mirrors seem to work as expected when configured via the daemon.json. But when configuring the registry mirrors via docker daemon arguments, the error above occurs.
tangtony
on 11 Nov 2018
👍5
I too have a problem with my private registry when buildkit is enabled in Docker 18.09.
This is my daemon.json :
{
"registry-mirrors": ["https://repo.mycompany.com"],
"insecure-registries" : ["repo.mycompany.com"],
"features":{
"buildkit": true
}
}
When trying to build a Dockerfile starting with :
# syntax = docker/dockerfile:experimental
FROM maven:3.5.3-slim
I am getting this (I had done "docker pull docker.io/docker/dockerfile:experimental" first):
me@ic:~/test$ docker build .
[+] Building 0.1s (6/11)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> ERROR [internal] load metadata for docker.io/library/maven:3.5.3-slim 0.0s
=> ERROR [1/6] FROM docker.io/library/maven:3.5.3-slim 0.0s
=> => resolve docker.io/library/maven:3.5.3-slim 0.0s
=> CACHED [internal] helper image for file operations 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 79B 0.0s
------
> [internal] load metadata for docker.io/library/maven:3.5.3-slim:
------
------
> [1/6] FROM docker.io/library/maven:3.5.3-slim:
------
failed to do request: Head https://repo.mycompany.com/v2/library/maven/manifests/3.5.3-slim: x509: certificate signed by unknown authority
If I set buildkit to false in daemon.json, everything works fine.
Platform : Ubuntu 18.04.1 LTS, docker-ce 18.09.0, build 4d60db4
lbndev
on 12 Nov 2018
^^ @AntaresS
tonistiigi
on 12 Nov 2018
This also means the GCP Container Registry's Docker Hub mirror cannot be used with buildkit: https://cloud.google.com/container-registry/docs/using-dockerhub-mirroring
Scenario
$ cat /etc/docker/daemon.json
{
"features": {"buildkit": true},
"debug": true,
"registry-mirrors": ["https://mirror.gcr.io"]
}
$ cat Dockerfile
FROM ubuntu:18.04
$ docker build .
[+] Building 0.5s (4/4) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 61B 0.0s
=> ERROR [internal] load metadata for docker.io/library/ubuntu:18.04 0.3s
=> ERROR [1/1] FROM docker.io/library/ubuntu:18.04 0.2s
=> => resolve docker.io/library/ubuntu:18.04 0.2s
------
> [internal] load metadata for docker.io/library/ubuntu:18.04:
------
------
> [1/1] FROM docker.io/library/ubuntu:18.04:
------
docker.io/library/ubuntu:18.04 not found
docker daemon logs:
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.788539962+01:00" level=debug msg="Calling GET /_ping"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789372744+01:00" level=debug msg="Calling POST /session"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789389343+01:00" level=debug msg="Calling POST /v1.39/build?buildargs=%7B%7D&buildid=0339f145e333056de3753e4c03e661987b94d15db2be555233bffa2a367dd041&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=%7B%7D&memory=0&memswap=0&networkmode=default&remote=client-session&rm=1&session=aye173ehx136zr83ojvc807lh&shmsize=0&target=&ulimits=null&version=2"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789498322+01:00" level=info msg="parsed scheme: \"\"" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789512022+01:00" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789613181+01:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{ 0 <nil>}]" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789657431+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789726700+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, CONNECTING" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789859839+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, READY" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.807385619+01:00" level=debug msg="reusing ref for local: sxtbvtno9sjb2fi3gsalzxu40"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.808593608+01:00" level=debug msg="diffcopy took: 1.05459ms"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.809481659+01:00" level=debug msg="reusing ref for local: y63o1okwyn5m255kfzv0ospr0"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.810173532+01:00" level=debug msg="diffcopy took: 605.114碌s"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.813437291+01:00" level=debug msg="saved sxtbvtno9sjb2fi3gsalzxu40 as local.sharedKey:dockerfile:Dockerfile:8521e1980bcd1479d3a309f95d273cb76bca41fcbe4b5da4a8e8760f345728ee"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.819195975+01:00" level=debug msg="saved y63o1okwyn5m255kfzv0ospr0 as local.sharedKey:context:.dockerignore:8521e1980bcd1479d3a309f95d273cb76bca41fcbe4b5da4a8e8760f345728ee"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.880379484+01:00" level=debug msg=resolving
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.880436003+01:00" level=debug msg="do request" request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD url="https://mirror.gcr.io/v2/library/ubuntu/manifests/18.04"
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.141929136+01:00" level=debug msg="fetch response received" response.headers="map[Content-Type:[application/json] Server:[Docker Registry] Cache-Control:[private] X-Frame-Options:[SAMEORIGIN] Alt-Svc:[quic=\":443\"; ma=2592000; v=\"44,43,39,35\"] Accept-Ranges:[none] Docker-Distribution-Api-Version:[registry/2.0] Date:[Mon, 14 Jan 2019 10:27:11 GMT] X-Xss-Protection:[1; mode=block] Vary:[Accept-Encoding]]" status="404 Not Found" url="https://mirror.gcr.io/v2/library/ubuntu/manifests/18.04"
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.142439491+01:00" level=debug msg=resolving
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.142480801+01:00" level=debug msg="do request" request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD url="https://mirror.gcr.io/v2/library/ubuntu/manifests/18.04"
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377198801+01:00" level=debug msg="fetch response received" response.headers="map[X-Frame-Options:[SAMEORIGIN] Vary:[Accept-Encoding] Docker-Distribution-Api-Version:[registry/2.0] Content-Type:[application/json] Date:[Mon, 14 Jan 2019 10:27:11 GMT] Server:[Docker Registry] Cache-Control:[private] X-Xss-Protection:[1; mode=block] Alt-Svc:[quic=\":443\"; ma=2592000; v=\"44,43,39,35\"] Accept-Ranges:[none]]" status="404 Not Found" url="https://mirror.gcr.io/v2/library/ubuntu/manifests/18.04"
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377730726+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, TRANSIENT_FAILURE" module=grpc
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377763406+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, CONNECTING" module=grpc
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377765596+01:00" level=warning msg="grpc: addrConn.createTransport failed to connect to { 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377807735+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, TRANSIENT_FAILURE" module=grpc
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.789751463+01:00" level=warning msg="grpc: addrConn.transportMonitor exits due to: context canceled" module=grpc
Actual behavior
The daemon seems to try to get image info 2 times from the same registry: the GCP mirror, which returns 404.
Expected behavior
The daemon should get image info from the GCP mirror first, and on 404 response try again on docker official registry.
Versions
$ cat /etc/os-release | grep VERSION
VERSION="18.04.1 LTS (Bionic Beaver)"
$ docker version
Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:49:01 2018
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:16:44 2018
OS/Arch: linux/amd64
Experimental: false
thomas-riccardi
on 14 Jan 2019
😕1
Tried with 18.09 and 19.03 with
cat /etc/docker/daemon.json
{
"features": {"buildkit": true},
"debug": true,
"registry-mirrors": ["https://mirror.gcr.io"]
}
and had no issues. Verified from the logs that mirror was being used.
There are some workarounds in 19.03 to avoid codepath for https://issuetracker.google.com/issues/123043691 but as it works in 18.09.4 as well then I'm not sure if there was some gcr side change.
~ # docker version
Client: Docker Engine - Community
Version: 18.09.4
API version: 1.39
Go version: go1.10.8
Git commit: d14af54
Built: Wed Mar 27 18:33:40 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.4
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: d14af54
Built: Wed Mar 27 18:41:10 2019
OS/Arch: linux/amd64
Experimental: false
I ran into this problem while testing the new buildkit UI on travis to measure how long each build step takes.
Travis uses the https://mirror.gcr.io mirror and upgrading to 19.03 on testing solved my problem.
Edit: No, it didn't. I cached the image while testing.
SuperSandro2000
on 19 May 2019
😕1
I use a local, HTTP, pull-through registry. I seem to have the same problem after having enabled buildkit.
Docker 18
[vagrant@docker-host exp]$ cat /etc/docker/daemon.json
{
"bip": "172.17.0.1/16",
"debug": false,
"dns": [
"8.8.8.8",
"10.80.1.10",
"10.80.1.11"
],
"experimental": true,
"features": {
"buildkit": true
},
"icc": false,
"init": true,
"insecure-registries": [
"192.168.56.1:5000",
"127.0.0.1:5000"
],
"live-restore": false,
"log-driver": "journald",
"registry-mirrors": [
"http://192.168.56.1:5000"
],
"storage-driver": "overlay2",
"userland-proxy": false
}
[vagrant@docker-host exp]$ docker version
Client:
Version: 18.09.8
API version: 1.39
Go version: go1.10.8
Git commit: 0dd43dd87f
Built: Wed Jul 17 17:40:31 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.8
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 0dd43dd
Built: Wed Jul 17 17:10:42 2019
OS/Arch: linux/amd64
Experimental: true
[vagrant@docker-host exp]$ cd /tmp/exp
[vagrant@docker-host exp]$ printf "FROM golang:alpine\nRUN echo 'test'" > Dockerfile
[vagrant@docker-host exp]$ docker build -t test .
[+] Building 0.1s (4/5)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 135B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> ERROR [internal] load metadata for docker.io/library/golang:alpine 0.0s
=> ERROR [1/2] FROM docker.io/library/golang:alpine 0.0s
=> => resolve docker.io/library/golang:alpine 0.0s
------
> [internal] load metadata for docker.io/library/golang:alpine:
------
------
> [1/2] FROM docker.io/library/golang:alpine:
------
failed to do request: Head https://192.168.56.1:5000/v2/library/golang/manifests/alpine: http: server gave HTTP response to HTTPS client
md5-50291f114f763ee549980076c0222274
[vagrant@docker-host exp]$ docker version
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:25:41 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:24:18 2019
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
[vagrant@docker-host exp]$ docker build -t test .
[+] Building 0.2s (4/5)
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 95B 0.1s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> ERROR [internal] load metadata for docker.io/library/golang:alpine 0.0s
=> ERROR [1/2] FROM docker.io/library/golang:alpine 0.0s
=> => resolve docker.io/library/golang:alpine 0.0s
------
> [internal] load metadata for docker.io/library/golang:alpine:
------
------
> [1/2] FROM docker.io/library/golang:alpine:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: failed to do request: Head https://192.168.56.1:5000/v2/library/golang/manifests/alpine: http: server gave HTTP response to HTTPS client
md5-fb4eab8432b07e9fd32215112d682e81
[vagrant@docker-host exp]$ sudo systemctl restart docker
[vagrant@docker-host exp]$ docker build -t test .
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM golang:alpine
alpine: Pulling from library/golang
89d9c30c1d48: Already exists
8ef94372a977: Extracting [================> ] 98.3kB/301.7kB
1ec62c064901: Download complete
339073ee4259: Downloading [=> ] 3.284MB/126.9MB
df12825c8c18: Download complete
```
jamiejackson
on 9 Dec 2019
👍3
Same exact issue. Using local Harbor repo. Reviewing logs on Harbor side shows the following. It seems like enabling BuildKit causes it to make some non-supported calls to the registry that leads to the failure...
2019/12/31 17:49:09 [D] [server.go:2774] | 10.1.131.206| 401 | 4.089799ms| match| GET /v2/ r:/v2/*
2019/12/31 17:49:10 [D] [server.go:2774] | 10.1.131.206| 200 | 24.498045ms| match| GET /service/token r:/service/token
2019/12/31 17:49:10 [D] [server.go:2774] | 10.1.131.206| 200 | 4.120089ms| match| HEAD /v2/ r:/v2/*
2019/12/31 17:49:11 [D] [server.go:2774] | 10.1.131.33| 200 | 1.799942ms| match| GET /api/ping r:/api/ping
2019/12/31 17:49:14 [D] [server.go:2774] | 10.1.131.206| 401 | 4.211535ms| match| GET /v2/ r:/v2/*
2019/12/31 17:49:14 [D] [server.go:2774] | 10.1.131.206| 200 | 64.820175ms| match| GET /service/token r:/service/token
2019/12/31 17:49:14 [D] [server.go:2774] | 10.1.131.206| 200 | 8.106133ms| match| GET /v2/ r:/v2/*
2019/12/31 17:49:15 [D] [server.go:2774] | 10.1.131.206| 401 | 3.63903ms| match| HEAD /v2/library/openjdk/manifests/8u232-jre-stretch r:/v2/*
2019/12/31 17:49:15 [C] [panic.go:522] the request url is /service/token
2019/12/31 17:49:15 [C] [panic.go:522] Handler crashed with error '_xsrf' argument missing from POST
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/runtime/panic.go:522
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/context/context.go:80
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/context/context.go:164
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/controller.go:649
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/router.go:788
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/net/http/server.go:2774
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/net/http/server.go:1878
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/runtime/asm_amd64.s:1337
2019/12/31 17:49:15 [D] [server.go:2774] | 10.1.131.206| 401 | 4.155078ms| match| HEAD /v2/library/openjdk/manifests/8u232-jre-stretch r:/v2/*
2019/12/31 17:49:15 [C] [panic.go:522] the request url is /service/token
2019/12/31 17:49:15 [C] [panic.go:522] Handler crashed with error '_xsrf' argument missing from POST
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/runtime/panic.go:522
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/context/context.go:80
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/context/context.go:164
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/controller.go:649
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/router.go:788
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/net/http/server.go:2774
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/net/http/server.go:1878
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/runtime/asm_amd64.s:1337
eroji
on 31 Dec 2019
I have the same problem when I use GCR as a registry mirror. After I went digging through the code I concluded that the problem is in Moby and not in BuildKit.
This already merged PR fixes the problem for me, so it should be resolved with the next release of Docker.
matevzmihalic
on 11 May 2020
tonistiigi
on 11 May 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
AkihiroSuda
路
3Comments
ImJasonH
路
5Comments
moikot
路
6Comments
alicefr
路
4Comments
daveisfera
路
4Comments
Most helpful comment
Is this supposed to be fixed in 18.09? Because I just updated to 18.09 and I'm getting the following error when I try to build now with a registry mirror:
Edit: The registry mirrors seem to work as expected when configured via the daemon.json. But when configuring the registry mirrors via docker daemon arguments, the error above occurs.