Buildkit: %2F injected when resolving images with registry mirror configured

[ImJasonH]

I'm trying to run a build using Buildkit as described here, in a Google Cloud Shell environment where gcr.io is used as a Dockerhub mirror (documented here).

After upgrading to Docker 18.09.1, I run:

$ curl https://raw.githubusercontent.com/r2d4/mockerfile/master/Mockerfile.yaml | DOCKER_BUILDKIT=1 docker build -f - .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1268  100  1268    0     0   9311      0 --:--:-- --:--:-- --:--:--  9323
[+] Building 0.0s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                                              0.0s
 => => transferring dockerfile: 1.31kB                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                   0.0s
 => ERROR resolve image config for docker.io/r2d4/mocker:latest                                                                                   0.0s
------
 > resolve image config for docker.io/r2d4/mocker:latest:
------
parse https://us-mirror.gcr.io%2F/v2/r2d4/mocker/manifests/latest: invalid URL escape "%2F"

Note the mirror configuration seems to be prepending "https://us-mirror.gcr.io%2F" which results in a bad URL.

I can docker pull the image, in which case it seems the mirror is not consulted and the build proceeds slightly further, now breaking when pulling alpine:3.6 through the mirror:

$ curl https://raw.githubusercontent.com/r2d4/mockerfile/master/Mockerfile.yaml | DOCKER_BUILDKIT=1 docker build -f - .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1268  100  1268    0     0  14377      0 --:--:-- --:--:-- --:--:-- 14247
[+] Building 0.4s (7/30)
 => [internal] load build definition from Dockerfile                                                                                              0.0s
 => => transferring dockerfile: 1.31kB                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                   0.0s
 => resolve image config for docker.io/r2d4/mocker:latest                                                                                         0.0s
 => CACHED docker-image://docker.io/r2d4/mocker:latest                                                                                            0.0s
 => [internal] load Mockerfile from Dockerfile                                                                                                    0.0s
 => => transferring dockerfile: 1.31kB                                                                                                            0.0s
 => ERROR docker-image://docker.io/library/ubuntu:16.04                                                                                           0.0s
 => => resolve docker.io/library/ubuntu:16.04                                                                                                     0.0s
 => ERROR docker-image://docker.io/library/alpine:3.6                                                                                             0.0s
 => => resolve docker.io/library/alpine:3.6                                                                                                       0.0s
------
 > docker-image://docker.io/library/ubuntu:16.04:
------
------
 > docker-image://docker.io/library/alpine:3.6:
------
rpc error: code = Unknown desc = failed to resolve dockerfile: rpc error: code = Unknown desc = parse https://us-mirror.gcr.io%2F/v2/library/alpine/manifests/3.6: invalid URL escape "%2F"

I also have to pull ubuntu:16.04, but after that the build proceeds as expected. :tada:

It's entirely possible this is the result of a misconfiguration on Cloud Shell's part in which case I'll happily close this and file this against them.

/cc @r2d4

[tonistiigi]

whats the contents of your mirror config

[DerekOverlock]

@ImJasonH - I am also seeing this issue with a similar setup (my company's internal docker registry is configured as a registry mirror). Have you found a workaround?

For me, it's failing to pull the frontend override specified in my Dockerfile:
# syntax=docker/dockerfile:1.0-experimental

Error:

error: "parse https://docker-hub.<intranet-hostname>.com%2F/v2/docker/dockerfile/manifests/1.0-experimental: invalid URL escape \"%2F\""

[codestation]

Same bug when invoking docker build from a drone plugin with DOCKER_BUILDKIT=1.

The docker daemon is started like this

/usr/local/bin/dockerd --data-root /var/lib/docker --registry-mirror http://mirror:5000

And it fails with

error: "parse https://mirror:5000%2F/v2/library/ruby/manifests/2.3: invalid URL escape \"%2F\""

The docker client and daemon are both on 18.09.3.

[kubaj]

Same problem in the Google Kubernetes Engine with node image Container-Optimized-OS with containerd cos-69-10895-138-0-c123 which has Docker 18.09.0

[tonistiigi]

Seems that this error appears when --registry-mirror is used as a command line flag and not when it is defined in the daemon json. Could reproduce this way and will look into it.