Buildkit: Error on cache query

[slamdev]

Image is failing to build when --cache-from flag is passed to docker build:

$ ls
Dockerfile  file.txt
$ cat Dockerfile
FROM nginx:alpine
COPY file.txt /opt/
$ DOCKER_BUILDKIT=1 docker build --cache-from nginx:alpine .
[+] Building 6.8s (6/7)
 => local://context (.dockerignore)                                                                                                                                                                                                      0.0s
 => => transferring context: 02B                                                                                                                                                                                                         0.0s
 => local://dockerfile (Dockerfile)                                                                                                                                                                                                      0.0s
 => => transferring dockerfile: 81B                                                                                                                                                                                                      0.0s
 => ERROR importing cache manifest from docker.io/library/nginx:alpine                                                                                                                                                                   6.8s
 => local://context                                                                                                                                                                                                                      0.0s
 => => transferring context: 29B                                                                                                                                                                                                         0.0s
 => docker-image://docker.io/library/nginx:alpine                                                                                                                                                                                        0.0s
 => => resolve docker.io/library/nginx:alpine                                                                                                                                                                                            0.0s
 => docker-image://docker.io/tonistiigi/copy:v0.1.3@sha256:e57a3b4d6240f55bac26b655d2cfb751f8b9412d6f7bb1f787e946391fb4b21b                                                                                                              0.0s
 => => resolve docker.io/tonistiigi/copy:v0.1.3@sha256:e57a3b4d6240f55bac26b655d2cfb751f8b9412d6f7bb1f787e946391fb4b21b                                                                                                                  0.0s
------
 > importing cache manifest from docker.io/library/nginx:alpine:
------
error on cache query: invalid build cache from docker.io/library/nginx:alpine

I am not sure should the issue be posted here or in https://github.com/docker/cli

[manishtomar]

Does this happen again? Maybe temporary glitch trying to get nginx:alpine manifest?

[tonistiigi]

The value of --cache-from has changed in buildkit and it now points to a special build cache manifest that buildkit can export. We have discussed inlining metadata for the image layers to the experted image config as well to keep some compatibility with current logic but it is not implemented yet.

@tiborvass I think we should at least suppress this error.

[RytisLT]

@tonistiigi in https://github.com/moby/buildkit/issues/62#issuecomment-313739986 you've mentioned that

buildkit already should support exporting cache to any registry

How can I achieve that? Pushing an image built with buildkit doesn't seem to do it. Is there and extra parameter for docker build to push cache to registry?

Edit: found that it's a current limitation of experimental implementation (https://github.com/moby/moby/pull/37151)

[nielskrijger]

Tried building an image with buildkit using --cache-from and noticed the same error as described in this issue.

If I understood correctly I was expecting that when the --cache-from image was built using buildkit it should work. Instead I noticed auth errors while using private registries:

$ docker build --cache-from gcr.io/my-project/my-service:builder  .

=> ERROR importing cache manifest from gcr.io/my-project/my-service:builder                                                                                                                                                                                                                                      2.2s
 => [internal] load build context                                                                                                                                                                                                                                                                                0.1s
 => => transferring context: 21.71kB                                                                                                                                                                                                                                                                             0.0s
 => [internal] helper image for file operations                                                                                                                                                                                                                                                                  0.1s
 => => resolve docker.io/tonistiigi/copy:v0.1.4@sha256:d9d49bedbbe2b27df88115e6aff7b9cd11ed2fbd8d9013f02d3da735c08c92e5                                                                                                                                                                                          0.1s
 => [builder 1/11] FROM docker.io/library/golang:1.11-alpine3.8                                                                                                                                                                                                                                                  0.0s
 => => resolve docker.io/library/golang:1.11-alpine3.8                                                                                                                                                                                                                                                           0.0s
 => CANCELED [builder 2/11] RUN apk add --no-cache git build-base openssh                                                                                                                                                                                                                                        0.7s
------
 > importing cache manifest from gcr.io/my-project/my-service:builder:
------
rpc error: code = Unknown desc = error on cache query: unexpected status code https://gcr.io/v2/my-project/my-service/manifests/builder: 401 Unauthorized

Tried a different repository (Gitlab), and got a 403 instead:

 => ERROR importing cache manifest from registry.gitlab.com/my-project/my-service:builder                                                                                                                                                                                                                         0.6s
 => [builder 1/11] FROM docker.io/library/golang:1.11-alpine3.8                                                                                                                                                                                                                                                  0.0s
 => => resolve docker.io/library/golang:1.11-alpine3.8                                                                                                                                                                                                                                                           0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                0.0s
 => => transferring context: 21.71kB                                                                                                                                                                                                                                                                             0.0s
 => [internal] helper image for file operations                                                                                                                                                                                                                                                                  0.1s
 => => resolve docker.io/tonistiigi/copy:v0.1.4@sha256:d9d49bedbbe2b27df88115e6aff7b9cd11ed2fbd8d9013f02d3da735c08c92e5                                                                                                                                                                                          0.1s
 => CANCELED [builder 2/11] RUN apk add --no-cache git build-base openssh                                                                                                                                                                                                                                        1.0s
------
 > importing cache manifest from registry.gitlab.com/my-project/my-service:builder:
------
rpc error: code = Unknown desc = error on cache query: failed to fetch anonymous token: unexpected status: 403 Forbidden

Pushing and pulling works just fine on both registries.

[RytisLT]

@nielskrijger the problem is that BuildKit uses a different cache format. To use --cache-from with BuildKit you need to export BuidKit cache first. Unfortunately, it's currently impossible with docker build (https://github.com/moby/moby/pull/37151). You would need to use BuidKit directly to export the cache.

[nielskrijger]

Thank you @RytisLT for explaining it again. It finally "clicked" what the your previous comment was about... and I am starting to understand the difficulty of the problem.

Is it possible to make the error more clear in these circumstances?

Existing v2 registries fetching cache manifests built with Buildkit are now returning (401, 403) it seems. Similarly the invalid build cache error found by @slamdev could be more clear; more akin to "this build cache format is not supported by ...".
Alternatively (or in addition to) I'm wondering if upon a push to an incompatible registry the user should be warned if certain features/data are lost.

[lig]

I'd expect this behavior to be implemented using a new command-line argument and the old --cache-from to have the same behavior as before. At least failing with not implemented error. This would make much more sense.
The current behavior just makes migration to buildkit a lot more complicated.

[tonistiigi]

This has been fixed by inline cache support https://github.com/moby/buildkit/pull/777 (in Docker 19.03)

[jeffreybrowning]

@tonistiigi can we post a link to relevant documentation to how this issue is fixed by #777

[zengqingfu1442]

I still come across this error when run CI pipeline in gitlab and gitlab registry. Is there any workaround for this error? @tonistiigi @RytisLT @manishtomar thanks

[javierguzman]

Hello @zengqingfu1442 I am getting this issue. Did you solve it at the end? Thank you in advance and regards.

p.s. I see it has been solve in version 19.03 ok I will give it a try