Buildkit: cache exporting / importing not working as expected - or just not working

[brad-jones]

Because --cache-from does not yet work in docker I figured I would have a go at using buildkit directly.

I am trying to setup CI/CD infrastructure that will use buildkit in a distributed fashion, across many hosts.
I note there are a few issues tagged with distributed, these are somewhat over my head at the moment, perhaps there is something in these that I have missed, my apologies if so.

Here is an example case: https://gist.github.com/brad-jones/536064fcc180b62be6febfd2fa843024

I feel like I am having similar frustrations to https://github.com/moby/buildkit/issues/699 but in my case I am using the Docker Registry and not ECR.

Actually in production we are using nexus but that has other issues (buildkit won't pull images from it for a start) and does not fully support the v2 docker registry spec, hence why my example case uses registry

[tonistiigi]

The first part of this is fixed with #730 (or just use 0.3.1). I need to validate if the rest of it now works as well or if there is another issue.

[tonistiigi]

Sorry for a delayed response.

You can find the fork of your gist (with modifications) in https://gist.github.com/tonistiigi/1ba21a5bae3b9fd895b28589e57175e9 . All outputs seem correct there to me now.

Two issues with the old one:

• 0.3.2 buildctl has an issue where it signals the daemon that it doesn't need cache for frontends. This is fixed in #730 . v0.3.1 also doesn't have an issue. I'll release v0.3.3 after that PR is merged.
• --import-cache only worked with LLB builds. On frontend builds, cache import can be controlled by the frontend itself. The dockerfile frontend accepts remote cache with --frontend-opt cache-from=ref. I admit this is very unintuitive and I even missed it myself on first look so I made a PR #731 that will automatically set the frontend-opt if --frontend is used from buildctl.

After these changes everything seemed ok to me.

Depending on the version of buildkit you use, you might notice a mismatch of the copy helper image when using docker build --cache-from=ref .. I'd suggest you using # syntax=docker/dockerfile to use the same external dockerfile implementation in both in case you need to use both docker and buildkit that have different versions. After that change all commands from your example script were also cached in docker 18.09 https://gist.github.com/tonistiigi/1ba21a5bae3b9fd895b28589e57175e9#file-_docker-out, even when built with older buildkit.

[brad-jones]

Thanks @tonistiigi looks great, will take a look shortly.

To confirm though, docker still doesn't have the ability to export an image that is compatible with --cache-from when DOCKER_BUILDKIT=1 but I can build an image with buildctl --export-cache ... and use that with docker --cache-from ...?

[tonistiigi]

To confirm though, docker still doesn't have the ability to export an image that is compatible with --cache-from when DOCKER_BUILDKIT=1 but I can build an image with buildctl --export-cache ... and use that with docker --cache-from ...?

Correct

[brad-jones]

Just tested out v0.3.3 working perfectly. Thanks alot @tonistiigi

[pikeas]

Thanks @tonistiigi looks great, will take a look shortly.

To confirm though, docker still doesn't have the ability to export an image that is compatible with --cache-from when DOCKER_BUILDKIT=1 but I can build an image with buildctl --export-cache ... and use that with docker --cache-from ...?

@tonistiigi Is there an open issue tracking this? It's surprising that DOCKER_BUILDKIT=1 cannot be used with --cache-from.

[AkihiroSuda]

@pikeas

tracked in https://github.com/moby/buildkit/issues/752

[virgile-hogman]

@tonistiigi @AkihiroSuda this is so confusing. If we can't use --cache-from with DOCKER_BUILDKIT=1 why is the option even allowed in the CLI?!

[virgile-hogman]

Or if this depends on what the image has been built with, there should be an error when trying to use an incompatible image. Otherwise how would the user make a difference between a format error or a simple cache miss?

[AkihiroSuda]

19.03 supports --import-cache

[eedwards-sk]

@AkihiroSuda

→ docker --version
Docker version 19.03.5, build 633a0ea

→ DOCKER_BUILDKIT=1 docker build --import-cache
unknown flag: --import-cache
See 'docker build --help'.

Is there a different argument I should be using?

[AkihiroSuda]

Sorry I meant --cache-from

[zengqingfu1442]

Can you give a example code of how to use buildctl --export-cache? I just found the doc https://github.com/moby/buildkit#--export-cache-options but don't know how to use BuildKit with --cache-from together.
@tonistiigi @brad-jones Thanks a lot!