Buildkit: Allow controlling cache mounts storage location

[tonistiigi]

related https://github.com/moby/moby/issues/14080

Allowing exporting contents of the type=cache mounts have been asked many times in different repositories and slack.

https://github.com/moby/buildkit/issues/1474
https://github.com/docker/buildx/issues/244

Regular remote instruction cache does not work for cache mounts that are not tracked by cache key and are just a location on disk that can be shared by multiple builds.

Currently, the best approach to maintain this cache between nodes is to do it as part of the build. https://github.com/docker/buildx/issues/244#issuecomment-602750160

I don't think we should try to combine cache mounts with the remote cache backends. Usually, cache mounts are for throwaway cache and restoring it would take a similar time to just recreating it.

What we could do is to allow users to control where the cache location is on disk, in case it is not on top of the snapshots.

We can introduce a cache mount backend concept behind a go interface that different implementation can implement.

Eg. for a Dockerfile like

RUN --mount=type=cache,target=/root/.cache,id=gocache go build ...

you could invoke a build with

docker build --cache-mount id=gocache,type=volume,volume=myvolume .

In that case, the cache could use a Docker volume as a backend. I guess good drivers would be volume in Docker and bind mount from the host for non-docker. If no --cache-mount is specified, the usual snapshotter-based location is used.

From the security perspective, BuildKit API is considered secure by default for the host, so I think this would require daemon side configuration to enable what paths can be bound.

Another complexity is buildx container driver as we can't easily add new mounts to a running container atm. Possible solutions are to force these paths to be set on buildx create or do some hackery with mount propagation.

[myitcv]

Just to clarify whether the scope of this proposal covers a use case I have.

Go has content-addressed build and module caches defined via go env GOCACHE and, as of 1.15, go env GOMODCACHE (which was (go env GOPATH)[0]/pkg/mod in previous go versions).

Have read the documentation at https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md, it does not appear possible to delegate a cache to a host directory. Therefore a buildkit Go build and module cache will likely largely duplicate the build and module caches that exists on the host.

However, this proposal seems to be heading in the direction of sharing such a cache:

In that case, the cache could use a Docker volume as a backend

Under this proposal, can I confirm, would it be possible to delegate these caches to host directories?

I note a couple of requirements:

• in the case of Go, the location of these directories is not guaranteed to be given by an environment variable, rather the output of go env GOCACHE and go env GOMODCACHE is definitive. Whilst not the end of the world if environment variables were the only way of passing values, support the output of go env GOCACHE and go env GOMODCACHE would be even better
• the UID and GID of writes to the cache should default to be that of the caller

Apologies if this covers old ground (covered elsewhere); I'm rather new to the buildkit approach but happened upon this issue and it sounded like exactly the thing I was after.

Many thanks

[morlay]

hope this feature could land asap.

it will be useful for ci caching. https://github.com/moby/buildkit/issues/1673#issuecomment-698348524

for host directories, could be hacking by

# ensure host path
$ mkdir -p /tmp/gocache

# create volume
$ docker volume create --driver local \
      --opt type=none \
      --opt device=/tmp/gocache \   
      --opt o=bind \
      myvolume 

$ docker run -it --volume myvolume:/go/pkg/mod busybox touch /go/pkg/mod/test.txt
# test.txt will be created under host dir /tmp/gocache/

# maybe work
$ docker buildx build --cache-mount id=gocache,type=volume,volume=myvolume .

@tonistiigi should we control the mount target too?
--cache-mount id=gocache,type=volume,volume=myvolume,target=/go/pkg/mod

[hansbogert]

Does the proposed solution cater to a scenario of a buildserver which uses docker-in-docker? I'm not sure tbh.

[ties-s]

Any news on this? This shouldn’t be a really big change, right?

[rhyek]

This would solve my life.

[summera]

Would love to see this. Would be a huge win for speeding up builds on CI

[Mahoney]

This would be brilliant for build systems like Gradle and Maven building on e.g. GitHub Actions.

They typically download all their dependencies to a cache dir. It's hard to benefit from layer caching - dependencies can be expressed in multiple files in a nested folder structure, so to avoid a maintenance nightmare it's generally necessary for the Dockerfile to do a COPY . . before running the Gradle / Maven command that downloads the dependencies, which in turn means the layer cache is invalid nearly every time. Downloading the transitive dependencies is very chatty, can be hundreds of HTTP requests, so it's well worth turning into a single tar ball.

I really want to use the same Dockerfile to build locally and on CI, which I think means I don't want to use the strategy suggested in https://github.com/docker/buildx/issues/244#issuecomment-602750160 of loading & exporting the cache to named locations as commands in the Dockerfile - it might work in CI but would be much less efficient building locally, as well as adding a lot of noise to the Dockerfile.

I'm currently caching the whole of the /var/lib/docker dir (!) and restarting the docker service on the GitHub Action runner, which is also pretty slow and expensive, and generally not a great idea!

I'm guessing it wouldn't be a great place for a buildx AND go newbie to start contributing, though...