Buildkit: mount=type=cache more in-depth explanation?

[peci1]

Hi, I'm trying to use cache mounts to speed up things like apt install, pip install or ccache when rebuilding my containers.

For each of these, I prefix each RUN command using apt/pip/ccache... with things like:

RUN --mount=type=cache,target=/var/cache/apt,rw --mount=type=cache,target=/var/lib/apt,rw  \

That seems to work sometimes, but I found out I can't understand the cache invalidation rules. The cache just from time to time disappears and I have to download all the cached packages again. I made explicit effort to remove all apt clean commands from the dockerfile, so the cache is not deleted programatically. And it also happens to CCache, which is not autodeleted.

Could somebody please explain how does it work and when exactly can the cache be reused, when (if at all) is it discarded, where is it saved...? What is the exact effect of the id parameter?

I also found out that if I want to create a cache directory for a non-root user, I can create it with cache mount option uid=1000. But if I use uid=1000,gid=1000, the folder gets ownership root:root, which is really weird. Can it be connected with me not using id and setting some caches with uid=0 and some with uid=1000? Does this create some kind of conflict?

And is there actually a good reason for setting the mode to anything else than 777? If it's only used during build, I don't get why anybody would care about permissions...

Thank you for helping.

Maybe some of the answers could be added to https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md ?

[peci1]

And is it affected by the --no-cache build option?

[kennethtxytqw]

If I may hijack this, I like to ask if RUN --mount=type=cache can use the cache image in --cache-from ?

[morlay]

confused for me too.

--cache-from and --cache-to do nothing for --mount=type=cache https://github.com/docker/buildx/issues/399

--mount=type=cache cached dir in buildkit container /var/lib/buildkit/ until gc cleanup.

[FernandoMiguel]

I have quite a few cache examples here https://github.com/FernandoMiguel/Buildkit

I suspect your issue is GC which by default is very small.
also --no-cache will not use cache for that run

[morlay]

@FernandoMiguel
use remote buildkit could resolve by increasing gc limits. i setup it for my private projects.

but some ci like Travis or GitHub Workflows

buildkit is always created. --mount=type=cache cached files in container, will be lost.

[FernandoMiguel]

Yes for any ephemeral host, cache is lost.
That is expected and the correct behaviour.

You can use cache-from to pull layers from a image repo in that case

[morlay]

close https://github.com/docker/buildx/issues/399 and discussing here.

@FernandoMiguel

any suggestion in this case?.
i need cache /go/pkg/mod for multi workflows in build stage.

but the --cache-to couldn't expose the cached files (/var/lib/buildkit) to host.
and buildkit recreated when each workflow starts, all cached files in /var/lib/buildkit losts

buildx not provide way to mount host path for /var/lib/buildkit when create buildkit

[FernandoMiguel]

No idea what your Dockerfile looks like or what ci you are using
That makes all the difference
Some ci will allow you to cache host data

[morlay]

here example

workflow.yml https://github.com/querycap/istio/blob/master/.github/workflows/istio-pilot.yml
Dockerfile https://github.com/querycap/istio/blob/master/build/istio/Dockerfile.pilot

and the logs
https://pipelines.actions.githubusercontent.com/qcc22rpVKD7YzUzpABpz6Nbu6TDqQ0MFAEbSsnK9GDsCU4Hq5g/_apis/pipelines/1/runs/325/signedlogcontent/3?urlExpires=2020-09-24T13%3A32%3A15.1287086Z&urlSigningMethod=HMACV1&urlSignature=MDAD0FQhxCis3TFZLKqjsdmvB2m0iiCBWv6G9FyDOGs%3D

--cache-from --cache-to host path /tmp/.buildx/cache

--mount=type=cache not working

i understands how --mount=type=cache work.

i have no idea to cache files in buildkit container /var/lib/buildkit with docker buildx directly.

[FernandoMiguel]

Type cache won't obviously work on github actions since every run is done a new host.
You can run your own host and control that, or hack a way to store the cache (github limits are too low for any real use)

[morlay]

@FernandoMiguel
mount host path like /tmp/buildkit to buildkit to /var/lib/buildkit, then i add /tmp/buildkit to actions cache.
could this be possible?

[FernandoMiguel]

What

[morlay]

docker buildx create --use --name localbuild
docker buildx inspect localbuild --bootstrap

# recreate buildkit with host path
docker rm -f buildx_buildkit_localbuild0
docker run --privileged -d --name=buildx_buildkit_localbuild0 -v=/tmp/buildkit:/var/lib/buildkit moby/buildkit:buildx-stable-1         

now /tmp/buildkit in host contains buildkit files /var/lib/buildkit
i think i could cache this folder /tmp/buildkit

@FernandoMiguel it's not work with lots of permission error for whole path.

hope --cache-from --cache-to could support --mount=cache.

or add a option to path to assign the snapshots host path --mount=type=cache,path=/tmp/gomod,target=/go/pkg/mod

or replace the snapshot id of dirname with the id of --mount=type=cache,id=gomod,target=/go/pkg/mod
/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/go/pkg/mod
/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/gomod/go/pkg/mod

[FernandoMiguel]

/tmp is a special folder and may have weird permissions

[morlay]

github actions use a non-root user, but /var/lib/buildkit/runc-overlayfs only for root user.

and /var/lib/buildkit/runc-overlayfs always in changing. i give up to use this way.
need find other hacks.