Kaniko: Mount cache directory inside build container

[urbaniak]

Trying to figure out some way of sharing cache between builds, though about mounting some directory like /cache inside the build container, so we can then have a shared cache there for things like pip, npm or cargo.

Will be possible to implement something like that?

[tejal29]

We do have a cache warmer feature for caching base layers.
https://github.com/GoogleContainerTools/kaniko#caching

The intermediate layers are caches remotely. Are you looking for caching layers locally?

Would that help?

[urbaniak]

Layer cache is invalidated on any change so wanted to mount shared directory and keep for example python pip cache there, so it doesn't require to fetch that from network even one dependency has changed.

Something like -v for docker to mount a directory/image (which is also not supported for docker build)

[cvgw]

@urbaniak going on the example of pip caching (which I don't know a ton about) I'm assuming pip looks for certain directories to check if a cache already exists. I would imagine you could just mount a volume at that directory into the kaniko docker container.

IIRC kaniko has some special handling for mounted volumes so I don't think it would cause issues for the image build if those cache files aren't directly referenced by the docker build, but I'm not positive.

In any case, I see no reason why it shouldn't work so if it doesn't we can certainly look into it

[victornoel]

I second this need, what you explained @cvgw is correct:

• pip, maven, npm, yarn and so on have cache directories for downloaded packages
• usually one would instruct the CI to cache such directory so that it can be reused between builds

When using kaniko, the build itself has no access to the host filesystem, so it would be neat to be able to instruct kaniko to mount a certain directory into the build container so that pip and co can take advantage of it.

[glen-84]

What about supporting RUN --mount=type=cache?

[victornoel]

@glen-84 I'm not exactly sure how this feature interact with the host filesystem? Because the use case I had in mind take advantage of the fact that the host filesystem (i.e., the CI machine) will have a directory with the cache available. I'm not sure if --mount=type=cache allows for this, does it?

[glen-84]

@victornoel,

If I understand correctly, the Docker engine on the host would manage the directory (sort of like a named volume). If you follow the link, they show an example of caching Go packages.

See also https://stackoverflow.com/a/57934504/221528.

(PS. You might want to consider up-voting this issue to indicate your interest.)

[victornoel]

@glen-84 ok, so in the case of kaniko, since there are no docker engine on the host, I suspect it would mean adding an option to the kaniko cli to specify which directory to use when --mount=type=cache is used inside the image. This would elegantly allow to choose the desired directory in a CI context.

Still there would be some thinking to do as to how this interacts with the from and source option I suppose...

[glen-84]

My thoughts were to "reuse" the syntax, as opposed to designing something completely specific to Kaniko. Kaniko could be updated to understand these types of mounts, and to manage the directories outside of the image (i.e. inside the Kaniko container).

[victornoel]

@glen-84 yes, that was my point too :)

[foosinn]

podman has support for this using the well known -v flag:

podman build -v /tmp/cache:/cache .

Using this with ENV COMPOSER_CACHE_DIR=/cache/composer_cache in PHP and ENV YARN_CACHE_FOLDER=/cache/yarn_cache saves us a ton of time and bandwidth.

I would love to see this supported in kaniko.