Moby: --cache-from and Multi Stage: Pre-Stages are not cached

[Schnitzel]

Description

If you want to use a Multi Stage Build together with --cache-from it's very hard and complicated to load the Cache of Pre Stages, as --cache-from disables the lookup in the local cache (see https://github.com/moby/moby/issues/32612). The only way is to tag the pre-stage images as well and add them to the --cache-from, which is very complicated.

Steps to reproduce the issue:

1. Assuming we have a Multistage Dockerfile like:

FROM busybox as builder
RUN echo "hello" > test

FROM busybox
COPY --from=builder test test
RUN echo test

1. Building it the first time:

$ docker build -t test:latest .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Running in b5e871ebd251
 ---> 6889762613a0
Removing intermediate container b5e871ebd251
Step 3/5 : FROM busybox
 ---> d20ae45477cb
Step 4/5 : COPY --from=builder test test
 ---> f9ee9cc534a7
Removing intermediate container 8d76fd7eb6be
Step 5/5 : RUN echo test
 ---> Running in 5b768ed39212
test
 ---> b4a81a0e7c96
Removing intermediate container 5b768ed39212
Successfully built b4a81a0e7c96
Successfully tagged test:latest

So far all good.

1. Now running it a second time, see how all layers are fully cached:

$ docker build -t test:latest .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Using cache
 ---> 6889762613a0
Step 3/5 : FROM busybox
 ---> d20ae45477cb
Step 4/5 : COPY --from=builder test test
 ---> Using cache
 ---> f9ee9cc534a7
Step 5/5 : RUN echo test
 ---> Using cache
 ---> b4a81a0e7c96
Successfully built b4a81a0e7c96
Successfully tagged test:latest

1. Now running it with --cache-from test:latest:

$ docker build -t test:latest --cache-from test:latest .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Running in 89d43713b017
 ---> 18e01d7690cb
Removing intermediate container 89d43713b017
Step 3/5 : FROM busybox
 ---> d20ae45477cb
Step 4/5 : COPY --from=builder test test
 ---> Using cache
 ---> f9ee9cc534a7
Step 5/5 : RUN echo test
 ---> Using cache
 ---> b4a81a0e7c96
Successfully built b4a81a0e7c96
Successfully tagged test:latest

See how Step 2/5 : RUN echo "hello" > test is not using any cache.
Interestingly Step 4 is using the cache again, as it finds that cache within the test:latest image.
So it actually builds the first stage image but never uses it. A lot of time the first stages are very heavy computations, like installing packages, building stuff etc. So we almost loose the niceness of Multi Stage Build.

There is a way to fix this, with tagging the first stage image via --target builder:

$ docker build -t test-builder:latest --target builder .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Using cache
 ---> 18e01d7690cb
Successfully built 18e01d7690cb
Successfully tagged test-builder:latest

and then using both images for --cache-from:

$ docker build -t test:latest --cache-from test:latest --cache-from test-builder:latest .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM busybox as builder
 ---> d20ae45477cb
Step 2/5 : RUN echo "hello" > test
 ---> Using cache
 ---> 18e01d7690cb
Step 3/5 : FROM busybox
 ---> d20ae45477cb
Step 4/5 : COPY --from=builder test test
 ---> Using cache
 ---> f9ee9cc534a7
Step 5/5 : RUN echo test
 ---> Using cache
 ---> b4a81a0e7c96
Successfully built b4a81a0e7c96
Successfully tagged test:latest

but IMHO that is super complicated and confusing.

I'm not 100% sure how we could fix this. Implementing --cache-from to also use the local cache as a secondary cache lookup would solve the problem (see https://github.com/moby/moby/issues/32612)

Output of docker version:

$ docker version
Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:53:38 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:54:55 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1059
Server Version: 17.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.41-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: moby
ID: FHCJ:CF22:VRF6:Y4HR:BM3W:ATJ3:3QGW:AGO5:OTKL:W2ES:OM6Q:WZ5Y
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 31
 System Time: 2017-09-03T19:59:13.529192672Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[thaJeztah]

/cc @tonistiigi @AkihiroSuda

[tonistiigi]

I think this can be considered dupe of #33002 if local lookup is a solution. I'd be ok with allowing --cache-from=*

[qqilihq]

@schnitzel Thanks for not only presenting the issue but also a practical workaround! This will for sure save me lots of time!

[falconmick]

@tonistiigi I've been going over heaps of issues threads regarding this so sorry if I have placed questions that relate to other issues here. But if feels like having to run my container building in multiple commands, so that I am able to create a tag for each step is really clumsy. Sure it all works 100%, but maybe there could be a flag for saving that includes all targets I built in the Dockerfile (and uhh I know this is Moby, but isn't Moby basically just the open part of Docker, so puting a docker question here makes sense?)

Cheers

[errordeveloper]

I am seeing this kind of behaviour also. Here is a Google Container Builder file that describes what I am doing exactly (I have single-stage build/Dockerfile and a multi-stage model/Dockerfile that uses the build/Dockerfile in the first stage. I see --cache-from breaks in all but the final sage of the of my multi-stage build). I am using 18.01.0-ce.

[Nowaker]

Here's a super-simple Dockerfile that never has a chance to use cache when --cache-from is defined:

FROM ubuntu:16.04 AS build
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get -qq update && apt-get -yqq upgrade && apt-get -qq clean
RUN apt-get -yqq install curl && apt-get -qq clean
ARG KUBE_VERSION=1.9.2
RUN curl -L https://dl.k8s.io/v${KUBE_VERSION}/kubernetes-client-linux-amd64.tar.gz | \
    tar -Ozxf - kubernetes/client/bin/kubectl > /usr/local/bin/kubectl \
    && chmod +x /usr/local/bin/kubectl
ARG HELM_VERSION=2.8.0
RUN curl -L https://storage.googleapis.com/kubernetes-helm/helm-v${HELM_VERSION}-linux-amd64.tar.gz | \
    tar -Ozxf - linux-amd64/helm > /usr/local/bin/helm && \
    chmod +x /usr/local/bin/helm

FROM ubuntu:16.04
COPY --from=build /usr/local/bin/kubectl /usr/local/bin/helm /usr/bin/

Subsequent calls of docker build . -t $TAG --cache-from $TAG won't reuse any cache.

[Nowaker]

I'm not 100% sure how we could fix this. Implementing --cache-from to also use the local cache as a secondary cache lookup would solve the problem (see #32612)

@Schnitzel Correct, cache is used correctly for all steps when no --cache-from is provided.

[stealthybox]

just adding another user report:
I'm running into the same walls @errordeveloper is with persisting build caches for Google Container Builder

[tbeadle]

I believe I'm running into this same issue. We have a multi-stage build running in a CI environment, so it would seem that using --cache-from=<image that was pushed to CI master> would be the way to go, but those "build" stages (which are by far the longest steps in the process of building the final image) are not using the cache. This is a major drag.

[ElectricWarr]

Also seeing this in a CI environment where enabling the local cache when using --cache-from won't be enough.

Another workaround is to break your multi-stage build into two Dockerfiles (using @Nowaker's example above), then as long as you also push the builder image up to your repo you can build each step with it's own --cache-from.

The extra steps are definitely a pain, but in the context of CI it's not a lot more complicated than the multi-stage build was in the first place.

$ docker build --tag=tool:builder --file=- . <<EOF
FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get -qq update && apt-get -yqq upgrade && apt-get -qq clean
RUN apt-get -yqq install curl && apt-get -qq clean
ARG KUBE_VERSION=1.9.2
RUN curl -L https://dl.k8s.io/v${KUBE_VERSION}/kubernetes-client-linux-amd64.tar.gz | \
    tar -Ozxf - kubernetes/client/bin/kubectl > /usr/local/bin/kubectl \
    && chmod +x /usr/local/bin/kubectl
ARG HELM_VERSION=2.8.0
RUN curl -L https://storage.googleapis.com/kubernetes-helm/helm-v${HELM_VERSION}-linux-amd64.tar.gz | \
    tar -Ozxf - linux-amd64/helm > /usr/local/bin/helm && \
    chmod +x /usr/local/bin/helm
EOF

$ docker build --tag=tool:latest --file=- . <<EOF
FROM tool:builder as build
FROM ubuntu:16.04
COPY --from=build /usr/local/bin/kubectl /usr/local/bin/helm /usr/bin/
EOF

[bryanlarsen]

Note that when using this strategy, the first line following the FROM in each stage must be unique. Docker build uses the 'first match' strategy rather than 'best match'

[ElectricWarr]

Note that when using this strategy, the first line following the FROM in each stage must be unique. Docker build uses the 'first match' strategy rather than 'best match'

@bryanlarsen I'd like to understand this a bit better, can you link any docs/blogs about first vs. best match in Docker build?

[dhinus]

@ElectricWarr this comment kind of explains it.

Imagine you have these two cached images:

# Dockerfile1
FROM node:alpine
ADD . .
RUN npm i
RUN npm test

# Dockerfile2
FROM node:alpine
ADD . .
RUN npm i
RUN npm build

If you re-build the second one, using --cache-from for both images, you'd like the cache for the second image to be used, but given the first layers are the same, Docker will decide to use the first image as the cache, and will re-run the last step because it's different from the one it was expecting from the cache.

docker build -t test1 -f Dockerfile1 .
docker build -t test2 -f Dockerfile2 .
docker build --no-cache -f Dockerfile2 . # purge cache

docker build --cache-from test1,test2 -f Dockerfile2 . # last step is not taken from cache
docker build --cache-from test2,test1 -f Dockerfile2 . # all steps are taken from cache

Hope this helps!

[whereisaaron]

We are seeing the same thing with multi-stage builds in CI environments like AWS CodeBuild.

Pushing a previous image on one CI instance and then pulling to a new CI instance and using --cache-from never works, it rebuilds even if the base image and build context has not changed.

Yet if you try the same thing on same instance then --cache-from works fine.

It seems that docker push is missing something that docker pull doesn't restore. So when you change instance --cache-from no longer matches the layers it should.

[tonistiigi]

This has been addressed by buildkit. DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 .

[glensc]

@tonistiigi --build-arg? isn't build arg supposed for Dockerfile use? it should be env like $DOCKER_BUILDKIT, looks like a huge hack. any links to issues/merge-requests that implemented it? maybe get a backstory...

[docwhat]

@glensc Apparently it is --build-arg per the docs.

[tonistiigi]

@glensc https://github.com/moby/buildkit/pull/777 https://github.com/moby/moby/pull/38882

[carlosgalvezp]

@tonistiigi This is not working for me right now. Procedure:

1. Remove all docker images and containers
2. $ docker pull registry:port/image:tag
3. $ DOCKER_BUILDKIT=1 docker build --cache-from=registry:port/image:tag --build-arg=BUILDKIT_INLINE_CACHE=1 ...

Then docker builds all layers from scratch, instead of using the cache. Even if the Dockerfile is identical to the one that was used to build the image that we pull as cache.

This happens on Docker 19.03.5.

Why is this happening? Am I understanding the docs and your comment correctly?

[thisismydesign]

@tonistiigi How was it addressed? The expectation seems to be that it should just work, but there are still reports of it not working so could you or anyone give an actual example of what's exactly supposed to happen in which exact version?

I'm spending a lot of time on this and have absolutely no idea if I just misspelled an image name somewhere or --cache-from isn't working at all. Add docker-compose to that and it's an absolute drag to try to work with this.

[tonistiigi]

https://github.com/moby/moby/issues/39003#issuecomment-573757398

[setpill]

@thisismydesign it can be fixed with a quite simple change to your docker build call; simply enable the use of the integrated buildkit and tell buildkit to export cache inline. Like so:

- docker build --cache-from [..]
+ DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from [..]

[setpill]

@carlosgalvezp Have you tried pushing the image built with buildkit to registry and seeing if the cache works in subsequent builds? In my experience enabling the buildkit and inline cache export broke the cache once but subsequent multistage builds correctly used the cache.

[carlosgalvezp]

@setpill Hi, we figured out that the issue was file permissions and users of files that are COPY'd into the image. In our CI servers the files have different usernames than locally, so this makes us unable to take advantage by pulling the pre-built images from the CI servers.

We do have issues with missing layers (0 bytes) when activating --cache-from, I think I created a separate ticket for that, not sure in which repo :) I don't know if it's due to docker or due to the place where we store the images.

[fabb]

In our CI servers the files have different usernames than locally, so this makes us unable to take advantage by pulling the pre-built images from the CI servers.

Is there no way around this, like telling docker to ignore file permissions when COPYing?

[tonistiigi]

@fabb @carlosgalvezp BuildKit does not transfer your CLI side user info with the build context. Files copied with COPY are always normalized to root or --chown value.

[virgile-hogman]

@tonistiigi but this only concerns the UID/GID, right? This is actually different from the file permissions (ex. 644). Differently said, there is no equivalent of --chmod.

From my observations, i could see that the Docker cache would miss if flags such as SGID were different on the hosts where the image was built. Maybe that is the expected behavior, i don't really know. I suppose you want to preserve some of the rights, but if you normalize to root, i'm not sure the answer is so trivial. What about the file permissions (e.g. group/other and special flags like SGID)?

It took me a lot of time to understand what was going wrong. The problem is that the logs are not much informative, they just say cache used or not. It would be nice to have some logs giving more details why the cache fails, or a clear documentation explaining what the cache really depends on.

Eventually i came up with my own custom tool querying some reference file with stats like this:
stat -c '%a %u %g' which empirically helped me to understand what could differ. What is Docker doing exactly during COPY?

[tonistiigi]

chmod is unrelated to this issue. It is tracked by your version control and can't differ if you go to another node.

[virgile-hogman]

In complex industrial CI setups, the configurations of the servers (used for build) can differ a lot. We have dozens of different teams with hundreds of engineers, working on different projects using shared and heterogeneous resources. Every single engineer doesn't have the permission to change the user or file permissions settings. In practice we need to find how the cache really works to be able to make use of it. The logs don't say why the cache fails and there is no documentation explaining in details how this works.

[virgile-hogman]

@tonistiigi the main question here is straightforward: what in a docker COPY instruction can make the cache to miss? How are the file permissions handled during the COPY?

[tonistiigi]

File permissions are copied as is, otherwise, none of the projects would work. If your different project use different files (and different mode is a different file for version control) then they are not supposed to match cache for the other. If it did it would mean that broken images are created.

[virgile-hogman]

I understand, but since the user is normalized to root the mapping between the users on the host and in the Docker is not necessarily that trivial. If we consider the COPY operation as creating a new file in the Docker, we may consider for example the umask for user root inside the Docker. Typically this is 022 for root and 002 for normal users which is quite a natural concept on Linux but maybe this is already too OS-specific, i don't know.

From what you say, we should consider the COPY as a copy --preserve=mode on Linux ignoring any umask.

It could have been a nice feature to have more control over this during the COPY also depending how it is done internally in Docker, that was also the reason for asking to understand a bit better how it works.

Also, having more detailed logs would be really helpful if this is possible. Finding that the cache miss was due to an inconsistent SGID flag was not easy to figure out.