Compose: cache_from works with docker, not with docker-compose v3.2

[Stono]

Hey,
I'm experiencing some strange behaviour.

See the following docker build:

$ docker build -t eu.gcr.io/peopledata-product-team/td.gateway . --cache-from eu.gcr.io/peopledata-product-team/td.gateway:latest
Sending build context to Docker daemon  208.9kB
Step 1/9 : FROM eu.gcr.io/peopledata-product-team/td.nodejs:latest
 ---> f102e68fb27d
Step 2/9 : ENV APP_ENV docker
 ---> Using cache
 ---> ab50400417cc
Step 3/9 : EXPOSE 9000
 ---> Using cache
 ---> d0fd739c96e2
Step 4/9 : COPY package.json /app/
 ---> Using cache
 ---> d6d0c39b1fcf
Step 5/9 : RUN npm install
 ---> Using cache
 ---> 3e48638935cd
Step 6/9 : COPY . /app/
 ---> Using cache
 ---> ed4d8b4a0aa2
Step 7/9 : RUN find . -type d \( -path ./node_modules \) -prune -o -exec chown nodejs:nodejs {} \;
 ---> Using cache
 ---> 8ebef246e2df
Step 8/9 : USER nodejs
 ---> Using cache
 ---> bc1a6683aabd
Step 9/9 : VOLUME /app
 ---> Using cache
 ---> ecb0dabe1bac
Successfully built ecb0dabe1bac
Successfully tagged eu.gcr.io/peopledata-product-team/td.gateway:latest

As you can see, nicely cached.
However trying to do the same thing with compose I get cache invalidations. I find it interesting it does seem to cache up until the point of a file based copy (the env/expose are fine).

$ docker-compose build
hawkeye uses an image, skipping
redis uses an image, skipping
rabbitmq uses an image, skipping
Building app
Step 1/9 : FROM eu.gcr.io/peopledata-product-team/td.nodejs:latest
 ---> f102e68fb27d
Step 2/9 : ENV APP_ENV docker
 ---> Using cache
 ---> ab50400417cc
Step 3/9 : EXPOSE 9000
 ---> Using cache
 ---> d0fd739c96e2
Step 4/9 : COPY package.json /app/
 ---> 73dbfff2fd48
Removing intermediate container dbbebf72e114
Step 5/9 : RUN npm install
 ---> Running in b59db8bf397c

my docker-compose.yml looks like this:

$ cat docker-compose.yml
version: '3.2'

volumes:
  code:

services:
  app:
    image: eu.gcr.io/peopledata-product-team/td.gateway
    restart: always
    links:
      - redis
      - rabbitmq

and my docker-compose.override.yml looks like this:

$ cat docker-compose.override.yml
version: '3.2'

services:
  app:
    build:
      context: .
      cache_from:
        - eu.gcr.io/peopledata-product-team/td.gateway:latest

docker versions:

$ docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:20:36 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:51:55 2017
 OS/Arch:      linux/amd64
 Experimental: false

docker-compose versions:

$ docker-compose --version
docker-compose version 1.15.0-rc1, build 2188098

[shin-]

This is most likely caused by https://github.com/docker/docker-py/issues/998. The fix to said issue is dependent on the docker engine using go1.9 - and once that happens, I expect it will only affect images built with this newer version.

[edmorley]

Worth noting: moby/moby#33935 reverted the upstream Go fix that's required for docker/docker-py#998, due to wanting to reduce the cache-breakage churn (see: https://github.com/moby/moby/pull/33935#issuecomment-326828700). As such the next time the caches are going to be invalidated, someone needs to remember to revert that change, or we'll still be in the same situation then.

[kilianc]

I am experiencing this as well, is there a workaround while we wait for a fix?

[shin-]

Workaround is to always use the same method to build (either from the docker CLI or from docker-compose).

[kilianc]

Then I think I am experiencing something different, I usually pull latest and build locally with cache_from enabled in my docker compose file. While CI is correctly re-using the layers my machine doesn't. CI runs the same script I run. Is it possible that mismatching versions of docker in CI and on my machine create this side effect?

[shin-]

Is the image you pull built with Compose as well?

[kilianc]

Yes, same exact script, containing docker-compose build

[kilianc]

I host my images on GCR, I jus SSH'd on CI again and manually pulled the image, set

    build:
      context: ./
      cache_from:
        - ${IMAGE_NAME}:development

and it doesn't use the cache. Is there a verbose/debug flag I can run to get to the bottom of this?

[shin-]

@kilianc Ok, for now let's assume this is a different bug. Can you open a new issue with the following details:

• Console output of the build command
• Dockerfile contents
• Compose and Docker versions
• Does it work as expected if you use the image built locally instead of the one you pull from GCR?

[shin-]

Closing to consolidate with #883

[timini]

@kilianc did you find a resolution to this?

[kilianc]

@timini nope.

Workaround in our shell wrapper:

if [[ "$(docker-compose build 2>&1 | tee /dev/stderr)" == *"invalid credentials"* ]]; then
  docker pull some_image
  docker-compose build
fi

[chaitan94]

After hours of searching, I found the solution. It seems even base image needs to be listed in the cache_from list as mentioned in https://stackoverflow.com/a/39772423. This does not seem intuitive/obvious for me at least, but it worked for my use case. Sharing it here so that it may help anyone coming across this issue.

[radostyle]

Note that cache_from in docker-compose 3.7 works

[digitalnomd]

Looks like it was added in version 3.2 https://docs.docker.com/compose/compose-file/compose-versioning/#version-32.