Compose: Compose `pull` should pull `cache_from` images

[AnthonyMastrean]

Description of the issue

It would be awesome if docker-compose pull also pulled cache_from images! Otherwise, I have to make a separate docker pull command.

Context information (for bug reports)

PS> docker-compose version
docker-compose version 1.20.1, build 5d8c71b2
docker-py version: 3.1.4
CPython version: 3.6.4
OpenSSL version: OpenSSL 1.0.2k  26 Jan 2017

PS> docker version
Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:06:28 2018
 OS/Arch:       windows/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:14:32 2018
  OS/Arch:      linux/amd64
  Experimental: true

Steps to reproduce the issue

1. Create a Compose manifest with a build / cache_from option.
2. Use docker-compose pull or docker-compose build --pull

Observed result

The cache_from images are not pulled and no "Using cache" messages are printed.

Expected result

The cache_from images should be pulled and we should see "Using cache" messages printed.

[shin-]

I don't think that's desirable, at least not as the default. I'm sure there's many cases where pulling the cache_from image would take longer than rebuilding without it, not to mention the cases where the cache_from image isn't actually available remotely. It would break a lot of the current user expectation around the feature, which also matches the Docker CLI (docker build --cache-from A . never attempts to pull A).

[AnthonyMastrean]

Yeah, the general argument of cost to pull vs. cost to rebuild is an interesting one. I'm only exploring this feature because we're migrating to transient CI runner infrastructure. These "runners" have no image cache when started. And we have a handful of images that take some 20+ minutes to build from scratch. So, at least for my use-case, I'm explicitly using cache_from to specifically speed up the CI build.

Only having read and experimented with the docker pull xxx && docker build --cache-from xxx command line, I _hoped_ that Compose's pull would pull cache_from images without a separate docker command. I can understand that it's not a default though...

Maybe a flag is in order? (here's a really bad name to encourage better submissions)

docker-compose pull --include-cache-from-images

[jimmiebtlr]

Any workarounds for this or updates?

[AnthonyMastrean]

We just run the pull separately before the Compose build. So, master is tagged latest and the branch is tagged by its name. So, we cache-from master and master or branch, respectively. Here's a chunk of our GitLab CI config (hopefully the automatic variables are self-explanatory).

build:
  stage: build
  script:
    - docker image pull "$CI_REGISTRY_IMAGE" || true
    - docker image build --cache-from "$CI_REGISTRY_IMAGE" --pull --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" --tag "$CI_REGISTRY_IMAGE" .
    - docker image push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
    - docker image push "$CI_REGISTRY_IMAGE"
  only:
    - master

build-other:
  stage: build
  script:
    - docker image pull "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME" || docker image pull "$CI_REGISTRY_IMAGE" || true
    - docker image build --cache-from "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME" --cache-from "$CI_REGISTRY_IMAGE" --pull --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME" .
    - docker image push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
    - docker image push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME"
  except:
    - master
    - tags

[maximumG]

I was in the same situation as @AnthonyMastrean and was thinking that pull would be automatic when referencing a remote repository in the cache_from list.

We also have some uses cases where pulling image from a remote registry would be faster that re-building the image from scratch, especially when the image tries to install some dependancies (php composer or python pip).

For now we have to analyze the yaml file, extract each services and iterate over each service with a docker pull command.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.

[NewVadim]

Will there be any solution to the issue?

[goetas]

This feature will be very handy... any plans on re-considering this ?