Compose: docker-compose build --pull fails on local image

[abitrolly]

It is impossible to run a docker-compose build that will use locally built images and pull updated remote image at the same time.

build --pull will fail on local image.

$ docker-compose build --pull app
Step 1/6 : FROM local-image as local
ERROR: Service 'app' failed to build: pull access denied for local-image, repository does
 not exist or may require 'docker login'

build without --pull won't fetch latest version of other base images.

Context information (for bug reports)

Output of docker-compose version

# docker-compose version
docker-compose version 1.23.1, build b02f130
docker-py version: 3.7.2
CPython version: 2.7.15
OpenSSL version: LibreSSL 2.6.5

Adding --ignore-pull-failures to docker-compose build --pull should be enough to fix the issue. Otherwise a separate builder script is needed to implement the logic without docker-compose.

[abitrolly]

An alternative solution is to add explicit filter --pull-ignore=image1,image2,... for docker-compose build.

[abitrolly]

The problem to ignore images with docker-compose build --pull is that docker-compose should parse Dockerfile to get image names, and it doesn't look like it parses Dockerfiles right now.

[abitrolly]

Here is a separate script that allow to explicitly ignore images - https://gist.github.com/abitrolly/24ddc1c05db2239d4689e7479d0b063f

[saikrishnav]

I can confirm that I am having same issue.

My DockerFile starts like this:

ARG BASEIMAGE
FROM ${BASEIMAGE} as base-image
RUN echo ${BASEIMAGE}
........

now, when I do :
docker build --build-arg BASEIMAGE=localimage:v1 .

• this works.

But my docker-compose yml which has this:

services: 
  web: 
    build:
      context: .
      dockerfile: service.dockerfile
      args: 
        BASEIMAGE: ''

when I issue command:
docker-compose build --build-arg BASEIMAGE=localimage:v1
this doesn't work and throws the below errors:
Building web Step 1/11 : ARG BASEIMAGE Step 2/11 : FROM ${BASEIMAGE} as base-image ERROR: Service 'web' failed to build: pull access denied for localimage, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

[mathroc]

same here, would welcome having --ignore-pull-failures

[ndeloof]

I tried to reproduce, here is my scenario :

# Create a localimage
➜  docker tag alpine localimage:v1
# Define Dockerfile to require a build-arg
➜  echo "ARG BASEIMAGE 
FROM ${BASEIMAGE} as base-image" > Dockerfile
# Check docker build is happy with it
➜  docker build --build-arg BASEIMAGE="localimage:v1" .
Sending build context to Docker daemon  4.096kB
Step 1/3 : ARG BASEIMAGE
Step 2/3 : FROM ${BASEIMAGE} as base-image
Successfully built 13e8991aa613
# Do the same using docker-compose
➜  docker-compose build --build-arg BASEIMAGE=localimage:v1
Building web
Step 1/2 : ARG BASEIMAGE
Step 2/2 : FROM ${BASEIMAGE} as base-image
 ---> 961769676411
Successfully built 961769676411
Successfully tagged toto_web:latest

I'm running docker-compose version 1.24.1, please try to upgrade to confirm this issue has already been fixed.

[mathroc]

@ndeloof you forgot the --pull arguement

$ docker-compose --version
docker-compose version 1.24.1, build 4667896b
$ docker --version
Docker version 19.03.3, build 1576bd7
$ docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01fb
Status: Image is up to date for alpine:latest
docker.io/library/alpine:latest
$ docker tag alpine localimage
$ echo FROM localimage > Dockerfile
$ echo "version: '2'
  services:
    service:
      build: ." > docker-compose.yml
$ docker-compose build --pull
Building service
Step 1/1 : FROM localimage
ERROR: Service 'service' failed to build: pull access denied for localimage, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

[ndeloof]

Ah ok for it.
But I'm not sure what should be expected here. You ask compose to pull images that actually don't exist on any registry, should it just ignore any error - and then as well ignore those that would have been relevant?

[mathroc]

@ndeloof I guess there 3 options:

A: compose always ignore error on its own. (I don't like it)

B: --ignore-pull-failures would work but might hide real issues

C: --ignore-pull-failures[=service:name] or --ignore-pull-failures[=image:name] would be great but more complicated to implement I suppose

I'd be okay for either B or C

[ndeloof]

Actualy the docker CLI behave the same :

➜  docker build --pull  .
Sending build context to Docker daemon  4.096kB
Step 1/1 : FROM localimage:1
pull access denied for localimage, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

So it would not make any sense for compose to have a different behaviour. Now, if/when the CLI offer some extra option to better control such a scenario, we could align compose with same options.

[mathroc]

:( the docker CLI don't have a pull --ignore-pull-failures either but compose does.
The issue here is that if you configure even only one service with a local image, you can't have docker build --pull working without listing each services to build (which can be error prone: it's likely to forget to update the service list in a script when adding a new one in docker-compose)

maybe it could be done another way:

docker-compose build --pull --no-pull=service1 --local-image=service2

that way the --pull argument would be omitted for service1 & service2

[abitrolly]

@ndeloof will you accept pull request that will allow docker-compose users to ignore local images?

[ndeloof]

I'm not the one who would take such a decision ;)
I wonder how you would handle such an option without CLI support for it. Please remember compose has an option to build with the CLI, not just by python-driven API call.

[abitrolly]

@ndeloof by using pre-processor to pull images prior to CLI run https://gist.github.com/abitrolly/24ddc1c05db2239d4689e7479d0b063f

[knasim]

this is still an issue. docker-compose is going to the repo instead of using the local image on my workstation

[nickwesselman]

This should be re-opened. It's definitely still problematic and frustrating. My compose has a mix of images w/ local and remote base and this makes it difficult to get the latest for remote parents.

[jflheureux]

I hit this issue today. I have a docker-compose.yml file like this:

version: "2.4"

services:

  solution:
    image: ${REGISTRY}build/lighthouse-solution:${SITECORE_VERSION}-${DEMO_VERSION}
    build:
      context: .
      target: solution
      dockerfile: .\docker\images\windows\demo-solution\Dockerfile
      args:
        BUILD_IMAGE: mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-${WINDOWSSERVERCORE_VERSION}
        BASE_IMAGE:  mcr.microsoft.com/windows/nanoserver:${NANOSERVER_VERSION}
    scale: 0

  mssql:
    image: ${REGISTRY}demo/lighthouse-mssql:${SITECORE_VERSION}-${WINDOWSSERVERCORE_VERSION}-${DEMO_VERSION}
    build:
      context: .
      dockerfile: .\docker\images\windows\demo-mssql\dockerfile
      args:
        SOLUTION_IMAGE: ${REGISTRY}build/lighthouse-solution:${SITECORE_VERSION}-${DEMO_VERSION}
        BASE_IMAGE: ${REGISTRY}demo/base/lighthouse-xp0-modules-base-mssql:${SITECORE_VERSION}-${WINDOWSSERVERCORE_VERSION}-${BASE_MODULE_VERSION}
    depends_on:
      - solution

I use the solution service to build all my code once. Then, I use the resulting lighthouse-solution image as a dependency and a base image in many other service dockerfiles. We never push the lighthouse-solution image to our Docker registry as it is only used for building the other images.

When building with docker-compose build --pull, we are getting the issue described here:

ERROR: Service 'mssql' failed to build : manifest for scr.sitecore.com/build/lighthouse-solution:10.0.0-1000.0 not found: manifest unknown: manifest tagged by "10.0.0-1000.0" is not found

I resolved this problem by building my own PowerShell pull and build script:

$configuration = docker-compose config
$images = @()

# Find images to pull in the docker-compose configuration
foreach ($line in $configuration) {
  if ($line -match "(BUILD_IMAGE|BASE_IMAGE|ASSETS_IMAGE):\s*([^\s]*)") {
    $images += $Matches.2
  }
}

# Pull images
$images | Select-Object -Unique | ForEach-Object {
  $tag = $_
  docker image pull $tag
  $LASTEXITCODE -ne 0 | Where-Object { $_ } | ForEach-Object { throw "Failed." }
  Write-Host ("External image '{0}' is latest." -f $tag) -ForegroundColor Green
}

docker-compose build

I know I am using only the BUILD_IMAGE, BASE_IMAGE, and ASSETS_IMAGE build arguments for base images in my docker-compose.yml file. I also know I am not using hardcoded base images in my dockerfile files. This allows me to avoid loading the dockerfile files as the Python solution shared in this thread.

I hope this helps others facing the same issue.