Compose: docker-compose ups a bad build target with multiple compose files

[greg-md]

Description of the issue

A bad build target in used on containers up when working with multiple compose files.

Seems like the image check doesn't consider the build.target value.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.25.4, build 8d51620a
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Steps to reproduce the issue

1. Use Dockerfile:

FROM node:13 AS builder

FROM builder AS development

CMD ["echo", "DEVELOPMENT"]

FROM builder AS ci

CMD ["echo", "CI"]

1. Create 3 different docker compose files:

docker-compose.yml:

version: '3.4'
services:
  api:
    build:
      context: .
      target: development

docker-compose-ci.yml:

version: '3.4'
services:
  api:
    build:
      context: .
      target: ci

docker-compose-local.yml:

version: '3.4'
services:
  api:
    build:
      context: .
      target: development

1. Run docker-compose -f docker-compose.yml -f docker-compose-ci.yml up.

2. Then run docker-compose -f docker-compose.yml -f docker-compose-local.yml up.

Observed result

On the 1st run, it will use ci build target and will execute echo CI, which is expected.

On the 2nd run, it should use development build target, but it still uses the previous ci target build and executes the same ci command, which is NOT expected.

The same bug happens if you will run commands in reverse order.

Expected result

Should run the expected build target based on extended docker file.

Stacktrace / full error message

grigorii-duca:testapp greg$ docker-compose -f docker-compose.yml -f docker-compose-ci.yml up
Creating network "testapp_default" with the default driver
Building api
Step 1/5 : FROM node:13 AS builder
 ---> f7756628c1ee

Step 2/5 : FROM builder AS development
 ---> f7756628c1ee
Step 3/5 : CMD ["echo", "DEVELOPMENT"]
 ---> Running in 960536e7bd45
Removing intermediate container 960536e7bd45
 ---> 2a69c4326cad

Step 4/5 : FROM builder AS ci
 ---> f7756628c1ee
Step 5/5 : CMD ["echo", "CI"]
 ---> Running in c68aa47d3686
Removing intermediate container c68aa47d3686
 ---> 28cd629f5770

Successfully built 28cd629f5770
Successfully tagged testapp_api:latest
WARNING: Image for service api was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating testapp_api_1 ... done
Attaching to testapp_api_1
api_1  | CI
testapp_api_1 exited with code 0
grigorii-duca:testapp greg$ docker-compose -f docker-compose.yml -f docker-compose-local.yml up
Recreating testapp_api_1 ... done
Attaching to testapp_api_1
api_1  | CI
testapp_api_1 exited with code 0
grigorii-duca:testapp greg$ 

Additional information

MacOS

docker-compose -f docker-compose.yml -f docker-compose-ci.yml config

services:
  api:
    build:
      context: /htdocs/combats/testapp
      target: ci
version: '3.4'

docker-compose -f docker-compose.yml -f docker-compose-local.yml config

services:
  api:
    build:
      context: /htdocs/combats/testapp
      target: development
version: '3.4'

[DracotMolver]

This is happening to me but with a different configuration. It's creating dangling images because when I target to prod it creates the dev stage as well, and this stage was previously created.

My docker file (I removed some info):

FROM node:12 as base

WORKDIR /server

COPY package.json ./

RUN npm install

COPY ./ ./

# ------------------------

FROM base as dev

ENV NODE_ENV=development
ENV APP_HOST="0.0.0.0"

EXPOSE 3000

CMD ["npm", "run", "dev"]

# ------------------------

FROM base as prod

ENV NODE_ENV=production
ENV PORT=5000

EXPOSE 5000

CMD ["npm", "start"]

My docker-compose (I removed some info too):

version: "3.7"
services:
  server:
    build:
      context: ./server
      dockerfile: Dockerfile
      target: dev
    volumes:
      - ./server:/server
      - /server/node_modules
    container_name: newlit_server
    depends_on:
      - db
    ports:
      - 3000:3000
      - 5000:5000

  db:
    image: mongo
    container_name: newlist_db
    ports:
      - 27017:27017
    volumes:
      - mongodb_data_container:/data/db

volumes:
  mongodb_data_container:

If I run docker-compose up for the very first time. It will pull the node image and then will use it as a base for the dev stage. The container is up and running fine.

base stage

... I skipped all the instalation of npm
dev stage

But then I face 2 situations when I change the target:

• If I change the target value to prod, the docker-compose up doesn't rebuild the image based on that stage on the Dockerfile. Instead, it's setting up my container like it was in development mode (devstage). Doesn't docker-compose up build (rebuild) the image if it doesn't exist (prod stage doesn't exist yet) and then run the container?.

rebuild dev stage

• When I run docker-compose build to avoid the step above. It rebuild the base image from cache, perfect!. The dev stage now is dangle because the prod stage has the same label (Am I right why the dangle exist?). But when I see the terminal, it is building the dev stage and then the prod. It make sense If I think docker-compose build runs through all the Dockerfile. If that the case, it is inevitable don't go through the dev stage.

images after all of that

Now. If I change again to dev and rebuild the image with docker-compose up server --build. It dangled the prev image (prod stage) and create again the dev from scratch, not form the cache. so know i have more dangle images, and this cycle will never ends.

What I finally ended up doing was:

• Remove target from docker-compose.yml
• Create two images running docker build ./server -t server-dev --target=dev and docker build ./server -t server-prod --target=prod
• Using these images on docker-compose.yml when I need them.

...
services:
  server:
    # for productions change it to server-prod
    image: sever-dev
...

[bog-h]

Hello Guys! I'm glad that it is a common problem because I started to lose my mind :D

Looks like docker-compose override is ignoring build: target argument. I thought the problem is in the cache, but not!

If you will specify a target (ex debug or base) in the main docker-compose.yaml file then everything will work as expected in depends on your target
build:
target: debug|base

But if you will specify the target in an override file (ex: docker-compose.debug.yaml) then this parameter is ignored!

[DracotMolver]

In my case @bog-h i only work with one docker compose file. I'm using version 3.7

[skyeagle]

@DracotMolver you might be missing DOCKER_BUILDKIT=1

As of Docker 18.09 you can use BuildKit. One of the benefits is skipping unused stages,
so all you should need to do is build with
DOCKER_BUILDKIT=1 docker build -t my-app --target prod

https://stackoverflow.com/a/55320725/368144

[DracotMolver]

@skyeagle That's interesting but even though it doesn't fix the issue of docker-compose not skipping the stage that is not in the target param

[probablykasper]

It seems to keep the build.target from whichever file you ran docker-compose build on.

For example, say my docker-compose.yml specifies target: dev and my docker-compose.prod.yml specifies target: bin. If I now run docker-compose -f docker-compose.prod.yml build and then docker-compose up, it starts the bin stage.

@aiordache Any word on this issue?

[mellertson]

I too am encountering this issue. In my case, I have the two services in the same docker-compose.yaml file:

services:

    server:
        image: registry.mydomain.com/algos_server:latest
        build:
            context: ./
            dockerfile: algos_server.dockerfile
            target: algos_server
        command: manage.py runserver --noreload 0.0.0.0:8000

    celery:
        image: registry.mydomain.com/algos_server:latest
        build:
            context: ./
            dockerfile: algos_server.dockerfile
            target: algos_celery
        command: celery -A algos.celery worker -l info

And I have the following in algos_server.dockerfile:

FROM python:3.7.7-buster AS algos_base
# bunch of commands to build the docker image

FROM algos_base as algos_server
RUN echo "last build line in algos_server"
ENTRYPOINT ["/usr/local/algos/scripts/algos-server-entrypoint.sh"]

FROM algos_base as algos_celery
RUN echo "last build line in algos_celery"
ENTRYPOINT ["/usr/local/algos/scripts/algos-celery-entrypoint.sh"]

And for completeness, I have these two entry point files.
algos-server-entrypoint.sh

#!/bin/bash
set -e
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
# run migrations, import fixtures, etc... etc...
echo "Starting up the Algos server..."
exec "$@"

And in algo-celery-entrypoint.sh I have:

#!/bin/bash
set -e
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
# run migrations, import fixtures, etc... etc...
echo "Starting up the Algos Celery worker..."
exec "$@"

Result

When I run docker-compose -f docker-compose.yaml build && docker-compose -f docker-compose.yaml up I get this output:

Successfully built 590673ed935d
Successfully tagged registry.mydomain.com/algos_server:latest
WARNING: Some networks were defined but are not used by any service: algos_public
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Recreating algos_server_1 ... done
Recreating algos_celery_1 ... done
Attaching to algos_celery_1, algos_server_1
celery_1    | --------------------------------------------------------------------------------
celery_1    | Starting up the Algos Celery worker...
celery_1    | --------------------------------------------------------------------------------
server_1    | --------------------------------------------------------------------------------
server_1    | Starting up the Algos Celery worker...
server_1    | --------------------------------------------------------------------------------

Conclusion

For some reason docker-compose seems to be ignoring the target directive in docker-compose.yaml. I will try separating both services into their own docker-compose.yaml file and see if that helps, but according to the docs for docker-compose` and multi-stage build docs I believe it should be executing each service's entrypoint, instead of executing the entrypoint for Algors Celery for both services.

[mellertson]

I found a work around, which is to describe my two docker services in two separate docker-compose.yaml files.

After I put algos_server into algos-server-base.yaml and algos_celery into algos-celery-base.yaml I get the following output, which is as I expected it to be.

server_1  | --------------------------------------------------------------------------------
server_1  | Starting up the Algos server...
server_1  | --------------------------------------------------------------------------------
celery_1    | --------------------------------------------------------------------------------
celery_1    | Starting up the Algos Celery worker...
celery_1    | --------------------------------------------------------------------------------

[eweidner]

We've run into the original problem here and it appears that this is only a problem if you don't rebuild the target. Docker-compose is not detecting that the built container would be different in each run without re-building.

In the first run of docker-compose -f docker-compose.yml -f docker-compose-ci.yml up, the system sees it doesn't have an image so it builds with the ci target.

In the second run of docker-compose -f docker-compose.yml -f docker-compose-local.yml up, since it's just trying to bring up the container (not build) it doesn't recognize that the already built image is not the same as what would get created with the different build target, and just brings up the existing image.

I think this is not too surprising given that the target is in the "build" area. It's a bit of a more advanced mode to go reevaluate everything about the compose targets during an up.

To solve this, just execute a build before the 2nd up to trigger docker-compose to reevaluate the images and rebuild with the wanted target.

% docker-compose -f docker-compose.yml -f docker-compose-ci.yml up
Creating network "docker-compose-target-bug-test_default" with the default driver
Building api
Step 1/5 : FROM node:14-slim AS builder
 ---> cdb457aa69ed

Step 2/5 : FROM builder AS development
 ---> cdb457aa69ed
Step 3/5 : CMD ["echo", "DEVELOPMENT"]
 ---> Running in 9431c3d0516b
Removing intermediate container 9431c3d0516b
 ---> 1e13634516d7

Step 4/5 : FROM builder AS ci
 ---> cdb457aa69ed
Step 5/5 : CMD ["echo", "CI"]
 ---> Running in 4e5a7801767e
Removing intermediate container 4e5a7801767e
 ---> fc8d5a1871f0

Successfully built fc8d5a1871f0
Successfully tagged docker-compose-target-bug-test_api:latest
WARNING: Image for service api was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating docker-compose-target-bug-test_api_1 ... done
Attaching to docker-compose-target-bug-test_api_1
api_1  | CI


% docker-compose -f docker-compose.yml -f docker-compose-local.yml build
Building api
Step 1/3 : FROM node:14-slim AS builder
 ---> cdb457aa69ed

Step 2/3 : FROM builder AS development
 ---> cdb457aa69ed
Step 3/3 : CMD ["echo", "DEVELOPMENT"]
 ---> Using cache
 ---> 1e13634516d7

Successfully built 1e13634516d7
Successfully tagged docker-compose-target-bug-test_api:latest


% docker-compose -f docker-compose.yml -f docker-compose-local.yml up
Recreating docker-compose-target-bug-test_api_1 ... done
Attaching to docker-compose-target-bug-test_api_1
api_1  | DEVELOPMENT
docker-compose-target-bug-test_api_1 exited with code 0

[probablykasper]

@eweidner For me, rebuilding does not fix it, and I'm not running multiple compose files at once. Do you think it's a different issue?

It seems to keep the build.target from whichever file you ran docker-compose build on.

For example, say my docker-compose.yml specifies target: dev and my docker-compose.prod.yml specifies target: bin. If I now run docker-compose -f docker-compose.prod.yml build and then docker-compose up, it starts the bin stage.

[eweidner]

@probablykasper I don't see any details on your specifics so I'm not sure if your issue is different.

If you are seeing what DracotMolver is seeing then that may be a different issue. I cannot reproduce that in some of my local code that seems similar to theirs. My builds in that situation skips dev when I build prod.

I only reproduced the issue presented by the OP and doing a build in between worked for me.