Compose: Docker-compose cannot orchestrate services together in Docker-dind properly

[pilec]

Hello, today my build process started to use docker-compose 1.27.0 and it starting to fail.

Description of the issue

My Django service cannot see my postgres docker container when I run it in Docker-dind

Context information (for bug reports)

Lets have a build script on gitlab:

Test and Lint:
  image: docker:19.03.5
  services:
    - docker:19.03.5-dind
  stage: Test and Lint
  script:
    - apk add python3-dev libffi-dev openssl-dev gcc libc-dev make
    - pip3 install docker-compose
    - docker-compose run --rm app sh -c "python manage.py wait_for_db && python manage.py test && flake8"

Output of docker-compose version

1.27.0

Output of docker version

19.03.5

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

version: "3"

services:
  app:
    build:
      context: .
    ports:
      - "8000:8000"
    volumes:
      - ./app:/app
    command: >
      sh -c "python manage.py wait_for_db &&
             python manage.py migrate &&
             python manage.py runserver 0.0.0.0:8000"
    environment:
      - DB_HOST=db
      - DB_NAME=app
      - DB_USER=postgres
      - DB_PASS=supersecretpassword
      - DEBUG=1
    depends_on:
      - db

  db:
    image: postgres:10-alpine
    environment:
      - POSTGRES_DB=app
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=supersecretpassword

Steps to reproduce the issue

1. Just run a pipeline on github

Observed result

psycopg2.OperationalError: could not connect to server: Connection refused
    Is the server running on host "db" (172.19.0.2) and accepting
    TCP/IP connections on port 5432?

Full build log:
faulty version with docker-compose 1.27.0: https://gist.github.com/pilec/bf9db289bbda04eb8897c66f657802f6

Expected result

Waiting for database...
Database available!
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...Waiting for database...
Database unavailable, waiting 1 second...
Database unavailable, waiting 1 second...
Database unavailable, waiting 1 second...
Database unavailable, waiting 1 second...
Database unavailable, waiting 1 second...
Database available!
.Waiting for database...
Database available!

Full build log
allright version with docker-compose 1.26.2: https://gist.github.com/pilec/8dd4d6ed86b61da950c3152cbf38a114

Additional information

If I downgrade docker-compose manually by changing my build command from

    - pip3 install docker-compose

to

    - pip3 install -I docker-compose==1.26.2

everything runs smoothly

wait_for_db command

Is a classical Django command, to be sure, that db is up and running:

class Command(BaseCommand):
    """ Django command to pause execution until database is available"""
    def handle(self, *args, **kwargs):
        self.stdout.write('waiting for db ...')
        db_conn = None
        while not db_conn:
            try:
                # get the database with keyword 'default' from settings.py
                db_conn = connections['default']
                # prints success messge in green
                self.stdout.write(self.style.SUCCESS('db available'))
            except OperationalError:
                self.stdout.write("Database unavailable, waiting 1 second ...")
                time.sleep(1)

[ulyssessouza]

Thanks for the report @pilec !

Looks like a regression.
We are having a look at this and it should be corrected in 1.27.1.

[ulyssessouza]

Hello @pilec !

Could you please try running your services with ipc: shareable and tell me if that fixes the problem?

In the file you've just posted above, that would be:

version: "3"

services:
  app:
    ipc: shareable
    build:
      context: .
    ports:
      - "8000:8000"
    volumes:
      - ./app:/app
    command: >
      sh -c "python manage.py wait_for_db &&
             python manage.py migrate &&
             python manage.py runserver 0.0.0.0:8000"
    environment:
      - DB_HOST=db
      - DB_NAME=app
      - DB_USER=postgres
      - DB_PASS=supersecretpassword
      - DEBUG=1
    depends_on:
      - db

  db:
    ipc: shareable
    image: postgres:10-alpine
    environment:
      - POSTGRES_DB=app
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=supersecretpassword

Looks like a regression on last release.

[pilec]

Hello @ulyssessouza

I copied and pasted your improved docker-compose.yml and unfortunately it ends up with the same error.

[ulyssessouza]

Actually, looking at your wait_for_db command, the messages in it doesn't match to the content of the logs, while one says "db available" and the other "Database available". Also for unsuccessful results and retries.
We actually see a "Database available" in the logs, suggesting that the command was successful.

Checking at this approach of service dependency, this doesn't look like the best way to check PostgreSQL availability. It has a binary to properly do the job. This combined to docker-compose's health_check feature, would do a proper job.
Here is how your docker-compose.yml would look like with this 2 things.

version: "3.8"

services:
  app:
    build:
      context: .
    ports:
      - "8000:8000"
    volumes:
      - ./app:/app
    command: >
      sh -c "python manage.py migrate &&
             python manage.py runserver 0.0.0.0:8000"
    environment:
      - DB_HOST=db
      - DB_NAME=app
      - DB_USER=postgres
      - DB_PASS=supersecretpassword
      - DEBUG=1
    depends_on:
      - db:
          condition: service_healthy

  db:
    image: postgres:10-alpine
    environment:
      - POSTGRES_DB=app
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=supersecretpassword
    healthcheck:
      test: ["pg_isready -U postgres"]
      interval: 5s

Could you please, try this version and provide feedback?

[pilec]

unfortunately:

The Compose file './docker-compose.yml' is invalid because:
services.app.depends_on contains {"db": {"condition": "service_healthy"}}, which is an invalid type, it should be a string

with docker-compose 1.27.0 and docker-compose.yml format version 3.8.

I don't want to be disrespectful or something, but according to documentation, docker-compose drop support for the condition in depends_on section in docker-compose.yml format version 3:

Version 3 no longer supports the condition form of depends_on.

is there any other way, how to remove the necesity of wait_for_db command?

[ulyssessouza]

Actually the new schema is a merge of all the prior schemas check the compose-spec. I actually missed one point when specifying the dependency.
It should be an object instead of a string in a list. The following should work (I've just tested):

version: "3.8"

services:
  app:
    build:
      context: .
    ports:
      - "8000:8000"
    volumes:
      - ./app:/app
    command: >
      sh -c "python manage.py migrate &&
             python manage.py runserver 0.0.0.0:8000"
    environment:
      - DB_HOST=db
      - DB_NAME=app
      - DB_USER=postgres
      - DB_PASS=supersecretpassword
      - DEBUG=1
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:10-alpine
    environment:
      - POSTGRES_DB=app
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=supersecretpassword
    healthcheck:
      test: "pg_isready -U postgres"
      interval: 5s

[pilec]

Your last version of docker-compose works as a charm. Thank you so much!

[ulyssessouza]

You are welcome!