Compose: Feature Request: Add depends_on support for external Compose files

[pcgeek86]

For more complex applications, with many components, it would be useful to be able to make a Docker Compose service depends_on an external docker-compose.yml file.

Is this something that can be added?

Cheers,
Trevor Sullivan
Docker Captain
Microsoft MVP: Cloud & Data Center Management
https://trevorsullivan.net
https://twitter.com/pcgeek86

[shin-]

Hi!

Do you mind giving an example of a case where you feel that would be useful? I have a feeling it's already possible to do what you need with the current Compose feature set, but I'd like to understand your setup better first.

[pcgeek86]

@shin- Sure, we are adding a separate application logging stack, that's made up of several containers. This logging stack is based on ElasticSearch, Logstash, and Kibana (ELK). We'd like to leave this in a separate Compose file, so we can deploy and validate it separately from our microservices. What is your recommended approach for this scenario?

[shin-]

You can have multiple Compose files, one called docker-compose.yml (default) for your microservices, and docker-compose-logging.yml with your logging stack. You can then start your entire stack with docker-compose -f docker-compose.yml -f docker-compose-logging.yml, with dependencies being resolved as if they were part of the same file. There's more docs on the -f option on this page

[pcgeek86]

@shin- Thanks -- we will give that a shot!

[olalonde]

I have a similar issue, asked on SO: http://stackoverflow.com/questions/39892588/avoiding-port-conflicts-between-docker-compose-yml-configurations

[alwaysastudent]

This issue has leg. The scenario we need the external depends_on is to start the services only if a reverse-proxy service is started and be running already from another project. The reverse-proxy project will compose the shared network and be ready as soon as the backend services come up.

[jayaskren]

We would also like this. I am running an integration tests inside jenkins. The first docker-compose file is used to set up and configure the services I'm testing. The second docker-compose file is to run the actual tests. I then want the Jenkins job to pass or fail depending on the results of the test in the second docker-compose file.

[shin-]

@jayaskren

docker-compose.yml

version: '2.1'
services:
  web:
   ...
  db:
   ...
volumes:
  ...
networks:
  ...

docker-compose-tests.yml

version: '2.1'
services:
  testrunner:
    ...

Run application: docker-compose up
Run tests: docker-compose -f docker-compose.yml -f docker-compose-tests.yml up --exit-code-from testrunner

[jayaskren]

@shin- One of my containers starts a database, the second one loads data into the database, then I have several containers that launch microservices that talk to that database. Then the last service runs my tests against those services. When I tried your suggestion, I see in the logs:
using --exit-code-from implies --abort-on-container-exit

and as I've seen before, the stack comes down once the second container finishes loading data into the database due to the abort-on-container-exit command. I really want to abort on container exit but only for my testrunner container.

[drzraf]

still an issue given that https://github.com/moby/moby/issues/31101 keeps us from split a docker-compose among multiple regular files:
Can't run a database-related compose first then run/rerun a (dependent) web-app compose.

[nyurik]

I also have a similar issue: I need to start several databases, run a very long import process, and finally start a set of microservices. Ideally, I should be able to break that into 3 distinct docker-compose.yml files, and I would not want to run them in the same up command. Rather, I would

1. docker-compose up -f databases.yml --detach
2. docker-compose up -f importjobs.yml (without --detach to wait until all of them finish)
3. docker-compose up -f services.yml --detach

It would be good to be able to use depends_on inside the second and third set to ensure the DB services are still running.

[nycynik]

You can have multiple Compose files, one called docker-compose.yml (default) for your microservices, and docker-compose-logging.yml with your logging stack. You can then start your entire stack with docker-compose -f docker-compose.yml -f docker-compose-logging.yml, with dependencies being resolved as if they were part of the same file. There's more docs on the -f option on this page

But this will require they are all in the same project as well, right?

[totopopov]

Has anyone managed to find a work around on depends_on in the case where you want one service to start after one has already completely loaded?

[123unix]

You can have multiple Compose files, one called docker-compose.yml (default) for your microservices, and docker-compose-logging.yml with your logging stack. You can then start your entire stack with docker-compose -f docker-compose.yml -f docker-compose-logging.yml, with dependencies being resolved as if they were part of the same file. There's more docs on the -f option on this page

This is exactly what the new feature request is all about!

Indeed, how else - in the new syntax which would be developed - would you specify external dependency between docker-compose files?