Compose: Building containers in parallel to speed up performance?

[specialtactics]

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
When using docker-compose up, and more than 1 container needs to have it's image build/rebuild, it would be fantastic if the building for all containers could be done in parallel rather than sequentially, to speed up the process.

Describe alternatives you've considered
None

Additional context
Situations where multiple docker containers need to have their image either built for the first time, or rebuilt because of some changes.

[chris-crone]

Hi @specialtactics,

Thanks for posting this. Have you taken a look at buildx? It's an experimental build tool that can build all service images in a Compose file using BuildKit. It will be released (as experimental) with the 19.03 release.

[specialtactics]

Hey @chris-crone - sounds interesting, what will be the relationship between buildx and docker/docker-compose though? I would prefer that docker-compose worked this way by default instead of having to also use some third-party tooling to facilitate this functionality.

[chris-crone]

@specialtactics Since docker-compose is written in Python and the rest of the Docker stack is Go, it's quite difficult to integrate things like buildx well. The general plan for buildx is for its functionality to land in the docker build command when it's stable.

[giorgiosironi]

Doesn't docker-compose build --parallel cover this?

[chris-crone]

@giorgiosironi Yes, --parallel will make it quicker if you have multiple service images that need building. This will still use the older build system though and not BuildKit.

[ostrolucky]

Use this to utilize both buildx and --parallel

COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build --parallel

[julianrutten]

--parallel is good but could this be added to docker-compose up too? I was quite confused not finding it there even though I was running latest versions. Edit: reasoning: saves a build step in my case, which is a lot cleaner.

[k1w1m8]

Using a separate build and then up is only a partial workaround for this.

This is because the existing separate build and up steps mean we have to wait for _all_ the containers to stop building before starting _any_ containers.

If the --parallel option was added to up, it would (in theory) be possible to begin starting containers before all the builds were finished.

This could reduce the overall start time considerably.

[onlinejudge95]

@k1w1m8 adding a --parallel option to docker-compose up has a caveat though.

Let's say I defined 2 services A & B in my docker-compose.yml with a constraint that A depends on upo B now creating a parallel build of the image is fine but creating containers in parallel might break things.

If we want to have a --parallel in docker-compose up we need to have a check that no depends clause is present in the service definitions. I prefer delegating this to the user as the last option.