Compose: Epic: Improve performance by running operations in parallel

[bfirsh]

Many of Compose's actions can be performed in parallel rather than in sequence. For larger apps, this would result in enormously reduced startup/redeploy times.

This is a meta-issue for keeping track of all of the stuff happening around that, in rough priority order.

• [x] docker-compose (stop|kill|remove): https://github.com/docker/compose/issues/1651
• [x] docker-compose scale: https://github.com/docker/compose/issues/1653
• [x] docker-compose up: Creating, pulling, building and starting in up can be done in parallel: #1984
• [x] docker-compose pull: Multiple services can be pulled in parallel.
• [x] docker-compose pull --parallel is the default (deprecate flag).

(Replaces #1395)

[dnephin]

931 is also related, doing pull and build in parallel (although I don't think build will benefit at all).

[gkze]

Bump -- this would be amazing since we are having to deal with more and more services in containers that don't depend on each other, so this would help speed up the process immensely

[aanand]

@gkze stop, kill, rm and scale now work in parallel - try it out in Compose 1.4.0 RC2.

[k2xl]

Cool. What about dc build?

[dnephin]

@k2xl because of this https://github.com/docker/docker/issues/9656 bug/limitation in docker doing build in parallel in compose would actually make things slower. Once that's fixed, it should be pretty easy to add.

[willejs]

+1 looking forward to this feature.

[khalilj]

+1

[IlyaSkriblovsky]

Starting phase of up made parallel in #2679 (and also start and stop)

[morlajb]

+1

[niieani]

How about build? When building on a docker-swarm, many nodes could build the images independently.

[chesleybrown]

No update on this issue in quite awhile. Is there still interest in:

docker-compose pull: Multiple services can be pulled in parallel.

I know for us it would be a huge improvement as we have many images that have to be pulled. At this point I've even debated scripting each service to be pulled individually so we could do it in parallel. Thoughts?

[fesor]

Until #4428 is not merged this simple bash script could be used:

for service in $(docker-compose config --services); do
    docker-compose pull $service &
done
wait

[shin-]

With #4428 merged in, it is now possible to do docker-compose pull --parallel. If feedback on the feature is good we'll be able to make it the default behavior in the future.

[schmod]

Is push on the list somewhere too?

[batytskyy]

+1

[jars]

+1 for build parallel.

[shin-]

Parallel build is still blocked by https://github.com/moby/moby/issues/9656