Compose: Docker-compose run significantly slower than docker run

[Maxim-Filimonov]

I've noticed that docker-compose run seems to be significantly slower than plain docker run. Based on verbose logs it looks like it's slow due to docker resize command.
See example:

See logs https://gist.github.com/Maxim-Filimonov/7c90d303d102a0074c32

Additionally i'm not sure is that a problem with my specific terminal but i've noticed that docker-compose run does not start terminal prompt until I type something into it. In the gif you can see me typing pwd otherwise it just sits there without prompt forever.

[hugochinchilla]

I was blaming docker lately for being so slow for running phpunit, but tested it without docker-compose and was blazing fast, docker-compose is not usable for running tests, it takes more time for the container to actually run than the time needed for the tests. This needs to be fixed.

[Maxim-Filimonov]

@hugochinchilla Are you running on Mac OS X too? I've noticed that this problem is not present on Ubuntu.

[hugochinchilla]

No, I'm running ubuntu.

[kikicarbonell]

I have the same problem running cucumber and rspec test on my rails app container with docker-compose on Ubuntu 14.04

[detierno]

+1 on Mac OS X

[karellm]

:+1: The commands are really slow for me too using rspec.

[MZAWeb]

+1. About 10x slower for me.

DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"
Docker version 1.5.0, build a8a31ef
docker-compose 1.1.0

[lluisball]

+1 Same problem here.

Docker version 1.5.0, build a8a31ef
docker-compose 1.2.0rc3
Ubuntu 13.10

[mgcpro]

+1 the same problem here

[dnephin]

Most of the slowness I've seen with compose will be fixed by #1066. If you have a large number of containers or images, it can take a while for compose to perform operations.

However, I don't think that matches the problem in the original post. There seems to be some delay before it even runs the first docker api call. I'm not sure what that might be. I notice from the log that the hostname is http://docker.local:2375 instead of the socket path. I wonder if that might be related.

For anyone +1'ing this issue, could you include more details about how it is slow? If you run with --verbose is there an initial delay before any output, or is it slow to execute each docker remote api call?

If anyone is familiar with python and able to provide a cprofile output with the results of a slow run, that would be extremely useful for debugging this issue.

[Maxim-Filimonov]

@dnephin hostname docker.local:2375 is due the the fact that I use docker inside virtual machine. docker.local maps to an ip of virtual machine (static via /etc/hosts). I can imagine if there was a problem with that connection docker run would be slow too, wouldn't it be?

[thaJeztah]

@Maxim-Filimonov perhaps you are able to ssh into the virtual machine and try to reproduce when running compose _inside_ the VM, using a socket connection. That would narrow down the cause.

Also, are you able to add the output of docker version and docker -D info?

[paurullan]

In my case I am not running docker inside a virtual machine.

This is the command I use to launch the tests and the url with the output

docker-compose --verbose -f ../docker-compose.yml run --rm 
  phpunit -c /data/Habitissimo/phpunit.xml 
             /data/Habitissimo/Application/Tests/Album/CreateAlbumTest.php

log:
https://gist.github.com/paurullan/02ca372009eb587f1bef

[Maxim-Filimonov]

Docker info (from inside virtual machine)

> docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

> docker -D info
Containers: 192
Images: 334
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 718
Execution Driver: native-0.2
Kernel Version: 3.13.0-24-generic
Operating System: Ubuntu precise (12.04.4 LTS)
CPUs: 4
Total Memory: 1.954 GiB
Name: docker
ID: RECB:WLXH:R5JJ:AOTU:A3CS:LEN2:TIQX:EDLP:7XYK:V3R7:S4ZJ:RKOS
Debug mode (server): false
Debug mode (client): true
Fds: 45
Goroutines: 60
EventsListeners: 0
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker

I have tried to run it inside virtual machine and it seems to be significantly faster then running from mac os x. Logs
I have recorded some visual comparison

[zbyte64]

Not sure if this is the correct issue to post this but I've noticed considerable lag when building images. docker-compose build will sit for minutes sporadically on various steps. While I sit waiting 10 minutes for a step to complete the target machine is pretty much idling, CPU usage being well under 1% with a Gig of ram to spare. It won't consistently lag on a particular step, it will run at "normal" speed and then decide to take a really long breather.

Server:

~$ sudo docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64

~$ sudo docker -D info
Containers: 7
Images: 127
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 141
 Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.13.0-40-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 2
Total Memory: 3.859 GiB
Name: webinator
ID: GWVS:RVMS:YDAI:7RIG:BA4A:GP3P:XFWL:7R4Z:ST5E:M7HG:CU2G:J2ML
Debug mode (server): false
Debug mode (client): true
Fds: 70
Goroutines: 275
System Time: Thu Apr 30 17:53:47 UTC 2015
EventsListeners: 0
Init SHA1: 9145575052383dbf64cede3bac278606472e027c
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support
Labels:
 provider=amazonec2

Client:

jason@jason-Lemur-Ultra:~/Repos/project-webinator/maker$ docker-compose --version
docker-compose 1.2.0
jason@jason-Lemur-Ultra:~/Repos/project-webinator/maker$ docker --version
Docker version 1.6.0, build 4749651

[zakhenry]

:+1: Same issue, running verbose shows a huge hang at this point:

...
docker resize <- (u'bc41b5295797ea3d3665a48fb7034d9ec26595dc3e7966f07ff58eaa5bbb3a27', width=179, height=81)
                                                                                                            docker resize -> None

However when I run another container I get near instantaneous loading. Neither container (the slow one or the fast one) have links to any other containers.

[zakhenry]

Actually scratch my above comment. Turns out my issue was due to using shared volumes. The container ran a gulp task runner, which involved a lot of reads on init to load plugins. The speed impact was due to the deeply nested nature of npm. Container speed improved massively when using NFS (as expected).

[Maxim-Filimonov]

Cannot replicate this issue with docker-compose 1.2.0 anymore. I presume it's fixed :+1:

[paurullan]

In my case I was always using docker-compose 1.2.0.

[OpakAlex]

same issue on mac.
docker-compose version 1.5.2, build 7240ff3

[thejpanganiban]

I'm still having the same issue on osx. I'm currently using the docker beta.

docker-compose version 1.7.0
docker version 1.11.0

Tried the DNS workaround to no avail.

[Edit 1]
All commands seem to be slow (compose logs, compose ps, etc). Docker client is fast though.

[smyth64]

Got the same issue like @thejpanganiban

[thaJeztah]

@thejpanganiban @smith64fx can you report that issue on the Beta for Mac feedback forum; https://forums.docker.com/c/docker-for-mac ?

[Erwyn]

I have docker-compose 1.7 on Mac OSX. I figured out that it comes (in my case) from my home network, but I cannot find why:

• docker-compose up take ages to run
• shut down my wifi
• docker-compose up is REALLY fast
• now I can re-enable my wifi

So it clearly comes from my home network, as I cannot reproduce it from work network. But what is happening there…

The previous version of the beta docker for Mac OS also had problems with my network (could not pull any images), but since the last update of the beta, no issue anymore. I don't know if this is related or not, just giving everything I observed.

[thaJeztah]

@Erwyn is that on Docker Toolbox or Docker for Mac beta?

[Erwyn]

This is from docker beta, but I haven't tried docker-compose with the docker toolbox, doing it right now

[thaJeztah]

@Erwyn thanks, if you can post the version of your Docker for Mac beta (from the "about" pane), that'd be useful

[Erwyn]

Back, sorry for the time, it had to build/pull whatever some stuff. So @thaJeztah I confirm it is not reproduced with the docker toolbox, so basically it only occurs with docker for mac beta.

My docker for mac beta version is Version 1.11.1-beta10 (build: 6662)

And as I said before I also had a (my) network related issue but with the docker client. This issue also occured with the docker for mac beta and has been solved since one of the last updates.

[thaJeztah]

@Erwyn thanks! Passing it on to the Docker for Mac team now

[Erwyn]

Thanks!

[thaJeztah]

@Erwyn just discussed some things with the team; first of all, can you open a new issue? It seems unrelated to the main issue discussed here, so better to keep it separate. Would you also be able to share your docker-compose file (if possible)?

[Erwyn]

@thaJeztah Yes, sure!. The issue is #3419

[davidchua]

Hi guys,

I'm having similar issues of docker-compose run taking a really long time.

docker-compose 1.7.1 build 0a9ab35
docker server: 1.11.1

[thnery]

I ran $ time docker-compose ps and it took 3:31 min with system 0% cpu. So, I tried changing my DNS, according to this discussion https://forums.docker.com/t/docker-compose-is-extremely-slow/8690/7
Then I ran the same command again and it took 0,7 sec with system 56% cpu

PS: I used opendns instead of google dns

[jperry]

I am also running into a 2X performance hit when comparing docker-compose up vs docker-compose run. I have a mongo, rabbit, mysql, and test containers where the test container interacts with mongo, rabbit and mysql. We run java gradle unit tests in this test container that produces a lot of log output to STDOUT. When I run using "up" the tests take about 12 minutes when using "run" it takes 24 minutes to execute. I was preferring the "run" option as I plan to execute this in jenkins but considering the performance issues I'll have to use "up" and put the process in the background and monitor it until it's done before finishing the jenkins job. Is there a chance when you use "run" it uses a different network resulting in slower communication as it does talk to mongo and rabbit a lot. Any ideas or suggested approaches for this?

[chicosilva]

I solveld this question staying my Dockerfile in sub-folder.

php:
restart: always
build: .
ports:

• "80:80"
• "443:443"
  volumes:
• .:/var/www/html
  links:
  
  
  
  • db
  
  

db:
restart: always
image: mysql:5.6
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: 123
DB_NAME: pass
DB_PASS: 123
DB_USER: pass
volumes:
- ./mysql/databases:/var/lib/mysql
- ./logs/mysql:/var/log/mysql

to

php:
restart: always
build: ./php
ports:

• "80:80"
• "443:443"
  volumes:
• .:/var/www/html
  links:
  
  
  
  • db
  
  

db:
restart: always
image: mysql:5.6
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: pass
DB_NAME: test
DB_PASS: pass
DB_USER: test
volumes:
- ./mysql/databases:/var/lib/mysql
- ./logs/mysql:/var/log/mysql

[gucki]

Same issue here with latest docker and latest docker-compose. Starting a rails console takes 12s when docker-compose up is used and about twice the time when docker-compose run is used. Really strange. How can I help to debug this?

Hint: If I add privileged: true to the docker-compose.yml, up and run are equally fast!

[DMW007]

Same issue on my Raspberry Pi 3 using Raspbian. Even on simple commands like showing the version, where not much work should be done, I got very slow responsiness:

pi@pi3:~ $ time docker --version
Docker version 17.09.0-ce, build afdb6d4

real    0m0,055s
user    0m0,030s
sys     0m0,020s
pi@pi3:~ $ time docker-compose --version
docker-compose version 1.16.1, build 6d1ac219

real    0m6,717s
user    0m6,380s
sys     0m0,260s

I installed docker-compose using pip as recommended in the official documentation. The latest Raspbian image at the time of writing is used:

pi@pi3:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.1 (stretch)
Release:        9.1
Codename:       stretch

[fracz]

+1 the same problem on Raspberry as @DMW007

[a-toms]

I was blaming docker lately for being so slow for running phpunit, but tested it without docker-compose and was blazing fast, docker-compose is not usable for running tests, it takes more time for the container to actually run than the time needed for the tests. This needs to be fixed.

Any suggestions in 2019 regarding how to run the tests without the very slow setup of docker-compose?