Compose: docker-compose slow, timing out

[raptor235]

The system was working properly on the toolbox but now after the upgrade and removal of toolbox I'm getting timeouts or very slow builds... 5-10mins +

My docker-compose build is hanging at

``` engine='sh'
[ home/docker ] - $ docker-compose --verbose build app
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: File doesn't exist
compose.cli.command.get_client: docker-compose version 1.8.0, build f3628c7
docker-py version: 1.9.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.2h 3 May 2016
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.4.15-moby, Os=linux, BuildTime=2016-07-28T21:15:28.963402499+00:00, ApiVersion=1.24, Version=1.12.0, GitCommit=8eab29e, Arch=amd64, GoVersion=go1.6.3
compose.service.build: Building app
compose.cli.verbose_proxy.proxy_callable: docker build <- (pull=False, stream=True, nocache=False, tag=u'docker_app', buildargs=None, rm=True, forcerm=False, path='/Users/bartdabek/Sites/docker', dockerfile='Dockerfile-app')
docker.api.build._set_auth_headers: Looking for auth config
docker.api.build._set_auth_headers: No auth config in memory - loading from filesystem
docker.auth.auth.load_config: File doesn't exist
docker.api.build._set_auth_headers: No auth config found

docker-compose up is failing 

``` engine='sh'
[ home/docker ] - $ docker-compose --verbose up app
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: File doesn't exist
compose.cli.command.get_client: docker-compose version 1.8.0, build f3628c7
docker-py version: 1.9.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.2h  3 May 2016
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.4.15-moby, Os=linux, BuildTime=2016-07-28T21:15:28.963402499+00:00, ApiVersion=1.24, Version=1.12.0, GitCommit=8eab29e, Arch=amd64, GoVersion=go1.6.3
compose.cli.verbose_proxy.proxy_callable: docker info <- ()
compose.cli.verbose_proxy.proxy_callable: docker info -> {u'Architecture': u'x86_64',
 u'BridgeNfIp6tables': True,
 u'BridgeNfIptables': True,
 u'CPUSet': True,
 u'CPUShares': True,
 u'CgroupDriver': u'cgroupfs',
 u'ClusterAdvertise': u'',
 u'ClusterStore': u'',
 u'Containers': 0,
 u'ContainersPaused': 0,
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- (u'docker_default')
compose.network.ensure: Creating network "docker_default" with the default driver
compose.cli.verbose_proxy.proxy_callable: docker create_network <- (ipam=None, driver=None, options=None, name=u'docker_default')
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 61, in main
  File "compose/cli/main.py", line 113, in perform_command
  File "contextlib.py", line 35, in __exit__
  File "compose/cli/errors.py", line 56, in handle_connection_errors
TypeError: log_timeout_error() takes exactly 1 argument (0 given)
docker-compose returned -1

I've tried all the tricks people have been mentioning of using google dns, turning off wifi, adding hosts entries to include

127.0.0.1 localunixsocket
127.0.0.1 localunixsocket.local
127.0.0.1 localunixsocket.home

nothing is helping...

The amount of time I've spent over the past 6 months fighting docker is soo frustrating...

[christianbundy]

I don't have a solution to your issue, but it would probably be useful to used fenced code blocks so that others can read your multi-line content. For example, using the console highlighter:

$ docker --version
Docker version 1.12.0, build 8eab29e

[cerebrotecnologico]

I have the same problem. I work with docker under OS X El Capitan.
7: Pulling from library/java 357ea8c3d80b: Pull complete 52befadefd24: Pull complete 3c0732d5313c: Pull complete 557cb7f84eb9: Pull complete 7bbd9fac5727: Pull complete 15f5ec8580f1: Downloading [===================================> ] 97.84 MB/139.6 MB
It downloads some parts pretty quick. Then it gets stuck.
I'm using:
Docker version 1.11.2, build b9f10c9
docker-compose version 1.7.1, build 0a9ab35

I also modified my /etc/hosts without success.

[christianbundy]

@cerebrotecnologico I don't think this is at all the same issue.

[maackle]

I'm having the same issue as @raptor235, also having tried all the tricks mentioned in various other issues to no avail. I also wish he would edit his post to use fenced code blocks so we could read the multi-line output.

[raptor235]

updated :)

[maackle]

I just resolved my problem. It probably wasn't the same issue, because in my case it was just that I had a very large directory that was unnecessarily getting ADDed by docker-compose, causing massive delays.

[xeger]

I'm having a similar issue to @raptor235; my docker-compose commands eventually complete, but they take an astoundingly long time -- commands that used to be instantaneous, such as logs or up, take a minute or more.

I'm using Docker for Mac and the issue began happening after an upgrade to 1.12.1-beta25 (build: 11807). The weird thing is, this slowness _only_ impacts Compose; if I use docker from the OS X CLI, it's just as zippy as ever. So, there's some sort of bad interaction between Compose and Docker for Mac (or its numerous proxy processes and other guest-to-host glue.)

Compose version info:

docker-compose version 1.8.0, build f3628c7
docker-py version: 1.9.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.2h  3 May 2016

[christianbundy]

@xeger Can you post the output of a command running with the --verbose flag? Thanks!

[xeger]

Sure; it's rather verbose, so I attached it:

slow-docker-compose.txt

I ran stdout/stderr through ts so you can see where the time is going. Something _very_ interesting is that output always happens exactly on a 5-second boundary! This might be an artifact of the pipe and/or the ts command, but a timer confirms that the total time reported by ts is correct.

[christianbundy]

@xeger Is that GOOGLE_API_CLIENT_SECRET real? If so, you should probably rotate your keys now that it's been published in the open. Fingers crossed it's just a fake, but it looks pretty convincing. Sorry to be the bearer of bad news.

[xeger]

D'oh! Thank you for pointing out the exposure. Not an issue in this case because the project isn't billing-enabled; still, had I ever flipped it on ... :(

Update about the issue: this slowness happens for me even when DOCKER_HOST is pointing to a nearby Linux machine! That more-or-less rules out my initial hypothesis about Docker for Mac.

No new hypothesis yet. I'm going to do an apples-to-apples timing comparison just to confirm what I'm seeing.

[xeger]

Nevermind; the original Linux/Mac test was invalid; I didn't notice that the Linux machine ended up pulling some images.

With a trivial project file, our apples-to-apples comparison gives us:

# OS X talking to Docker via docker.sock
time docker-compose --verbose up -d test
       56.63 real         0.27 user         0.06 sys

# OSX talking to remote daemon via tcp
time docker-compose --verbose up -d test
        6.15 real         0.24 user         0.05 sys

The --verbose output and its timestamps confirm the pattern that I saw at first: about 5 seconds elapse between blocks of activity relating to docker requests. New output attached, with 100% fewer secrets:
slow-docker-compose.txt

[xeger]

One more salient detail: I just switched locations, and also switched network interfaces from Ethernet (with WiFi disabled) to my Mac's built-in WiFi interface. The performance problem has disappeared.

This is very good evidence that the root cause is some weird interaction between Docker for Mac's VM, its networking glue, and my host -- and that the cause is either related to routing or interface snooping.

In any case, it's obvious that _my_ particular slowness issue isn't closely related to Docker Compose. If I discover anything noteworthy I'll file a ticket with the DfM team.

[sha1sum]

I was having this issue, and as @xeger points out, switching networks did the trick. In our case, we have found that the issue is resolved by changing the DNS server (we used Google's 8.8.8.8). Don't ask me why everything else works perfectly fine except docker-compose, but you may want to look into it.

[herecydev]

Seeing the same problems listed here and have tried running a docker-compose --verbose up
There's a seemingly fixed 5s delay between these lines, as others have pointed out it's probably a hard timeout:

docker.api.build._set_auth_headers: Sending auth config (u'https://index.docker.io/v1/')
compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object _stream_helper at 0x0000000003EA39D8>

Even trying something like docker-compose ps is incredibly slow. You can see the slowness is in the same sort of areas:

compose.config.config.find: Using configuration files: .\docker-compose.yml
docker.auth.find_config_file: Trying paths: [foo]
docker.auth.find_config_file: Found file at path: foo\config.json
docker.auth.load_config: Found 'auths' section
docker.auth.parse_auth: Auth data for {0} is absent. Client might be using a credentials store instead.
docker.auth.load_config: Found 'credsStore' section
**compose.cli.command.get_client: docker-compose version 1.11.2, build f963d76f**
docker-py version: 2.1.0
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.2j  26 Sep 2016
compose.cli.command.get_client: Docker base_url: http+docker://localnpipe
compose.cli.command.get_client: Docker version: KernelVersion=4.9.13-moby, Arch=amd64, BuildTime=2017-03-15T20:28:18.193664702+00:00, ApiVersion=1.27, Version=
17.03.1-ce-rc1, MinAPIVersion=1.12, GitCommit=3476dbf, Os=linux, Experimental=True, GoVersion=go1.7.5
**compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=foo', u'com.docker.compose.oneoff=
False']})**
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={u'label': [u'com.docker.compose.project=foo', u'com.docker.compose.oneoff
=True']})
**compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)**
Name   Command   State   Ports
------------------------------

I've highlighted the 3 problem lines, and they all come from compose.cli. Running a docker ps completes immediately. What else can I do to highlight the issue? This is absolutely crushing the experience.

[shin-]

Closing as duplicate of #3419

Should be fixed in recent versions of Compose.

[luckydonald]

For me, using the gzip compression fixed it.

docker-compose --verbose build --compress {service}