Compose: docker-compose often takes a long time to do anything

[Mossop]

Description of the issue

docker-compose takes a long time to run some functions that I would expect to be very fast. For example, the last time I timed running docker-compose --help it took three and a half minutes before outputting the help and finishing.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.24.0-rc1, build 0f3d4dda
docker-py version: 3.7.0
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018

Output of docker version

Client:
 Version:           18.09.3
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        774a1f4
 Built:             Thu Feb 28 06:47:20 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       774a1f4
  Built:            Thu Feb 28 05:59:55 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

networks:
  web:
    driver: bridge
services:
  nginx:
    container_name: nginx
    image: nginx:alpine
    networks:
      web: {}
    ports:
    - 80:80/tcp
    - 443:443/tcp
    restart: always
    volumes:
    - /srv/docker/nginx/conf.d:/etc/nginx/conf.d:ro
    - /src/docker/nginx/hosts.d:/etc/nginx/hosts.d:ro
    - /srv/docker/nginx/certificates:/etc/nginx/certificates:ro
    - /srv/docker/nginx/logs:/var/log/nginx:rw
    - /srv/www:/srv/www:rw
version: '3.0'

Steps to reproduce the issue

1. Run time docker-compose --help

Observed result

See help output followed by:

real    3m33.025s
user    0m0.692s
sys 0m0.127s

Expected result

See help output almost immediately.

Stacktrace / full error message

No error message displayed

Additional information

OS version / distribution, docker-compose install method, etc.

Ubuntu 18.10 running on a standard digital ocean droplet. Installed using the method recommended.

[Mossop]

I also see this with using the run.sh container version of docker-compose.

[ijc]

Thank you for reporting. For me I see:

real    0m0.558s
user    0m0.459s
sys 0m0.091s

Which seems fine/normal. I guess there must be something about your local configuration which trips things up somewhere.

I'm afraid this probably means you'll need to dig in a bit deeper and see if you can find what is slow. (My gut reaction in these cases is usually something networky, like DNS, although I don't know if that would apply to compse).

You could try e.g. strace -r docker-compose --help and see if anything leaps out as taking a very long time.

[Mossop]

Everything in strace looks reasonable apart from four lines looking like this:

29.398716 futex(0x55af6046b0e8, FUTEX_WAIT, 0, NULL) = 0

The help output shows immediately after the fourth.

[ijc]

futex suggests it is blocking somewhere else, then when it is unblocked the futex is released allowing it to continue to the help. If you post the strace log(s) I can see if I can divine anything, but otherwise I'm afraid you'll need to keep digging on your end.

Perhaps python -m trace -t «path»/docker-compose --help will give a clue?

[Mossop]

Perhaps python -m trace -t «path»/docker-compose --help will give a clue?

I looked at this, but docker-compose is a binary file, not a python script so I think this is wrong?

[ijc]

compose is written in Python, but it is (in some circumstances) shipped as a https://www.pyinstaller.org/ bundle.

You mentioned that you also saw this with the run.sh/containerized version and I mistakenly thought that case didn't use the pyinstaller form (so discounted that case), but i was wrong and the container does in fact use pyinstaller.

There's certainly potential for that to introduce startup overhead, but I've no idea how one debugs such things.

[Mossop]

Well, I wish I knew how but somehow this seems to have resolved itself. I haven't seen it act slow for a few days now and I've been using it a bunch. Not sure if you want to leave this open or not, but feel free to close it.

[ijc]

Mysterious! Thanks for letting us know. I will close but of course if it reoccurs do feel free to come back.

[ntindle]

I am also having this issue.

[kassanmoor]

I have this issue too, any workaraound? why the issue is clossed @Mossop

[ijc]

Most likely this was the same issue as #6678 i.e. lack of entropy on a cloud host (digital ocean here) or similar.

[rudyryk]

Sometimes docker-compose command just hangs up until Ctrl-C. Looks like some initialization issues with crypto libs, here's the traceback:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 7, in <module>
    from compose.cli.main import main
  File "/usr/local/lib/python3.6/dist-packages/compose/cli/main.py", line 17, in <module>
    import docker
  File "/usr/local/lib/python3.6/dist-packages/docker/__init__.py", line 2, in <module>
    from .api import APIClient
  File "/usr/local/lib/python3.6/dist-packages/docker/api/__init__.py", line 2, in <module>
    from .client import APIClient
  File "/usr/local/lib/python3.6/dist-packages/docker/api/client.py", line 10, in <module>
    from .build import BuildApiMixin
  File "/usr/local/lib/python3.6/dist-packages/docker/api/build.py", line 6, in <module>
    from .. import auth
  File "/usr/local/lib/python3.6/dist-packages/docker/auth.py", line 9, in <module>
    from .utils import config
  File "/usr/local/lib/python3.6/dist-packages/docker/utils/__init__.py", line 3, in <module>
    from .decorators import check_resource, minimum_version, update_headers
  File "/usr/local/lib/python3.6/dist-packages/docker/utils/decorators.py", line 4, in <module>
    from . import utils
  File "/usr/local/lib/python3.6/dist-packages/docker/utils/utils.py", line 13, in <module>
    from .. import tls
  File "/usr/local/lib/python3.6/dist-packages/docker/tls.py", line 5, in <module>
    from .transport import SSLHTTPAdapter
  File "/usr/local/lib/python3.6/dist-packages/docker/transport/__init__.py", line 11, in <module>
    from .sshconn import SSHHTTPAdapter
  File "/usr/local/lib/python3.6/dist-packages/docker/transport/sshconn.py", line 1, in <module>
    import paramiko
  File "/usr/local/lib/python3.6/dist-packages/paramiko/__init__.py", line 22, in <module>
    from paramiko.transport import SecurityOptions, Transport
  File "/usr/local/lib/python3.6/dist-packages/paramiko/transport.py", line 90, in <module>
    from paramiko.ed25519key import Ed25519Key
  File "/usr/local/lib/python3.6/dist-packages/paramiko/ed25519key.py", line 22, in <module>
    import nacl.signing
  File "/usr/local/lib/python3.6/dist-packages/nacl/signing.py", line 17, in <module>
    import nacl.bindings
  File "/usr/local/lib/python3.6/dist-packages/nacl/bindings/__init__.py", line 377, in <module>
    sodium_init()
  File "/usr/local/lib/python3.6/dist-packages/nacl/bindings/sodium_core.py", line 32, in sodium_init
    ffi.init_once(_sodium_init, "libsodium")
  File "/usr/local/lib/python3.6/dist-packages/nacl/bindings/sodium_core.py", line 22, in _sodium_init
    ensure(lib.sodium_init() != -1,
KeyboardInterrupt

[Phuker]

I have the same problem with @rudyryk .
This problem won't happen 100% of the times, but about 10 % - 30%.
I tried to trace it by run:

python -m cProfile `which docker-compose` --version

there is a lot of output, here is lines contains 112.:

docker-compose version 1.24.1, build 4667896
         130382 function calls (126780 primitive calls) in 112.339 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 <string>:1(ArgInfo)
        ......
        2    0.001    0.000  112.085   56.043 __init__.py:15(<module>)
        1    0.001    0.001  112.142  112.142 __init__.py:20(<module>)
        1    0.005    0.005  112.152  112.152 auth.py:1(<module>)
        2    0.001    0.000  112.153   56.077 build.py:1(<module>)
        2    0.003    0.001  112.211   56.106 client.py:1(<module>)
        1    0.000    0.000  112.144  112.144 decorators.py:1(<module>)
        1    0.001    0.001  112.343  112.343 docker-compose:3(<module>)
        1    0.000    0.000  112.087  112.087 ed25519key.py:17(<module>)
        1    0.003    0.003  112.333  112.333 main.py:1(<module>)
        1    0.000    0.000  112.086  112.086 signing.py:15(<module>)
        1    0.000    0.000  112.084  112.084 sodium_core.py:21(_sodium_init)
        1    0.000    0.000  112.084  112.084 sodium_core.py:27(sodium_init)
        1    0.000    0.000  112.142  112.142 sshconn.py:1(<module>)
        1    0.000    0.000  112.143  112.143 tls.py:1(<module>)
        1    0.004    0.004  112.139  112.139 transport.py:22(<module>)
        4    0.000    0.000  112.144   28.036 utils.py:1(<module>)
        1  112.084  112.084  112.084  112.084 {built-in method sodium_init}
        1    0.000    0.000  112.084  112.084 {method 'init_once' of 'CompiledFFI' objects}
        ......

It seems like that this problem is related with sodium_init()

Additional information

18.04.2 LTS (Bionic Beaver)
Linux 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 
GNU/Linux
Docker version 18.09.7, build 2d0083d
docker-compose version 1.24.1, build 4667896

# dpkg -l | grep sodium
ii  libsodium23:amd64     1.0.16-2     amd64   Network communication, cryptography and signaturing library

# python --version
Python 2.7.15+

[Phuker]

I was wondering what does docker-compose have to do with libsodium, since libsodium is a library about networking and cryptography. And then I found out that the python library docker and docker-compose did a lot of work that not everyone need.
libsodium(pynacl) is used to connect to the Docker Engine over SSH in the library paramiko, and paramiko is imported in docker.
Here is my ugly hack to get rid of pynacl, libsodium, paramiko and anything else that you never need.
Just patch /usr/local/lib/<YOUR_PYTHON_VERSION>/dist-packages/docker/transport/__init__.py like this:

try:
    SSHHTTPAdapter = None
    # from .sshconn import SSHHTTPAdapter
except ImportError:
    pass

and docker-compose won't hang up anymore.

[beat2]

There is an explanation here about libsodium and entropy with proposed solutions: https://libsodium.gitbook.io/doc/usage#sodium_init-stalling-on-linux

From the quoted article, to know if you suffer of low entropy:
cat /proc/sys/kernel/random/entropy_avail
If the number is less than 160, this is the issue.

[immanuelfodor]

For me the solution was to install haveged and it solved the issue out of the box:

cat /proc/sys/kernel/random/entropy_avail
# around 5

apt install -y haveged
systemctl status haveged.service

cat /proc/sys/kernel/random/entropy_avail
# 2400 or more

[lonix1]

cc @phuker and others:

If this was a problem for you please upvote https://github.com/docker/compose/issues/6931 so we can get better docs, which would have avoided lots of wasted time chasing down this problem.

[meabed]

Its also very slow on mac, hanging before doing any task 10-30 seconds. any workaround or fixes?

$ time docker-compose --help 
# docker-compose --help  0.36s user 0.11s system 4% cpu 10.129 total

$ docker-compose version
docker-compose version 1.24.1, build 4667896b
docker-py version: 3.7.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018



md5-6d5af6c72c02d1dcc3e796839c12ec4f



$ pip3 install docker-compose 

$ docker-compose version
docker-compose version 1.24.1, build 4667896
docker-py version: 3.7.3
CPython version: 3.7.4
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

[gendelbendel]

@meabed Thanks for the tip, I was experiencing the same issues and it appears as if getting the latest version and installing via pip resolved the slowness for me.

[l3ender]

Just patch /usr/local/lib/<YOUR_PYTHON_VERSION>/dist-packages/docker/transport/__init__.py

@Phuker I'm experiencing the same issue using Docker Desktop on my Mac. Is there a similar file I can patch to avoid the issue? I can't seem to find one after poking around /usr/local/lib/<my_python> or /Applications/Docker.app.

[Phuker]

@l3ender I have no idea, never use Docker Desktop for Mac before.
According to the docs:

On desktop systems like Docker Desktop for Mac and Windows, Docker Compose is included as part of those desktop installs.

Maybe it is located at /Applications/Docker.app? You can find it out and test whether this patch works.

[heraldovalenti]

Hey there,

I'm facing the same issue on a mac:

$ time docker-compose version
docker-compose version 1.24.1, build 4667896b
docker-py version: 3.7.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018

real    0m25.013s
user    0m0.562s
sys 0m0.210s

I am trying to use the docker-compose installed from pip3 but I'm not sure how. I've already installed with pip3 install docker-compose but docker-compose still points to the binary installed with the Docker Bundle:

$ which docker-compose
/usr/local/bin/docker-compose

$ ls -l /usr/local/bin/docker-compose
lrwxr-xr-x  1 root  admin  62 Oct  2 18:57 /usr/local/bin/docker-compose -> /Applications/Docker.app/Contents/Resources/bin/docker-compose

How do I use the docker-compose installed with pip?
Any other information I can provide just let me know.
Thanks a lot!

[Martlark]

I had too many files that docker-compose had to examine to build. I fixed it by removing things that should not be copied using .dockerignore.

[LouisWayne]

DigitalOcean user here I have the same problem. Running the below command:

cat /proc/sys/kernel/random/entropy_avail

Returns 18 - which is an issue

[nikkhouy]

For Mac users, who are using docker desktop:
My docker-compose version or any docker-compose related commands was taking forever to run, I replaced the docker-compose from docker desktop by brew:

brew install docker-compose
brew link --overwrite docker-compose

[chovy]

I'm on linux. My entropy is 3745. I'm still getting slow deploys as of a few days ago. Nothing changed on my server besides rebooting it. Server load is about 2.3 and ram is free.

[LiamKarlMitchell]

Same here, a build can take 18+ minutes and its stuck at Building nameofcontainer

Don't know if it is coincidence or not, but today the moment I ssh'd in with another putty terminal the build continued as expected.

This is on a linux server.

[merit-mthompson]

Was having a slow-down similar to what @LiamKarlMitchell is describing (read: on a Linux host) with compose 1.26.0 and docker 19.03.11. What seems to have fixed things for me was to edit my /etc/hosts to look like:

127.0.0.1 localhost localunixsocket localunixsocket.local localunixsocket.home

I found related discussions around this helping and without changing anything else this seems to have fixed things for me on a Linux host.

[SteffenAnders]

We have solved the problem by installing docker-compose as a container.
See https://docs.docker.com/compose/install/ → "Alternative Install Options".

sudo curl -L --fail https://github.com/docker/compose/releases/download/1.26.0/run.sh -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

docker-compose --version

[LiamKarlMitchell]

Tried the hosts file trick from @merit-mthompson no noticeable improvement.

[LiamKarlMitchell]

What helped the most was making a .dockerignore file and putting html and other folders like es-data into that so they were ignored from build context.