Compose: Pip install of docker-compose 1.24 fails

[carlmjohnson]

Description of the issue

Running pip install docker-compose==1.23.2 succeeds but pip install docker-compose fails.

Context information (for bug reports)

Error output is: /usr/bin/python2 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-Jeq6Db --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel "cffi>=1.1; python_implementation != 'PyPy'"" failed with error code 1 in None

Steps to reproduce the issue

Against docker:stable image run apk add --quiet --no-cache bash py-pip && pip install --quiet docker-compose==1.24.0

Observed result

/usr/bin/python2 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-Jeq6Db --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel "cffi>=1.1; python_implementation != 'PyPy'"" failed with error code 1 in None`

Expected result

Successful install.

Additional information

Running Docker-in-Docker for Gitlab test runner. Probably not relevant?

[AnthonyMastrean]

@carlmjohnson are you running inside of an Alpine container? I am and I'm seeing the same issue. The top half of my .gitlab-ci.yml file looked like this:

image: docker:stable

services:
  - docker:stable-dind

before_script:
  - apk add --no-cache --quiet py-pip
  - pip install --quiet docker-compose
  - docker login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

But, I had to change it to pin the Compose version.

image: docker:stable

services:
  - docker:stable-dind

before_script:
  - apk add --no-cache --quiet py-pip
  - pip install --quiet docker-compose~=1.23.0
  - docker login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

[carlmjohnson]

Yes, my .gitlab-ci.yml was essentially the same, and my fix has been to use the prior version of docker-compose.

[outofmemoryagain]

I can confirm the same issue, and same fix... revert to previous version.

[kynetiv]

Ran into this as well in gitlab-ci. I added a few more dependencies to fix my issue with the 1.24 release. Although, May be better to pin to the previous as others mentioned.

image: docker:stable

before_script:
  - apk update
  - apk upgrade
  - apk add python python-dev py-pip build-base libffi-dev openssl-dev libgcc
  - pip install docker-compose

Notably, i had to add libffi-dev openssl-dev libgcc to my previous dependencies.

[kball]

Also ran into this in bitbucket pipelines; pinning to 1.23.2 resolves there as well.

[cdunford]

Same issue with gitlab-ci in docker:latest; reverting to 1.23.2 resolved it.

[ju5t]

A changelog would have been nice. Same issue here.

[samson-s]

Same issue with gitlab-ci in docker:stable.

[lukad]

Please document breaking changes like these. My builds started failing out of nowhere. It's not documented in the official installation instructions and there's not even a changelog.
This blows.

[reverseuu]

Same issue using gitlab dind with image docker:stable.

before_script:
  - apk add --no-cache py-pip
  - pip install docker-compose

suggested solution with docker-compose fix version resolves it.

before_script:
  - apk add --no-cache py-pip
  - pip install docker-compose~=1.23.0

[BottlecapDave]

I have also been hit by this issue with CI builds that started failing. Wasted a good hour until I came across this thread. Thanks to those that posted solutions :)

[lukad]

So I looked at the other installation method (downloading a precompiled binary) but it doesn't work in docker:stable. The binary is linked against glibc for which alpine has the libc6-compat package but there are still some symbols that can't be resolved. For now installing docker-compose==1.23.2 via pip is the best solution.

[WolfspiritM]

Same issue here. However I moved to using the "official" installation method.
Even in the alpine image.
I'm installing glibc like:
https://github.com/docker/compose/blob/master/Dockerfile.run
Just don't do the "apk del curl" in the end.
And then using:

curl -L https://github.com/docker/compose/releases/download/1.24.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

To download docker-compose.

[ulyssessouza]

This is related to the new SSH support which added some new dependencies.
On our test systems these dependencies are already available once pip is installed.

Unfortunately on alpine this is not the case and you need to install them manually:

$ apk add py-pip python-dev libffi-dev openssl-dev gcc libc-dev make

We will work on updating the documentation ASAP. Apologies for the inconvenience.

As an alternative, docker-compose is also released as Docker image, so a quick fix would be replacing docker:stable by docker/compose:1.24.0 as base image.

[AnthonyMastrean]

@ulyssessouza thanks for the explanation.

It was extremely convenient and fast to install just the docker-compose package. I'm loathe to add six indirect development dependencies to my application's build. The effect on build time is noticeable. Of course, this isn't a good benchmark, but I hope it reflects the general trend.

$ docker run --rm -it docker:stable time sh -c 'apk add --no-cache --quiet py-pip && pip install --quiet docker-compose~=1.23.0'
real    0m 7.09s
user    0m 3.67s
sys     0m 0.76s

$ docker run --rm -it docker:stable time sh -c 'apk add --no-cache --quiet py-pip python-dev libffi-dev openssl-dev gcc libc-dev make && pip install --quiet docker-compose'
real    1m 38.93s
user    1m 7.77s
sys     0m 7.44s



md5-c2434814f58b461b5e8e78fb9ff5f3c0

Running with gitlab-runner 11.0.0 (5396d320)
on nextgen-aws-east1d-autoscale.amazonaws.com ec265edf
Using Docker executor with image docker/compose:1.23.2 ...
Starting service docker:stable-dind ...
Pulling docker image docker:stable-dind ...
Using docker image sha256:85e924caedbd3e5245ad95cc7471168e923391b22dcb559decebe4a378a06939 for docker:stable-dind ...
Waiting for services to be up and running...
Pulling docker image docker/compose:1.23.2 ...
Using docker image sha256:65f21bf3ffc929410490621982ff648d02f268fefb063f55dc3129eb70a9cdf3 for docker/compose:1.23.2 ...
Running on runner-ec265edf-project-10964250-concurrent-0 via runner-ec265edf-gitlab-docker-machine-1553872466-7e37c8d8...
Cloning repository...
Cloning into '/builds/vaeit/nextgen/itpie-api'...
Checking out d073b9e2 as infra-457-compose...
Skipping Git submodules setup
No such command: sh
...
ERROR: Job failed: exit code 1

 * I don't think the docker/compose image is compatible with docker:stable-dind, the way we're supposed to use it with GitLab's CI, even if I override the ENTRYPOINT problem.
```yaml
image: 
  name: docker/compose:1.23.2
  entrypoint: [""]

services:
  - docker:stable-dind

before_script:
  - docker login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

Running with gitlab-runner 11.0.0 (5396d320)
  on nextgen-aws-east1d-autoscale.amazonaws.com ec265edf
Using Docker executor with image docker/compose:1.23.2 ...
Starting service docker:stable-dind ...
Pulling docker image docker:stable-dind ...
Using docker image sha256:85e924caedbd3e5245ad95cc7471168e923391b22dcb559decebe4a378a06939 for docker:stable-dind ...
Waiting for services to be up and running...
Pulling docker image docker/compose:1.23.2 ...
Using docker image sha256:65f21bf3ffc929410490621982ff648d02f268fefb063f55dc3129eb70a9cdf3 for docker/compose:1.23.2 ...
Running on runner-ec265edf-project-10964250-concurrent-0 via runner-ec265edf-gitlab-docker-machine-1553872466-7e37c8d8...
Cloning repository...
Cloning into '/builds/vaeit/nextgen/itpie-api'...
Checking out 1efaf55f as infra-457-compose...
Skipping Git submodules setup
$ docker login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: Job failed: exit code 1

[fergyfresh]

Running with gitlab-runner 11.7.0~beta.3.g41feb4a1 (41feb4a1)
  on Kubernetes Runner dHhx65ta
Using Kubernetes namespace: gitlab-ci
Using Kubernetes executor with image docker:stable ...

$ pip3 install docker-compose
Collecting docker-compose
  Downloading https://files.pythonhosted.org/packages/51/56/5745e66b33846e92a8814466c163f165a26fadad8b33afe381e8b6c3f652/docker_compose-1.24.0-py2.py3-none-any.whl (134kB)
Collecting cached-property<2,>=1.2.0 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/3b/86/85c1be2e8db9e13ef9a350aecd6dea292bd612fa288c2f40d035bb750ded/cached_property-1.5.1-py2.py3-none-any.whl
Collecting docopt<0.7,>=0.6.1 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz
Collecting texttable<0.10,>=0.9.0 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/02/e1/2565e6b842de7945af0555167d33acfc8a615584ef7abd30d1eae00a4d80/texttable-0.9.1.tar.gz
Collecting dockerpty<0.5,>=0.4.1 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/8d/ee/e9ecce4c32204a6738e0a5d5883d3413794d7498fe8b06f44becc028d3ba/dockerpty-0.4.1.tar.gz
Collecting jsonschema<3,>=2.5.1 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting PyYAML<4.3,>=3.10 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)
Collecting requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/ff/17/5cbb026005115301a8fb2f9b0e3e8d32313142fe8b617070e7baad20554f/requests-2.20.1-py2.py3-none-any.whl (57kB)
Collecting docker[ssh]<4.0,>=3.7.0 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/48/68/c3afca1a5aa8d2997ec3b8ee822a4d752cf85907b321f07ea86888545152/docker-3.7.2-py2.py3-none-any.whl (134kB)
Collecting websocket-client<1.0,>=0.32.0 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/29/19/44753eab1fdb50770ac69605527e8859468f3c0fd7dc5a76dd9c4dbd7906/websocket_client-0.56.0-py2.py3-none-any.whl (200kB)
Collecting six<2,>=1.3.0 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting certifi>=2017.4.17 (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
Collecting urllib3<1.25,>=1.21.1 (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Collecting idna<2.8,>=2.5 (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
Collecting docker-pycreds>=0.4.0 (from docker[ssh]<4.0,>=3.7.0->docker-compose)
  Downloading https://files.pythonhosted.org/packages/f5/e8/f6bd1eee09314e7e6dee49cbe2c5e22314ccdb38db16c9fc72d2fa80d054/docker_pycreds-0.4.0-py2.py3-none-any.whl
Collecting paramiko>=2.4.2; extra == "ssh" (from docker[ssh]<4.0,>=3.7.0->docker-compose)
  Downloading https://files.pythonhosted.org/packages/cf/ae/94e70d49044ccc234bfdba20114fa947d7ba6eb68a2e452d89b920e62227/paramiko-2.4.2-py2.py3-none-any.whl (193kB)
Collecting pynacl>=1.0.1 (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<4.0,>=3.7.0->docker-compose)
  Downloading https://files.pythonhosted.org/packages/61/ab/2ac6dea8489fa713e2b4c6c5b549cc962dd4a842b5998d9e80cf8440b7cd/PyNaCl-1.3.0.tar.gz (3.4MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /usr/bin/python3.6 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dx3r45lk --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel "cffi>=1.4.1; python_implementation != 'PyPy'":
  Collecting setuptools
    Downloading https://files.pythonhosted.org/packages/d1/6a/4b2fcefd2ea0868810e92d519dacac1ddc64a2e53ba9e3422c3b62b378a6/setuptools-40.8.0-py2.py3-none-any.whl (575kB)
  Collecting wheel
    Downloading https://files.pythonhosted.org/packages/96/ba/a4702cbb6a3a485239fbe9525443446203f00771af9ac000fa3ef2788201/wheel-0.33.1-py2.py3-none-any.whl
  Collecting cffi>=1.4.1
    Downloading https://files.pythonhosted.org/packages/64/7c/27367b38e6cc3e1f49f193deb761fe75cda9f95da37b67b422e62281fcac/cffi-1.12.2.tar.gz (453kB)
  Collecting pycparser (from cffi>=1.4.1)
    Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
  Installing collected packages: setuptools, wheel, pycparser, cffi
    Running setup.py install for pycparser: started
      Running setup.py install for pycparser: finished with status 'done'
    Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-vhx9hxx_/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-hnobiin5/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-dx3r45lk --compile:
      unable to execute 'gcc': No such file or directory
      unable to execute 'gcc': No such file or directory

          No working compiler found, or bogus compiler options passed to
          the compiler from Python's standard "distutils" module.  See
          the error messages above.  Likely, the problem is not related
          to CFFI but generic to the setup.py of any Python package that
          tries to compile C code.  (Hints: on OS/X 10.8, for errors about
          -mno-fused-madd see http://stackoverflow.com/questions/22313407/
          Otherwise, see https://wiki.python.org/moin/CompLangPython or
          the IRC channel #python on irc.freenode.net.)

          Trying to continue anyway.  If you are trying to install CFFI from
          a build done in a different context, you can ignore this warning.

      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.6
      creating build/lib.linux-x86_64-3.6/cffi
      copying cffi/model.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/cparser.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/api.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/recompiler.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/__init__.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/lock.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/error.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/commontypes.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/verifier.py -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/_embedding.h -> build/lib.linux-x86_64-3.6/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.6/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-x86_64-3.6
      creating build/temp.linux-x86_64-3.6/c
      gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.6m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.6/c/_cffi_backend.o
      unable to execute 'gcc': No such file or directory
      error: command 'gcc' failed with exit status 1

      ----------------------------------------
  Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-vhx9hxx_/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-hnobiin5/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-dx3r45lk --compile" failed with error code 1 in /tmp/pip-install-vhx9hxx_/cffi/
  You are using pip version 18.1, however version 19.0.3 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.

  ----------------------------------------
Command "/usr/bin/python3.6 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dx3r45lk --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel "cffi>=1.4.1; python_implementation != 'PyPy'"" failed with error code 1 in None
You are using pip version 18.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
ERROR: Job failed: command terminated with exit code 1

Also using a docker alpine image in gitlab-ci. Same issue, rolling back to 1.23.2 fixed it for me.

[ijc]

@fergyfresh you haven't actually installed the packages @ulyssessouza said. Your logs:

  unable to execute 'gcc': No such file or directory
  unable to execute 'gcc': No such file or directory

From https://github.com/docker/compose/issues/6617#issuecomment-477974325:

$ apk add py-pip python-dev libffi-dev openssl-dev gcc libc-dev make

note the gcc in there.

[fergyfresh]

I know, I had to do a similar fix before on a different package, isn't libffi-dev and openssl-dev also important? So where is the documentation about the ssh support? I manually put ssh stuff in my private dockerfiles at my company and would love to read about the SSH changes. Change log for 1.24 doesnt even have ssh in it at all.

[fergyfresh]

@ijc I was just posting the debug logs since noone else did. Thanks for the repost of the suggested workaround though.

[cubikrichard]

@fergyfresh https://github.com/docker/compose/releases

First line:

Added support for connecting to the Docker Engine using the ssh protocol.

[fergyfresh]

I'll see myself out.

:man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming: :man_facepalming:

But how do we use it?

[AnthonyMastrean]

@fergyfresh You can pin to the previous version of Compose, like https://github.com/docker/compose/issues/6617#issuecomment-477724968

pip install --quiet docker-compose~=1.23.0

Or, you can install the new development dependencies, but you're gonna pay for that in build time. See what I've found so far in this comment https://github.com/docker/compose/issues/6617#issuecomment-478028445

[nielskrijger]

Had the same issue.

I stubbornly tried out @AnthonyMastrean suggestion:

image:
  name: docker/compose:X.X.X
  entrypoint: [""]

While Anthony got a docker daemon connection error for me it worked!

I tried it with versions 1.23.2 and 1.24.

Not exactly sure why, it's a long time ago I setup this CI file. I did notice I'm using docker run --network host when running the integration tests. Possibly that has something to do with it.

I'll share the interesting bits of my ci file:

image:
  name: docker/compose:1.24.0
  entrypoint: [""]

stages:
  - build
  - deploy

services:
  - docker:dind

variables:
  # lots of vars

# Run lint, test, and release in one job, this makes the build +/- 2,5 minutes
# faster because images do not have to be repulled between stages.
build:
  stage: build
  script:
    - docker pull $CONTAINER_IMAGE:builder || true
    - docker build --target builder --cache-from $CONTAINER_IMAGE:builder --tag $CONTAINER_IMAGE:builder .
    - docker push $CONTAINER_IMAGE:builder
    - docker run $CONTAINER_IMAGE:builder make test
    - docker run $CONTAINER_IMAGE:builder make lint
#    - apk add --no-cache py-pip .       THIS IS NO LONGER NECESSARY
#    - pip install docker-compose .       THIS IS NO LONGER NECESSARY
    - docker-compose up -d redis db
    - docker run --network host $CONTAINER_IMAGE:builder make integration
    - docker build --target release --cache-from $CONTAINER_IMAGE:builder --tag $CONTAINER_IMAGE:$CI_COMMIT_SHORT_SHA --tag $CONTAINER_IMAGE:latest .
    - docker push $CONTAINER_IMAGE:$CI_COMMIT_SHORT_SHA
    - docker push $CONTAINER_IMAGE:latest

The application itself is not run inside the docker-compose file. I wanted to run the linters and unit tests first and fail early. Only afterwards install docker compose using pip and run the integration tests. That is why I'm probably using docker run --network host to run the integration tests; the docker commands all relate to the image being built, while the docker-compose setup only provides the dependant services.

The only thing I don't like about this is the docker/compose image does not have latest or stable tags. Knowing myself I'll probably use 1.24 for the next couple of years until it breaks 😂

Seems I could refactor the compose file and and run everything using docker-compose given it's built-in the image already, but I fear I'll hit Anthony's issue and waste a couple of hours. Rather enjoy the 🌞 for now. Hope this helps someone!

(p.s. just realized I have to update this in over 20 repos... fml)

[doraeball22]

Description of the issue

Running pip install docker-compose==1.23.2 succeeds but pip install docker-compose fails.

Context information (for bug reports)

Error output is: /usr/bin/python2 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-Jeq6Db --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel "cffi>=1.1; python_implementation != 'PyPy'"" failed with error code 1 in None

Steps to reproduce the issue

Against docker:stable image run apk add --quiet --no-cache bash py-pip && pip install --quiet docker-compose==1.24.0

Observed result

/usr/bin/python2 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-Jeq6Db --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel "cffi>=1.1; python_implementation != 'PyPy'"" failed with error code 1 in None`

Expected result

Successful install.

Additional information

Running Docker-in-Docker for Gitlab test runner. Probably not relevant?

It'work
pip install docker-compose==1.23.2
and docker:stable

[sashkab]

My workaround is to use pre-built alpine packages for cffi and paramiko python modules before installing docker-compose 1.24:

/ # apk add --no-cache python3 py3-paramiko py3-cffi
/ # python3 -mpip install docker-compose
/ # python3 -mpip freeze | grep docker-compose
docker-compose==1.24.0

[uleodolter]

I have build my own docker-git-compose image using the following Dockerfile:

ARG VERSION=stable
FROM docker:${VERSION}

RUN apk update \
   && apk add --no-cache \
      git \
      git-daemon \
   && apk add --no-cache \
      py-pip \
      py-paramiko \
      py-cffi \
   && pip install docker-compose

Image build ist about 50 seconds.

[lracicot]

I was wondering why my whole build setup was failing for a few days. Installing 1.23.0 seems to be a nice temporary workaround, but it would be better to have a real solution. I'm not so willing to install gcc&co inside my build machines.

[fergyfresh]

I'm using docker in docker with gitlab-ci and using their docker-compose docker build as suggested here https://github.com/docker/compose/issues/6617#issuecomment-477974325 and you're going to need those deps moving forward > 1.24.0

[AnthonyMastrean]

Ah, you’re using someone else’s compose image. I’ll have to see how that one differs that it works when the official one doesn’t.

On Apr 3, 2019, at 5:05 PM, Billy Ferguson [e-mail] wrote:

docker:
stage: docker
image: tmaier/docker-compose:latest
services:
- docker:dind
before_script:
- docker info
- docker-compose --version
- echo $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
script:
- docker-compose build
- docker-compose push
- docker-compose up -d
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[cubikrichard]

job:
  image:
    name: docker/compose:1.24.0
    entrypoint:
      - ""
  stage: stage
  script:
    - docker-compose --version

[fergyfresh]

What's the log say?

On Thu, Apr 4, 2019, 4:08 AM cubikrichard [e-mail] wrote:

job:
image:
name: docker/compose:1.24.0
entrypoint:
- ""
stage: stage
script:
- docker-compose --version

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/6617#issuecomment-479795121,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJN8OIq4KI49fnTvjjhDWZqC_2usZi9uks5vdbL2gaJpZM4cQ2FT
.

[AnthonyMastrean]

I was getting socket errors (see the end of this comment) https://github.com/docker/compose/issues/6617#issuecomment-478028445

Looks like it might have been an intermittent failure or a problem on my run or my runner only? Others, including you, Billy, are using this image just fine. https://github.com/docker/compose/issues/6617#issuecomment-478231188

(Oh, I guess comment edits don’t get a new notification... that stinks, I use edit a lot)

On Apr 4, 2019, at 8:03 AM, Billy Ferguson [e-mail] wrote:

What's the log say?

On Thu, Apr 4, 2019, 4:08 AM cubikrichard [e-mail] wrote:

job:
image:
name: docker/compose:1.24.0
entrypoint:

• ""
  stage: stage
  script:
• docker-compose --version

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/6617#issuecomment-479795121,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJN8OIq4KI49fnTvjjhDWZqC_2usZi9uks5vdbL2gaJpZM4cQ2FT
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[fergyfresh]

@AnthonyMastrean I deleted my old comment. I pulled from the wrong repo lol.

clair:
  image: docker/compose:1.24.0
  stage: docker-security
  variables:
    DOCKER_DRIVER: overlay2
    CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
  allow_failure: true
  services:
    - docker:stable-dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker-compose pull

But, you can also use that other repo I linked you to, he pre-vets out all the docker/docker-compose versions for you so you can either pass arguments to the image, or just use the image.

[AnthonyMastrean]

I don't know if it's my script or my runner or what, but I can't get any other result. Always this socket error.

image: 
  name: docker/compose:1.23.2
  entrypoint: [""]

services:
  - docker:stable-dind

before_script:
  - docker login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

Running with gitlab-runner 11.0.0 (5396d320)
  on nextgen-aws-east1d-autoscale.amazonaws.com ec265edf
Using Docker executor with image docker/compose:1.23.2 ...
Starting service docker:stable-dind ...
Pulling docker image docker:stable-dind ...
Using docker image sha256:85e924caedbd3e5245ad95cc7471168e923391b22dcb559decebe4a378a06939 for docker:stable-dind ...
Waiting for services to be up and running...
Pulling docker image docker/compose:1.23.2 ...
Using docker image sha256:65f21bf3ffc929410490621982ff648d02f268fefb063f55dc3129eb70a9cdf3 for docker/compose:1.23.2 ...
Running on runner-ec265edf-project-10964250-concurrent-0 via runner-ec265edf-gitlab-docker-machine-1553872466-7e37c8d8...
Cloning repository...
Cloning into '/builds/vaeit/nextgen/itpie-api'...
Checking out 1efaf55f as infra-457-compose...
Skipping Git submodules setup
$ docker login --username "$CI_REGISTRY_USER" --password "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: Job failed: exit code 1

[ElectricWarr]

FWIW solving this by installing Python and my other dependencies on top of the official docker/compose:1.24.0 image results in an image almost half the size of one based on python:3.7.3-alpine3.9 (plus compilation dependencies).

[ulyssessouza]

For the ones getting problems using gitlab-ci with the docker image, looks like the solution is described in https://github.com/docker/compose/issues/6614. And this is implemented in https://github.com/docker/compose/pull/6641 which just got merged, meaning that this will be available with the next release.

[blindrood]

docker-compose fails on clean alpine as 1.24 introduces dependency on paramiko.
Pip install paramiko fails on clean alpine as it depends on gcc, make and some other packages.
This can be resolved by installing paramiko using apk
apk add --no-cache py-pip py-paramiko
pip install docker-compose

[satshabad-cr]

I was disappointed that this broke our build given that it was a minor version bump

[dduportal]

Went into the same. This works with a fresh Alpine 3.9.4 (tested on VM image and container image):

apk add --no-cache py3-pip py3-paramiko
pip3 install --no-cache-dir --upgrade pip
pip3 install --no-cache-dir "docker-compose==1.24.0"

[KCErb]

There is now a docker/compose image that will work with gitlab (https://github.com/docker/compose/issues/6614). So the solution is to drop the script for building docker compose with pip and just use a different base image in your .gitlab-ci.yml.

image: docker/compose:1.25.0-rc1

[mosal2000]

Thank you. This docker-compose image works for me.