Compose: running inside alpine linux

[tjamet]

Hi,

I'm about to run docker-compose inside a container built with Dockerfile:

FROM alpine

RUN apk update && apk add curl
RUN curl -o /usr/local/bin/docker-compose -L https://github.com/docker/compose/releases/download/1.6.2/docker-compose-`uname -s`-`uname -m` && chmod +x /usr/local/bin/docker-compose
ENTRYPOINT ["/usr/local/bin/docker-compose"]

but then, running:
docker run -v /var/run/docker.sock:/var/run/docker.sock compose --help
leads to a no such file or directoryerror.
I read that this is due to docker-compose being binary compiled from python sources, using the glibc while alpine runs musl.
Would it be possible to get a docker-compose binary running on alpine?

[aanand]

I don't know if the binary can be made to run on Alpine, but you can run Compose on Alpine as a Python package - look at the Dockerfile we use for building the official Compose image to see a working example.

[sanmai-NL]

The executable will run independent of C the standard library implementations installed on the user's system if you statically link it to e.g. musl libc during build. I've found that 1.7.1 executables have been linked dynamically instead.

[mahnunchik]

alpine:3.4

curl -L https://github.com/docker/compose/releases/download/1.8.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose \
  && chmod +x /usr/local/bin/docker-compose

/# docker-compose 
sh: docker-compose: not found

bash-4.3# docker-compose 
bash: /usr/local/bin/docker-compose: No such file or directory

[sanmai-NL]

@mahnunchik: Can you please explain what information your comment adds? You repeat the OP and the issue hasn't been solved yet, as can be concluded from the @aanand's response.

[tjamet]

To have an idea of the feasibility, I've been building compose for alpine, here is what I ended with if anyone wish to do so:
https://github.com/tjamet/docker-compose/blob/master/Dockerfile

[mahnunchik]

@sanmai-NL

I've wanted to show that the problem is still actual in the version 1.8.0

[sanmai-NL]

@tjamet:
I'm wondering why you did it in that relatively complicated way, instead of:

apk add 'py-pip==8.1.2-r0'
pip install 'docker-compose==1.8.0'

[tjamet]

@sanmai-NL I agree it looks a bit complicated although the idea is to provide the smallest possible image as well as understanding how compose is packaged for linux:

docker images | grep compose
tjamet/compose                                    latest                                     08dd508e9e67        4 minutes ago       10.72 MB
compose:alpine                                     latest                                     a20ffaf715b8        7 minutes ago       143.1 MB

[ashishapy]

I also got the same problem. Trying to build docker-compose on alpine:3.4 using

curl -L https://github.com/docker/compose/releases/download/1.8.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Trying not to defeat purpose of alpine (having smallest possible images).

/# docker-compose 
sh: docker-compose: not found

any help?

[ashishapy]

3856

The dockerfile generates a more lightweight image. Thanks to @marcosnils

[sanmai-NL]

See: #3741.

[ebuildy]

@ashishapy just:

RUN apk --update add 'py-pip==8.1.2-r0' && \
pip install 'docker-compose==1.8.0'

This install docker-compose and works.

[marcosnils]

@ashishapy yes but it makes the docker image supper big. Because it installs python, pip and all the compose dependencies

[JonathonReinhart]

As I just diagnosed for someone on Stack Overflow before realizing this issue existed.

The problem is that PyInstaller really isn't very good at building applications for systems other than the one it was built on. Specifically, PyInstalled applications depend on the libc version of the system on which they were built.

In this case, the release binary of docker-compose has its interpreter set to /lib64/ld-linux-x86-64.so.2, because it was linked against GLIBC and that's the path to the dynamic loader on the build system. However, Alpine Linux is built using musl-libc and doesn't use ld-linux-x86-64.so.2. In fact, it doesn't even have a /lib64 directory.

I even tried going to the extreme, installing the libc6-compat package and symlinking /lib64 to /lib, only to find out that there are GLIBC-specific symbols in use (namely __vsnprintf_chk and __sprintf_chk).

It's not going to work. Installing via pip is the only way right now.

It's a shame that PyInstaller isn't capable of producing more system-independent packages. It's possible that they could make the outer bootstrap code statically-linked, but then you would run into the problem of all of the C Python libraries in use. It's hopeless.

The only straightforward way to make well-redistributable binaries would be to re-write docker-compose in something like Go.

[marcosnils]

@JonathonReinhart, I proposed a fix for this here #3856

[tjamet]

@JonathonReinhart There are in fact ways to generate the same static binary with PyInstaller @marcosnils has done one, there is another one here: https://github.com/tjamet/docker-compose/blob/master/Dockerfile

I'm not sure re-writing things in go would help in handling multiple c libraries. There are still several libraries in go that requires "C", making it as weak as PyInstaller on running on a different kernel: https://github.com/golang/sys/blob/master/unix/types_linux.go#L142
I have made a simple example to illustrate it: https://github.com/tjamet/go-libc

[pwFoo]

Alpine docker-compose is working fine with glibc package.

apk --allow-untrusted --no-cache -X http://apkproxy.heroku.com/andyshinn/alpine-pkg-glibc add glibc glibc-bin

[diorman]

Alpine uses musl libc instead of glibc. This base image solves the problem frolvlad/alpine-glibc. Here's an example:

1. Dockerfile

FROM frolvlad/alpine-glibc

RUN apk --no-cache add curl

ENV DOCKER_COMPOSE_VERSION 1.12.0

RUN curl -L https://github.com/docker/compose/releases/download/$DOCKER_COMPOSE_VERSION/docker-compose-Linux-x86_64 > /usr/local/bin/docker-compose \
  && chmod +x /usr/local/bin/docker-compose

ENTRYPOINT ["docker-compose"]

1. Build

$ docker build -t docker-compose-alpine .

1. Run

$ docker run --rm docker-compose-alpine --version 

Cheers

[marcosnils]

@diorman my PR in the official compose repo has been merged which fixes this. https://github.com/docker/compose/pull/3856

@dnephin @shin- @tjamet can we close this?

[diorman]

@marcosnils sorry for missing your comment. Good PR!. At least the example might help anyone that wants to use a binary previously built ¯_(ツ)_/¯

[JonathonReinhart]

@tjamet I wrote a tool, not unlink PyInstaller, to bundle dynamically-linked executables with all of their dependencies:

https://github.com/JonathonReinhart/staticx

Coupled with PyInstaller, this allows one to create Python applications which run completely independent of the target system.

It's still in a beta stage, but I'd love for the docker-compose team to try it out and let me know what you think.

[llitfkitfk]

apk add --no-cache python3
pip3 install docker-compose

[mohamnag]

I was shoked that compose is not present in official docker image https://hub.docker.com/r/library/docker and @llitfkitfk's was the only way that I could install it inside that image.

[santhoshkumarhirekerur]

```ruby
RUN apk add --no-cache python3 && \
python3 -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip3 install --upgrade pip setuptools && \
if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi && \
if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi && \
rm -r /root/.cache

RUN pip install 'docker-compose==1.20.1'

[gilescope]

docker-stack is statically linked go. It should be able to run docker-compose files. Might be an easier (and faster) option.

[kondaurovDev]

Piece of my dockerfile:

FROM alpine:3.8

RUN curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

RUN chmod +x /usr/local/bin/docker-compose

Inside container:

/ # docker-compose
/bin/sh: docker-compose: not found

Looks like it doesn't work even PR was merged

I replaced

FROM alpine:3.8

with this

FROM frolvlad/alpine-glibc:alpine-3.8_glibc-2.28 

now it works

[damlys]

I'm currently use

RUN COMPOSE_VERSION="1.23.2" \
&& apk add --no-cache \
  py-pip \
&& pip install --no-cache-dir \
  docker-compose==${COMPOSE_VERSION}

instead of

RUN COMPOSE_VERSION="1.23.2" \
&& curl -fsSL https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose \
&& chmod a+rx /usr/local/bin/docker-compose

and it works well for me. Image is a little more heavy than it should be, but it is ok for me for now.

[shin-]

Here's the official solution:

FROM docker/compose:1.23.2
# install whatever else you need

[zazapeta]

@shin- Thanks you thanks you thanks you !!!!

[AliMD]

The best solution I founded.

apk add python-dev libffi-dev openssl-dev gcc libc-dev make
pip install docker-compose

[sjwl]

Here's the official solution:

FROM docker/compose:1.23.2
# install whatever else you need

add

ENTRYPOINT [ ]

else your image inherits docker/compose ENTRYPOINT ["sh", "/usr/local/bin/docker-compose-entrypoint.sh"]

[programmingworld1]

I am writing an gitlab-ci.yml file and have a docker compose based project. how should my gitlab-ci.yml file look? Anyone? I have been struggling a lot with this one..

[kern-panic]

FROM docker:19.03.2-dind

ENV EDGE_MAIN http://dl-cdn.alpinelinux.org/alpine/edge/main
ENV EDGE_COMMUNITY http://dl-cdn.alpinelinux.org/alpine/edge/community

RUN apk update --repository=$EDGE_MAIN --repository=$EDGE_COMMUNITY \
    && apk --no-cache add curl make python3 python3-dev gcc libc-dev libffi-dev \
        openssl-dev --repository=$EDGE_MAIN --repository=$EDGE_COMMUNITY \
    && pip3 --no-cache-dir install --upgrade pip \
    && pip3 --no-cache-dir install docker-compose \
    && rm -f /var/cache/apk/* \
    && rm -rf /root/.cache

$ docker exec dind docker-compose --version
docker-compose version 1.24.1, build 4667896

[kern-panic]

I am writing an gitlab-ci.yml file and have a docker compose based project. how should my gitlab-ci.yml file look? Anyone? I have been struggling a lot with this one..

Hi
I created my image following the example above, including docker-compose.
And use it in Gitlab CI.

.gitlab-ci.yml

image: my_image_from_docker_dind

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""

services:
  - docker:19.03.2-dind

before_script:
  - docker info

my_job:
  tags:
    - my_tags
  script:
    my_command

[BeyondEvil]

RUN apk update --repository=$EDGE_MAIN --repository=$EDGE_COMMUNITY \
&& apk --no-cache add curl make python3 python3-dev gcc libc-dev libffi-dev \
openssl-dev --repository=$EDGE_MAIN --repository=$EDGE_COMMUNITY \
&& pip3 --no-cache-dir install --upgrade pip \
&& pip3 --no-cache-dir install docker-compose \
&& rm -f /var/cache/apk/* \
&& rm -rf /root/.cache

Would you mind explaining what's going on here? Especially what the significance of the EDGE repo has?

Thanks in advance!

@kern-panic

[llitfkitfk]

@BeyondEvil
you can search it https://pkgs.alpinelinux.org/packages?name=openssl-dev&branch=edge
for installed edge version software

[matti]

RUN apk add --no-cache python2 \
  && apk add --no-cache --virtual .docker-compose-deps \
  py-pip python-dev libffi-dev openssl-dev gcc libc-dev make \
  && pip install docker-compose==1.25.0 \
  && apk del .docker-compose-deps

[Tazovsky]

Following worked with docker image python:3.6-alpine3.11:

cd /tmp/ && \
        wget --no-check-certificate -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
        wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.31-r0/glibc-2.31-r0.apk && \
        wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.31-r0/glibc-bin-2.31-r0.apk && \
        wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.31-r0/glibc-i18n-2.31-r0.apk && \
        apk add glibc-2.31-r0.apk glibc-bin-2.31-r0.apk glibc-i18n-2.31-r0.apk && \
        /usr/glibc-compat/bin/localedef -i en_US -f UTF-8 en_US.UTF-8 && \
        rm -rf /tmp/* && \
        cd - && \
    sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose && \
    sudo chmod +x /usr/local/bin/docker-compose && \
    sudo ln -sf /usr/local/bin/docker-compose /usr/bin/docker-compose