Compose: docker-compose.yml doesn't seem to reference image / container in Dockerfile

[dyerrington]

I've been at this for weeks, and I'm assuming I'm overlooking something completely simple. I've created a Dockerfile that creates a Python:2.7 image, copies a few files and directories over, and also an nginx and MySQL image as well. Originally, I constructed a Dockerfile and a docker-compose file, ported my apps to use this setup, then deployed onto an AWS EC2 instance, no problems. At some point, my Django startup script was showing up as command not found. So, I tried to just rebuild everything, and while everything built successfully, I just had the same error. So Naturally, I double checked my configuration in production with my local and found everything to be exactly 1:1 with regards to my Dockerfile and docker-compose config.

The relevant part of my Dockerfile looks like this:

FROM python:2.7
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD environment.yml /code/
ADD requirements.txt /code/
ADD start_maxlabs.sh /code/
RUN chmod +x /code/start_maxlabs.sh
RUN apt-get update
RUN apt-get install -y libmysqlclient-dev gunicorn 
RUN apt-get install -y libcairo2-dev 
RUN apt-get install -y libjpeg62-turbo-dev 
RUN apt-get install -y libpango1.0-dev
RUN apt-get install -y libgif-dev 
RUN apt-get install -y build-essential 
RUN apt-get install -y g++ 
RUN apt-get install -y python-imaging
RUN pip install -U pip
RUN pip install mysql-python
RUN pip install -r requirements.txt
ADD . /code/
ADD maxlabs /code/
ADD webapp /code/

My composer file looks like this:

  web:
    image: python:2.7
    container_name: maxlabs
    restart: always
    build: .
    depends_on:
      - db
    command: /bin/bash /code/start_maxlabs.sh

After everything builds, and comes online, I see this message:

maxlabs exited with code 127
maxlabs | /bin/bash: /code/start_maxlabs.sh: No such file or directory
maxlabs | /bin/bash: /code/start_maxlabs.sh: No such file or directory
maxlabs | /bin/bash: /code/start_maxlabs.sh: No such file or directory
maxlabs | /bin/bash: /code/start_maxlabs.sh: No such file or directory

Indeed when I login to the container, there is no /code directory. Nothing has changed. My versioned copy in my repo of Dockerfile and docker-compose.yml hasn't changed.

If there's a better place to ask this, please let me know.

Thanks in advance!

[dyerrington]

Also see this:

webapp dyerrington$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                            PORTS               NAMES
b0f2a3145b18        python:2.7          "/bin/bash"              27 seconds ago      Exited (1) 15 seconds ago                             eloquent_pare
03e4bd710408        python:2.7          "/bin/bash /code/s..."   2 minutes ago       Exited (127) About a minute ago                       maxlabs

[dyerrington]

After I deleted all containers / images, and run docker-compose up --build, I can find that when I log into "5c2f0cdc8450" (the 2nd image) listed, I can see that /code is listed with all my files (yay!). I added this to volumes in the docker-compose.yml and it seems to find code just fine (command: /code/start_maxlabs.sh in the docker-compose.yml) and can run my startup script. Now it can't find gunicorn. I just installed packages from the DockerFile though? When I run the startup script I wrote directly on that 2nd image, it runs just fine.

web_1  | /code/start_maxlabs.sh: line 6: gunicorn: command not found
web_1  | /code/start_maxlabs.sh: line 14: gunicorn: command not found

This is an improvement but I'm not sure I understand why, after removing all my images and containers, then build again, I see the following:

webapp dyerrington$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
python              2.7                 7a043256604f        28 minutes ago      182 MB
<none>              <none>              5c2f0cdc8450        28 minutes ago      2.11 GB
<none>              <none>              b89f2d3a8932        About an hour ago   2.11 GB
<none>              <none>              f5a64d9146e8        3 hours ago         2.11 GB
<none>              <none>              4b5355cf166d        4 hours ago         2.11 GB
<none>              <none>              cb9529a52079        4 hours ago         1.72 GB
python              <none>              ca388cdb5ac1        12 days ago         676 MB
nginx               latest              6b914bbcb89e        12 days ago         182 MB
mysql               latest              22be5748ecbe        12 days ago         406 MB

Only one of these images has my /code directory and the packages I installed. Why does it create so many of these?

[shin-]

Remove the image entry from your compose file. build and image can be used together, but they don't have the meaning you seem to attribute it: https://docs.docker.com/compose/compose-file/compose-file-v2/#build

As it stands, your web service starts off a fresh python:2.7 image without any of your dependencies or code, which is why nothing appears to be working.

HTH!

[dyerrington]

Thanks @shin- I was able to consolidate a bunch of rogue images and containers this way. However, I'm still getting this error:

gunicorn: command not found

Here is the updated composer file I created:

  web:
    build:
      context: .
      dockerfile: Dockerfile
    restart: always
    depends_on:
      - db
    command: /bin/bash /code/start_maxlabs.sh
    ports:
      # - "85:80"
      - "8001:8000"
      - "8002:8005"
    expose:
      - "8000"
    links:
      - db:mysql
    volumes:
      - .:/code
      - /usr/src/app
      - /usr/src/app/static
      - /usr/share/nginx/html

However, my images and container space are nice and clean now!

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
web_web             latest              26517bdbc24b        19 hours ago        182 MB
nginx               latest              6b914bbcb89e        2 weeks ago         182 MB
mysql               latest              22be5748ecbe        2 weeks ago         406 MB

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                           PORTS               NAMES
49099ad97b75        web_web             "/bin/bash /code/s..."   19 hours ago        Exited (127) About an hour ago                       web_web_1
92e9c8c69f87        mysql               "docker-entrypoint..."   19 hours ago        Up 19 hours                      3306/tcp            web_db_1

[shin-]

When is gunicorn being installed? I don't see it explicitly mentioned in the Dockerfile you shared above. Is it part of your requirements.txt?

[dyerrington]

Ah sure! I added it to both my requirements and my apt list:

FROM python:2.7
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD environment.yml /code/
ADD requirements.txt /code/
ADD start_maxlabs.sh /code/
RUN chmod +x /code/start_maxlabs.sh
RUN apt-get update && \
    apt-get install -y python-pip python-dev build-essential \
                       libmysqlclient-dev gunicorn \
                       libcairo2-dev \
                       libjpeg62-turbo-dev \
                       libpango1.0-dev \
                       libgif-dev \
                       build-essential \
                       g++ \
                       python-imaging
RUN pip install -U pip \
                   mysql-python \
                   gunicorn
RUN pip install -r requirements.txt
ADD . /code/
ADD maxlabs /code/
ADD webapp /code/

[shin-]

Does it work if you call docker-compose build before docker-compose up? You might be using an image from an earlier / outdated build if you haven't rebuilt after modifying your Dockerfile.

[dyerrington]

This is how I build exactly. First docker-compose build then docker-compose up. I also tried wiping all of my containers and images first.

[shin-]

Does it work if you execute docker-compose run web gunicorn? You should see something like this:

$ docker-compose run web gunicorn
usage: gunicorn [OPTIONS] [APP_MODULE]
gunicorn: error: No application module specified.

If so, are you able to share the contents of the start_maxlabs.sh? gunicorn is undoubtedly installed in your container, but there might be an issue with the context or user you're calling it from not having access to it.

[dyerrington]

Sure!

Davids-MacBook-Pro-4:web davidyerrington$ docker-compose run web gunicorn
ERROR: Cannot start service web: oci runtime error: container_linux.go:247: starting container process caused "exec: \"gunicorn\": executable file not found in $PATH"

The contents of my startup script is as follows:

echo "Starting maxlabs..."
cd /code/maxlabs

/usr/local/bin/gunicorn maxlabs.wsgi:application \
        --name maxlabs \
        --bind 0.0.0.0:8000 \
        --workers 3 \
        --log-level=info \
        --log-file=/dev/stdout &
echo "Starting rapstats..." &&
cd /code/webapp &&
/usr/local/bin/gunicorn webapp.wsgi:application \
        --name rapstats \
        --bind 0.0.0.0:8005 \
        --workers 3 \
        --log-level=info \
        --log-file=/dev/stdout

Also I should mention that my startup script was inaccessible (ie: Command not found), until I added .:/code to my volumes, if that's helpful.

[shin-]

Davids-MacBook-Pro-4:web davidyerrington$ docker-compose run web gunicorn
ERROR: Cannot start service web: oci runtime error: container_linux.go:247: starting container process caused "exec: \"gunicorn\": executable file not found in $PATH" 

That error message doesn't seem right. Can you run docker-compose build --no-cache and share the output?

[dyerrington]

Sure here's the output:

http://textuploader.com/dtmkd

[shin-]

Thanks! Do you still get the same error when running docker-compose run web gunicorn ? This is very different from what I'm seeing when trying to reproduce locally.

[dyerrington]

Thanks shin-! The error still persists unfortunately:

$ docker-compose run web gunicorn
Starting web_db_1
ERROR: Cannot start service web: oci runtime error: container_linux.go:247: starting container process caused "exec: \"gunicorn\": executable file not found in $PATH"

However given these images (after running build from a clean slate):

Davids-MacBook-Pro-4:web davidyerrington$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
web_web             latest              a834dbed32c0        14 minutes ago      182 MB
<none>              <none>              34179ad76776        14 minutes ago      2.3 GB
<none>              <none>              636c18d164e2        18 hours ago        182 MB
<none>              <none>              a26bcc79d927        18 hours ago        2.3 GB
<none>              <none>              b40765f66449        36 hours ago        182 MB
<none>              <none>              e5b047e9e383        37 hours ago        2.3 GB
python              2.7                 ca388cdb5ac1        2 weeks ago         676 MB
nginx               latest              6b914bbcb89e        2 weeks ago         182 MB
mysql               latest              22be5748ecbe        2 weeks ago         406 MB

I notice gunicorn not on the latest created, but it's on the containers that have a size of 2.3G:

$ docker run -ti  a834dbed32c0  gunicorn
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"gunicorn\": executable file not found in $PATH".

$ docker run -ti  34179ad76776   gunicorn
usage: gunicorn [OPTIONS] [APP_MODULE]
gunicorn: error: No application module specified.

[shin-]

Wow, okay - that's a weird one. Can we recap where you are at with your docker-compose.yml contents and Dockerfile contents at this point (I know you posted them earlier in the thread but I'm assuming they've been modified somewhat since then)? I'm pretty sure it's just a small thing we're missing at this point that's throwing a wrench in the system.

[dyerrington]

Sure, here's my DockerFile:

FROM python:2.7
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD environment.yml /code/
ADD requirements.txt /code/
ADD start_maxlabs.sh /code/
RUN chmod +x /code/start_maxlabs.sh
RUN apt-get update && \
    apt-get install -y python-pip python-dev build-essential \
                       libmysqlclient-dev \
                       gunicorn \
                       libcairo2-dev \
                       libjpeg62-turbo-dev \
                       libpango1.0-dev \
                       libgif-dev \
                       build-essential \
                       g++ \
                       python-imaging
RUN pip install -U pip \
                   mysql-python \
                   gunicorn
RUN pip install -r requirements.txt
ADD . /code/
ADD maxlabs /code/
ADD webapp /code/

And my docker-compose.yml:

version: '2'
services:
  db:
    image: mysql
    environment:
      MYSQL_ROOT_PASSWORD: dfhjdsfhjkhjkdfhkj
      MYSQL_USER: mysql
      MYSQL_PASSWORD: mysql
      MYSQL_DATABASE: maxlabs

  web:
    build:
      context: .
      dockerfile: Dockerfile
    restart: always
    depends_on:
      - db
    command: /bin/bash /code/start_maxlabs.sh
    ports:
      # - "85:80"
      - "8001:8000"
      - "8002:8005"
    expose:
      - "8000"
    links:
      - db:mysql
    volumes:
      - .:/code
      - /usr/src/app
      - /usr/src/app/static
      - /usr/share/nginx/html

[shin-]

Hm, everything builds fine for me here. Your directory structure looks something like this, right?

$ ls -F
docker-compose.yml  environment.yml  requirements.txt  webapp/
Dockerfile          maxlabs/         start_maxlabs.sh

And you run docker-compose from inside that directory? What's the output of docker-compose version? Should be something like

$ docker-compose version
docker-compose version 1.11.2, build --------
docker-py version: 2.1.0
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016

Finally, what happens if you run docker build -t webapp:temp . ; docker run webapp:temp gunicorn?

[dyerrington]

Pretty close actually:

$ ls -F
Dockerfile          maxlabs/            rapstats_nginx.conf     sync_source.sh*
config/             maxlabs_nginx.conf      requirements.txt        webapp/
docker-compose.yml      nginx/              reset_docker.sh*        webapp.tar
environment.yml         no_cache_build_output.txt   start_maxlabs.sh

My version:

$ docker-compose version
docker-compose version 1.11.2, build dfed245
docker-py version: 2.1.0
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2j  26 Sep 2016

Then the output of my docker build -t webapp:temp . ; docker run webapp:temp gunicorn is:

Removing intermediate container e23f8c927e10
Successfully built f1f9947510b4
container_linux.go:247: starting container process caused "exec: \"gunicorn\": executable file not found in $PATH"
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"gunicorn\": executable file not found in $PATH".

[shin-]

Sorry about the delay!

It looks like your issue is independent of Compose, since building and running from your Dockerfile with docker directly seems to cause the same issue with the container apparently not able to find the gunicorn binary in its PATH.

It may be worth x-posting your issue on the docker/docker issue tracker: https://github.com/docker/docker/issues - I'm out of ideas at that point :thinking:

[dyerrington]

@shin- Thank you so much for taking the time to investigate my query! Really appreciate it!

[dusantrtica]

@dyerrington I've recently came across the issue where - no matter how simple my python app would be (assume that I have hello.py which outputs 'hello'), running through docker-compose build & up I had had problem like: file not found: hello.py even though it was there. The problem was virtualbox and this resolved my problems immediately:
https://stackoverflow.com/questions/35005974/docker-ubuntu-virtualbox-volumes-directive-in-dockerfile-not-working/35013305#35013305

[jmgamboa]

@dusantrtica @dyerrington also having the same issue. Running locally on my mac is fine but on ubuntu 16.0.4.2 is not working with 'unable to find g unicorn' error. docker-compose version is 2, docker version is 1.13.1.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.