Gunicorn: ModuleNotFoundError running inside a Docker container

[ye]

I have an application Docker containerized. Dockerfile looks like this:

FROM python:3.7-alpine

...

COPY .  /app

RUN echo 'manylinux1_compatible = True' > /usr/local/lib/python3.7/site-packages/_manylinux.py \
    && pip install --upgrade pip \
    && pip install -r /app/requirements.txt

WORKDIR /app/src

CMD ["/usr/local/bin/gunicorn", "-w", "4", "-b", "0.0.0.0:5000", "-t", "30", "--pythonpath", "/app/src", "'myapp:create_app()'"]

docker inspect 61db228f0e42 | jq '.[0]["Config"]' looks like this:

{
  "Hostname": "61db228f0e42",
  "Domainname": "",
  "User": "",
  "AttachStdin": false,
  "AttachStdout": false,
  "AttachStderr": false,
  "Tty": false,
  "OpenStdin": false,
  "StdinOnce": false,
  "Env": [
    "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "LANG=C.UTF-8",
    "GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D",
    "PYTHON_VERSION=3.7.4",
    "PYTHON_PIP_VERSION=19.3",
    "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/65986a26949050d26e6ec98915da4aade8d8679d/get-pip.py",
    "PYTHON_GET_PIP_SHA256=8d412752ae26b46a39a201ec618ef9ef7656c5b2d8529cdcbe60cd70dc94f40c"
  ],
  "Cmd": [
    "/usr/local/bin/gunicorn",
    "-w",
    "4",
    "-b",
    "0.0.0.0:5000",
    "-t",
    "30",
    "--pythonpath",
    "/app/src",
    "'myapp:create_app()'"
  ],
  "Image": "54410e804752",
  "Volumes": null,
  "WorkingDir": "/app/src",
  "Entrypoint": null,
  "OnBuild": null,
  "Labels": {}
}

Docker run in daemon crashes immediately. However, passing in the run command would make it work.

$ docker run -d 54410e804752
61db228f0e4223898f706e018baf284d99f0a4767de3ddac82abce0607188276

$ docker ps -a
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS                      PORTS                    NAMES
61db228f0e42        54410e804752         "/usr/local/bin/guni…"   3 seconds ago      Exited (3) 2 seconds ago                            jovial_stonebraker

$ docker logs jovial_stonebraker
[2019-11-12 21:24:57 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-11-12 21:24:57 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2019-11-12 21:24:57 +0000] [1] [INFO] Using worker: sync
[2019-11-12 21:24:57 +0000] [8] [INFO] Booting worker with pid: 8
[2019-11-12 21:24:57 +0000] [8] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
ModuleNotFoundError: No module named "'myapp"

$ docker run -d -p 5000:5000  54410e804752 /usr/local/bin/gunicorn -w 4 -b 0.0.0.0:5000 -t 30 --pythonpath /app/src 'myapp:create_app()'
1c95a105b02c02105a6a5a6479c983223dc6de8a44b5e9e2d5d4de53dd4c0f95

$ docker ps -a
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS                      PORTS                    NAMES
1c95a105b02c        54410e804752         "/usr/local/bin/guni…"   3 seconds ago       Up 2 seconds                0.0.0.0:5000->5000/tcp   sleepy_hawking

Any know why is the --pythonpath not working inside of the Docker container?

[ye]

So it appears to be that gunicorn doesn't work with the exec form of Dockerfile CMD, only the shell form of Dockerfile CMD would work with custom PYTHONPATH, in other words, changing the last line in Dockerfile to below works.

...
CMD /usr/local/bin/gunicorn -w 4 -b 0.0.0.0:5000 -t 30 --pythonpath /app/src 'myapp:create_app()'

I would really love to make the exec form of CMD work. What would it take to establish a custom PYTHONPATH?

[tilgovi]

What happens if you define PYTHONPATH using a Dockerfile ENV directive?
https://docs.docker.com/engine/reference/builder/#env

[tilgovi]

I also noticed your invocation of myapp:create_app(). Be aware there's an open issue about this form: https://github.com/benoitc/gunicorn/issues/2159

[ye]

@tilgovi I've tried ENV PYTHONPATH /app/src, it didn't work :( Somehow I feel that something is missing from the shell init in the exec form, I just don't know what it is.

thanks for the heads up for that myapp:create_app() issue - I don't think that's relevant to me though, simply because /usr/local/bin/gunicorn -w 4 -b 0.0.0.0:5000 -t 30 --pythonpath /app/src 'myapp:create_app()' works in shell form CMD in my setup, so gunicorn has no problem importing myapp as a module. Also, 2159 is a version 20.0.0 issue, I am on version 19.9.0 as you can see in the logs.

[benoitc]

Looking at the trace it seems to try to load 'myapp instead of myapp (so adding ') . Did you try to escape the app uri?

[benoitc]

@ye bump

[ye]

@benoitc thanks! sorry I missed your notes. How do I escape the single quote'?

[benoitc]

@ye i would try without

[ye]

@benoitc YES! That was IT! By removing the double quoting, it worked! Thank you!

So the last line in the Dockerfile would be:

...

CMD ["/usr/local/bin/gunicorn", "-w", "4", "-b", "0.0.0.0:5000", "-t", "30", "--pythonpath", "/app/src", "myapp:create_app()"]