Uvicorn: docker-compose graceful shutdown of uvicorn --reload

[sbv-trueenergy]

I'm trying to run uvicorn --reload in a docker-compose setup with a mounted directory of my code.
When using "ctrl+c" and other ways to end the process it ends up just haning for the graceful shutdown periode and then it gets the SIGKILL because nothings happens.
When looking at the strace it seems to be hanging indeed:

wait4(7, 0x7ffcf1409d84, WNOHANG, NULL) = 0
getpid()                                = 1
write(2, "INFO: Stopping reloader process "..., 36) = 36
getsockname(3, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
getpeername(3, 0x7ffcf1409f70, [16])    = -1 ENOTCONN (Transport endpoint is not connected)
close(3)                                = 0
wait4(7, 0x7ffcf140afe4, WNOHANG, NULL) = 0
wait4(7, 0x7ffcf140afe4, WNOHANG, NULL) = 0
getpid()                                = 1
wait4(7,  <unfinished ...>)             = ?
+++ killed by SIGKILL +++

If I disable the reloader it shutdowns down fine.
I've uploaded a minimal sample here: https://gist.github.com/sbv-trueenergy/a9a6971778a01d1acd3c466719e690b8

Am I doing something wrong?

[sbv-trueenergy]

I'm seeing the same with a minimal docker-compose.yml:

hello:
  image: tiangolo/uvicorn-gunicorn-fastapi
  command: /start-reload.sh

Run like:

docker-compose up

and then either ctrl+c or docker-compose stop from another terminal:

Starting bla_hello_1 ... done
Attaching to bla_hello_1
hello_1  | Checking for script in /app/prestart.sh
hello_1  | Running script /app/prestart.sh
hello_1  | Running inside /app/prestart.sh, you could add migrations to this file, e.g.:
hello_1  | 
hello_1  | #! /usr/bin/env bash
hello_1  | 
hello_1  | # Let the DB start
hello_1  | sleep 10;
hello_1  | # Run migrations
hello_1  | alembic upgrade head
hello_1  | 
hello_1  | INFO: Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
hello_1  | INFO: Started reloader process [1]
hello_1  | email-validator not installed, email fields will be treated as str.
hello_1  | To install, run: pip install email-validator
hello_1  | INFO: Started server process [7]
hello_1  | INFO: Waiting for application startup.
Stopping bla_hello_1 ...  (Waiting for complete graceful timeout timespan)
Stopping bla_hello_1 ... done
bla_hello_1 exited with code 137

[tomchristie]

Are you able to replicate outside of docker at all?

[sbv-trueenergy]

No, I don't seem to be able to.

[tomchristie]

Can we confirm that docker passes the Ctrl-C signal along? Can we replicate with a simpler signal catching example or not? Can we replicate with a simple Gunicorn + Flask (or whatever) setup or not?

[svrist]

I figure the ctrl c signal is what triggers the
write(2, "INFO: Stopping reloader process "...,?

And it works fine without the reloader

[euri10]

I found this read interesting https://hynek.me/articles/docker-signals/

[gnat]

Add init: true to the service you are running unicorn under to forward the signals.

[tomchristie]

Thanks @gnat.

[sbv-trueenergy]

Same issue, with or without init: true

[tomchristie]

Okay. This is out-of-scope for us to resolve. Would be very happy to see an update from any docker users on the right way to set this up, but it's a docker configuration issue, rather than a uvicorn issue.

[gvbgduh]

@sbv-trueenergy there was a similar discussion in https://github.com/encode/uvicorn/issues/430.

[gnat]

Can confirm @sbv-trueenergy's findings . Uvicorn signals are only working properly if both --workers or --reload are NOT used. Something to do with the child processes?

Signals work fine with Gunicorn in front of Uvicorn workers- this may be the interim solution if you're running in a container.

[gvbgduh]

It might highly depend on the docker image, have you tried the plain python (or even plain linux) image (with uvicorn only?)?

[gnat]

Issue is present on Ubuntu 19.10 with the default Python 3. Latest uvicorn, latest starlette.

My Dockerfile for reference:

FROM ubuntu:19.10

RUN apt update
RUN apt -y upgrade
RUN apt install -y git python3 python3-pip

RUN mkdir -p /srv/python
WORKDIR /srv/python

RUN pip3 install cython==0.29.13
RUN pip3 install uvicorn
RUN pip3 install starlette[full]
ENV PATH="/usr/local/bin/uvicorn:${PATH}"
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

CMD uvicorn app:app --host 0.0.0.0 --port 5000 --reload

[euri10]

Not sure it's the issue but at least the first thing would be to use the exec form for the CMD, the shell form doesn't propagate signals
see https://stackoverflow.com/questions/42805750/dockerfile-cmd-shell-versus-exec-form

[gnat]

@euri10 No difference in signal handling, rebuilt using both forms here:

CMD ["uvicorn", "app:app", "--host=0.0.0.0", "--port=5000", "--reload"]

ENTRYPOINT ["uvicorn", "app:app", "--host=0.0.0.0", "--port=5000", "--reload"]

[gvbgduh]

@gnat I tried the set up from https://github.com/encode/uvicorn/issues/430#issuecomment-558194168, and seems fine to me

$ docker-compose up
Recreating test_the_whole_web_1 ... done
Recreating test_the_whole_nginx_1 ... done
Attaching to test_the_whole_web_1, test_the_whole_nginx_1
web_1    | INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
web_1    | INFO:     Started reloader process [1]
web_1    | INFO:     Started server process [10]
web_1    | INFO:     Waiting for application startup.
web_1    | INFO:     Application startup complete.
^CGracefully stopping... (press Ctrl+C again to force)
Stopping test_the_whole_nginx_1   ... done
Stopping test_the_whole_web_1     ... done
$

[gnat]

Just to be sure, did it take 10 seconds? docker-compose will forcefully kill the uvicorn container after 10 seconds.

You can experience a true graceful shutdown of uvicorn by removing --reload and --workers

[gvbgduh]

Hm, interesting, yes, you're right, it takes 10 seconds.

[euri10]

my gut feeling is that ctrl+c kills the sh from the container and it's not
passed down to the child processes

On Wed, Dec 11, 2019 at 3:01 PM George Bogodukhov [e-mail]
wrote:

Hm, interesting, yes, you're right, it takes 10 seconds.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/encode/uvicorn/issues/364?email_source=notifications&email_token=AAINSPV42FISUFUACTYJ2OTQYDXDRA5CNFSM4HOH2SJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGTG2IY#issuecomment-564555043,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAINSPWMQMERW4AM66UFGMTQYDXDRANCNFSM4HOH2SJQ
.

--
benoit barthelet
http://pgp.mit.edu/pks/lookup?op=get&search=0xF150E01A72F6D2EE

[euri10]

from this real ugly test it seems to me that without reload, the container runs uvicron on PID 1 and that killing it works fine
If --reload is used, docker runs the relaoder as PID 1 and the server as PID 9, and sending a SIGINT has no effect whatsoever, you dont reach INFO: Shutting down in that case

no reload log

-------------------------------- live log call ---------------------------------
INFO     test_statreload:test_statreload.py:109 number of process in container top: 1
INFO     test_statreload:test_statreload.py:110 [['root', '200841', '200823', '15', '16:35', '?', '00:00:00', '/usr/local/bin/python /usr/local/bin/uvicorn app:app --host=0.0.0.0 --port=5000']]
INFO     test_statreload:test_statreload.py:113 INFO:     Started server process [1]
INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO:     Shutting down
INFO:     Finished server process [1]

PASSED

with reload log

-------------------------------- live log call ---------------------------------
INFO     test_statreload:test_statreload.py:109 number of process in container top: 3
INFO     test_statreload:test_statreload.py:110 [['root', '201270', '201253', '15', '16:35', '?', '00:00:00', '/usr/local/bin/python /usr/local/bin/uvicorn app:app --host=0.0.0.0 --port=5000 --reload'], ['root', '201416', '201270', '2', '16:35', '?', '00:00:00', '/usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(4)'], ['root', '201417', '201270', '8', '16:35', '?', '00:00:00', '/usr/local/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=5, pipe_handle=7) --multiprocessing-fork']]
INFO     test_statreload:test_statreload.py:113 INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
INFO:     Started reloader process [1]
INFO:     Started server process [9]
INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.

PASSED

ugly test here

[euri10]

I think I got a fix on this branch, fancy trying it @gnat ?
https://github.com/euri10/uvicorn/tree/docker_signal

this was my inspiration after I read both excellent posts here and here

[gnat]

Works well, and the implementation looks sound. Top effort @euri10.

Would love to see this merged, docker test included or not, as the change is straightforward.

[tomchristie]

Nice one!