Uwsgi: uWGSI graceful shutdown

[dmk23]

I am looking at the full list of uWGSI signals to figure out the proper way to shut down:
http://uwsgi-docs.readthedocs.org/en/latest/Management.html

It is quite surprising to see no way to exit the service gracefully vs. just brutally. Yes, it is good to be able to gracefully reload, but some deployments may rely on outside scripts invoking a "service restart" command, which would stop and then start the service - in this case destroying the requests in process.

Is there a workaround to enforce graceful shutdown? A possible new feature? Why have two signals (SIGINT and SIGQUIT) do exact same thing when one could designate a graceful option?

[unbit]

check the master fifo. it is the way to go in modern environments

[dmk23]

OK, found it already.

One suggestion though - why not add config options for overriding _any_ signal? It just seems quite cumbersome to have to set up these pipes for something as simple as graceful shutdown...

[unbit]

Well, the signal+pidfile paradigm is so unreliable that people spit out new solutions every time by ages, so solving it with a fifo (that is automatically created by uWSGI) seemed a good compromise :)

By the way, just make a proposal, if it is easy (and not invasive) to implement we can absolutely add it

[dmk23]

Definitely did not mean to criticize - been using uWSGI for just a few days as a replacement of a very old and unreliable container and very impressed with what you've done here :)

Do you want a proposal in any specific format? It is really simple, a config like this:

signal-command = SIGQUIT r

This would override the "default" behavior of SIGQUIT to be identical to 'r' sent to Master FIFO, effecting a graceful shutdown. The benefit is not having to set up Master FIFO and track / manage / access these files if we only care about a couple simple commands.

It should be possible to have multiple lines like this to override multiple signals.

Does this sound clear/easy enough to implement?

[unbit]

So you suggest to directly maps 'fifo' commands to unix signals. It could be an interesting approach

[dmk23]

Yes - think of this as a different interface to submit the same commands...

[WST]

I confirm this could be a nice feature to have (if it’s easy to add).

[anthonyrisinger]

@dmk23, AFAIK, sending SIGINT to master process is a graceful shutdown; uWSGI waits for the workers to complete their request (for a configurable amount of time, grep --help for word mercy) before brutally killing them.

@unbit is talking about the unreliability of pidfiles in general, which is what the "convenience" tools rely on.

[dmk23]

I've confirmed that the shutdown on SIGINT is brutal and merciful one is not possible with any current signals... Only reload by signal can be merciful... Note the page linked in the OP.

[unbit]

@xtfxme the proposal made by @dmk23 is interesting (mapping signals to fifo commands), someone forced me to note that some filesystem (in the embedded world) does not support named pipes (nor unix sockets), so signals are the only viable approach (albeit suboptimal).

[anthonyrisinger]

What's the purpose of --reload-mercy and friends (which mention reload or shutdown) if SIGINT was never clean? I swear I tested this years ago and the callable was allowed to finish, but apparently I was mistaken, so this is very much news to me :-(

I'm trying now to recall situations where uWSGI is forced to brutally kill the worker... even in a busy loop with --py-reload-osafterfork (which I expected a KeyboardException to be thrown in Python) I can't seem to shutdown properly.

[anthonyrisinger]

I'm just amazed I never noticed this :-) all this time I was thinking the INT and HUP code paths were the same as I haven't worked on or walked those paths myself, with the only real difference being HUP continued... the good news is our apps have tolerated it just fine for a very long time!

[unbit]

The SIGINT triggers other SIGINT (a signal that workers are allowed to trap) and wait for processes death. It could not happen (maybe for a stuck worker, or an overriden signal). That is why mercy is required (in addition to graceful reloads). So technically SIGINT is not very brutal, this is why you probably never noticed it

[anthonyrisinger]

The mapping of OS sigs to management chars does seem interesting, but the OP, and I suppose others, just want to be able to kill the stack without interrupting any active requests -- couldn't we do something like (pseudo):

def INT_handler():
    return HUP_handler(restart=False)

def HUP_handler(restart=True):
    ...
    if restart:
        execv(...)
    else:
        exit(0)

To me, a graceful shutdown is little more than a graceful restart that never actually restarts. Would there be problems with listen queue filling during this time, and then they are lost after the process dies? Or what am I missing here?

[dmk23]

I suppose uWSGI could respond to the remaining requests in the queue with 503, when shutting down.

[NicolasLM]

Any news on this issue? I still couldn't find a way to shutdown uwsgi without killing the ongoing requests.

[unbit]

@NicolasLM this is about mapping UNIX signals to master-fifo commands for those setups where fifos are not available. You can safely graceful shutdown using them.

http://uwsgi-docs.readthedocs.org/en/latest/MasterFIFO.html

[NicolasLM]

Looking at the base issue, it seems pretty close to what I'm trying to achieve: running uwsgi as PID 1 in a Docker container. When Docker stops a container it sends a SIGTERM to the process with PID 1, here the uwsgi master.

There is the --die-on-term flag in uwsgi that allows to kill workers instead of reloading them when receiving a SIGTERM, why not adding --stop-on-term to let the workers finish their job?

I understand that pipes are way more flexible than a bunch of signals but in some environments signals are easier.

[unbit]

In master and uwsgi-2.0 branch there is a new hook action called "unix_signal" that allows you to remap any unix signal number to a specific function symbol:

./uwsgi --hook-master-start "unix_signal:1 gracefully_kill_them_all" --http-socket :9090 -M

will call gracefully_kill_them_all() every time signal 1 is received

This is obviously a way more generic solution to the problem described in this issue

[NicolasLM]

Great, it works! This a pretty flexible approach. Thanks.

[ionelmc]

Is there a list of all the available function symbols? Can those functions be user-defined?

[anthonyrisinger]

Looks like anything in the process space

[unbit]

@ionelmc yes anything in the process address space, so eventually even a symbol exposed by a library

#include <stdio.h>

void hello_world(int signum) {
        printf("Hello World\n");
}

gcc -o libhello.so -shared hello.c

uwsgi --dlopen ./libhello.so --hook-master-start "unix_signal:1 hello_world" ...

[anthonyrisinger]

Does this allow signal handlers set by workers (python) a chance to respond? Atexit, etc?

What happens to backlogged connections during shutdown (in general)?

If the worker misbehaves, master still comes in after the time out with a brutal kill, yes? If things work well, will this be default?

Maybe interesting to note, embed_config would allow for a uWSGI build with this behavior, by default, today :-)

[askoretskiy]

Is it possible that somebody adds this to uwsgi documention? Maybe another section "Shutdown" with these recipes -- brutal shutdown and graceful one.

At the moment it is rather hard to find these recipes.

[braedon]

Sorry to bump this, but the solution described here still appears to be undocumented.

I also can't find an answer to this question (asked by @anthonyrisinger above):

What happens to backlogged connections during shutdown (in general)?

[unbit]

@braedon pull-requests for docs are way more welcomed than code one, so if anyone want to write it i will merge it for sure.

Regarding backlogged connections RST should be generated as no socket is no more bound to that address/port pair