Gunicorn: Shutting down Gunicorn gracefully when embedded in another app with multiprocessing

[al4]

I'm using Gunicorn in an application which starts a couple of multiprocessing.Process instances, and I'm having trouble exiting gracefully. The application calls gunicorn.app.base.BaseApplication (as per http://docs.gunicorn.org/en/stable/custom.html), and on starting the application everything is fine. But when I do Ctrl+C (SIGINT), it appears that the worker process is attempting to join and kill the parent:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 144, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process

I added a print to multiprocessing.process to see what PIDs were involved, in the log below they look like ('parent', 4488, 'me', 4488) (where parent is self._parent_pid and me os.getpid()).

The only useful reference I could find for this exception was http://stackoverflow.com/questions/37692262/#37709169 , which suggests a few solutions, but the only solution that is in my control
(importing multiprocessing after executing fork()) would be impossible as I need to import multiprocessing on module load, and Gunicorn is started in a function.

Is there something I can do to close down the Gunicorn workers cleanly? I can't publish this code at this stage but could contrive an example if it would help. I am using gunicorn 19.6.0 from pypi, full log follows:

[2016-11-14 19:34:28 +0000] [4488] [INFO] Starting gunicorn 19.6.0
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
[     MainProcess:    INFO  __main__.py:125] Database is configured
[2016-11-14 19:34:28 +0000] [4488] [INFO] Listening at: http://0.0.0.0:5000 (4488)
[     MainProcess:    INFO  manager.py:38] Starting with pid: TaskConsumer-1
[     MainProcess:    INFO  manager.py:112] Starting with pid: ResultConsumer-3
[     MainProcess:    INFO  manager.py:151] Starting 1 TaskConsumers
[     MainProcess:    INFO  manager.py:155] Starting 1 ResultConsumers
[2016-11-14 19:34:28 +0000] [4488] [INFO] Using worker: sync
[2016-11-14 19:34:28 +0000] [4502] [INFO] Booting worker with pid: 4502
^C[  TaskConsumer-1:    INFO  util.py:17] [TaskConsumer-1] got sigint, exiting
[ResultConsumer-3:    INFO  util.py:17] [ResultConsumer-3] got sigint, exiting
[2016-11-14 19:34:37 +0000] [4488] [INFO] Handling signal: int
[2016-11-14 19:34:37 +0000] [4502] [INFO] Worker exiting (pid: 4502)
('parent', 4488, 'me', 4502)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 144, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 144, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
[2016-11-14 19:34:37 +0000] [4488] [INFO] Shutting down: Master
('parent', 4488, 'me', 4488)
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
    finalizer()
  File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 625, in _finalize_manager
    process.terminate()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 137, in terminate
    self._popen.terminate()
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 171, in terminate
    os.kill(self.pid, signal.SIGTERM)
OSError: [Errno 3] No such process

[al4]

After further research I've answered my own question... basically, one should not run Gunicorn and multiprocessing in the same process.

The traceback is coming from an atexit handler registered by the multiprocessing module. It adds this to the parent process when multiprocessing is imported: https://hg.python.org/cpython/file/2.7/Lib/multiprocessing/util.py#l330

Any forks will therefore have the same handler registered, and when it is called in the gunicorn worker it understandably bombs.

Thus I will close this as it is clearly not a problem with Gunicorn!

With regards to resolving the issue, I'm going to change tack and use systemd to spawn separate processes rather than have all the processes forking from one python master process.

[comtihon]

I don't know, whether it helps you but I found an ugly solution for myself.
I have a flask application with several gunicorn workers. And also I spawn a service process via multiprocessing.Process from when_ready gunicorn hook.
I had the same AssertionError: can only join a child process when stopping the application, as workers share (perhaps, during the fork) the main process state and thought my service process is their child also.
Workaround for this is:

• start process with name: Process(target=self.run_linked, name='Service')
• on every worker start find this process by name and remove it from children:

def post_worker_init(worker):
    for child in process.active_children():
        if child.name == 'Service':
            process._children.remove(child)

[golf-player]

What worked for me is

import atexit
from multiprocessing.utils import _exit_function
atexit.unregister(_exit_function)

in the child processes. (I call it in post_worker_init)

[Lorne0]

Thanks to @golf-player it works!
Just a little typo: multiprocessing.utils -> multiprocessing.util

[abnercloud]

What worked for me is

import atexit
from multiprocessing.utils import _exit_function
atexit.unregister(_exit_function)

in the child processes. (I call it in post_worker_init)

Hello! Can you describe it in more detail?

[yungkit]

What worked for me is

import atexit
from multiprocessing.utils import _exit_function
atexit.unregister(_exit_function)

in the child processes. (I call it in post_worker_init)

Hello! Can you describe it in more detail?

def post_worker_init(worker):
    import atexit
    from multiprocessing.util import _exit_function
    atexit.unregister(_exit_function)
    worker.log.info("worker post_worker_init done, (pid: {})".format(worker.pid))

define this func in your gunicorn config python file