Uvicorn: Subprocess returncode is not detected when running Gunicorn with Uvicorn (with fix PR companion)

[tiangolo]

Checklist

• [x] The bug is reproducible against the latest release and/or master.
• [x] There are no similar issues or pull requests to fix it yet.

Describe the bug

When starting Gunicorn with Uvicorn worker(s), if the app uses subprocess to start other processes and captures the output, their returncode is in most cases 0, even if the actual exit code was 1.

To reproduce

Take this minimal FastAPI app (or replace with Starlette), main.py:

import subprocess

from fastapi import FastAPI

app = FastAPI()

@app.post("/run")
def run_subprocess():
    result = subprocess.run(
        ["python", "-c", "import sys; sys.exit(1)"], capture_output=True
    )
    return {"returncode": result.returncode}

Then run it with:

$ gunicorn -k uvicorn.workers.UvicornWorker main:app

Open the browser at http:127.0.0.1:8000/docs and send a request to /run.

Expected behavior

The detected returncode should always be 1, as the subprocess always exits with 1.

Actual behavior

In most of the cases it will return a returncode of 0. Strangely enough, in some cases, it will return a returncode of 1.

Debugging material

This is because the UvicornWorker, which inherits from the base Gunicorn worker, declares a method init_signals() (overriding the parent method) but doesn't do anything. I suspect it's because the signal handlers are declared in the Server.install_signal_handlers() with compatibility with asyncio.

But the UvicornWorker process is started with os.fork() by Gunicorn (if I understand correctly) and by the point it is forked, the Gunicorn "Arbiter" class (that handles worker processes) already set its own signal handlers.

And the signal handlers in the Gunicorn base worker reset those handlers, but the UvicornWorker doesn't. So, when a process started with subprocessing is terminated, the SIGCHLD signal is handled by the Gunicorn Arbiter (as if the terminated process was a worker) instead of by the UvicornWorker.

Disclaimer: why the SIGCHLD signal handling in the Gunicorn Arbiter alters the returncode of a process run with subprocess, when capturing output, is still a mystery to me. But I realized the signal handler in the Arbiter is expected to handle dead worker processes. And worker subclasses all seem to reset the signal handlers to revert those signals set by the Arbiter.

I'm also submitting a PR to fix this: https://github.com/encode/uvicorn/pull/895. It's just 3 lines of code. But debugging it and finding it took me almost a week. :sweat_smile:

Environment

• OS / Python / Uvicorn version: just run uvicorn --version: Running uvicorn 0.13.1 with CPython 3.8.5 on Linux (it's actually installed from source, for debugging)
• Gunicorn version (also installed from source, for debugging): gunicorn (version 20.0.4)
• The exact command you're running uvicorn with, all flags you passed included. If you run it with gunicorn please do the same. If there is a reverse-proxy involved and you cannot reproduce without it please give the minimal config of it to reproduce.

$ gunicorn -k uvicorn.workers.UvicornWorker main:app

Additional context

I'm pretty sure this issue https://github.com/encode/uvicorn/issues/584 is related to the same problem.

[florimondmanca]

Was able to reproduce with a Starlette app as well.

import subprocess

from starlette.applications import Starlette
from starlette.routing import Route
from starlette.responses import JSONResponse


async def run_subprocess(request):
    result = subprocess.run(
        ["python", "-c", "import sys; sys.exit(1)"], capture_output=True
    )
    return JSONResponse({"returncode": result.returncode})


app = Starlette(routes=[Route("/run", run_subprocess, methods=["POST"])])

With plain Uvicorn: always returns 1.

$ uvicorn app:app
$ curl -X POST localhost:8000/run
{"returncode":1}

With Gunicorn: always getting 0

$ gunicorn -k uvicorn.workers.UvicornWorker app:app
$ curl -X POST localhost:8000/run
{"returncode":0}

When running off from #895, both cases always return 1. :-)

[tiangolo]

Awesome! Thanks @florimondmanca ! :rocket: :coffee: