falcon + gunicorn + gevent SSL monkey_patch crash loop

[guongle-ssense]

I have already raised this with the Datadog maintainers of dd-trace-py:
https://github.com/DataDog/dd-trace-py/issues/1624

Hoping to see if anyone has encountered this from the falcon front and would be able to have some guidance/solution instead.

Which version of falcon are you using?

2.0.0

Which version of the libraries are you using?

attrs==19.3.0
boto3==1.12.39
botocore==1.15.39
falcon==2.0.0
flake8==3.7.9
gevent==1.4.0
gunicorn==20.0.4
jsonschema==3.2.0
more-itertools==8.0.2
pyflakes==2.1.1
pyrsistent==0.15.6
python-dateutil==2.8.1
python-http-client==3.2.4
s3transfer==0.3.3
urllib3==1.25.8
zipp==0.6.0
datadog==0.35.0
openapi-core==0.13.3
dependency-injector==3.15.6
ddtrace==0.41.0

How can we reproduce your problem?

a simple falcon api.py file

import falcon

from src.components.controllers.healtcheck_controller import HealthCheckController
from src.components.enums.routes import Routes
from configs.general import APP_NAME

# Disable test coverage for this file
__test__ = False

app = falcon.API()

# Load healthcheck routes
health_check_controller = HealthCheckController()
app.add_route(str(Routes.HEALTH_CHECK.value), health_check_controller)

Then the command is:

ddtrace-run gunicorn src.api.api:app -b 0.0.0.0:80 --workers=3 --reload --worker-class gevent

What is the result that you get?

At first I would get this:

/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py:53: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['botocore.httpsession (/usr/local/lib/python3.7/site-packages/botocore/httpsession.py)', 'urllib3.util (/usr/local/lib/python3.7/site-packages/urllib3/util/__init__.py)', 'urllib3.util.ssl_ (/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py)'].
  monkey.patch_all()
````

Then as per:
https://github.com/DataDog/dd-trace-py/issues/506

The sequence is now:

import gevent.monkey
gevent.monkey.patch_all()
from requests.packages.urllib3.util.ssl_ import create_urllib3_context
create_urllib3_context()

import falcon
from ddtrace import tracer, patch
patch(falcon=True)

I have a new crash loop issue:
```bash
  File "/code/src/api/api.py", line 4, in <module>
    create_urllib3_context()
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 273, in create_urllib3_context
    context.options |= options
  File "/usr/local/lib/python3.7/ssl.py", line 518, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/local/lib/python3.7/ssl.py", line 518, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/local/lib/python3.7/ssl.py", line 518, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 485 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

I have also tried the --preload directive:

ddtrace-run gunicorn src.api.api:app -b 0.0.0.0:80 --workers=3 --preload --reload --worker-class gevent
- DATADOG TRACER DIAGNOSTIC - Agent not reachable. Exception raised: [Errno 111] Connection refused
- DATADOG TRACER DIAGNOSTIC - Agent not reachable. Exception raised: [Errno 111] Connection refused
/code/src/api/api.py:2: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['botocore.httpsession (/usr/local/lib/python3.7/site-packages/botocore/httpsession.py)', 'urllib3.util (/usr/local/lib/python3.7/site-packages/urllib3/util/__init__.py)', 'urllib3.util.ssl_ (/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py)'].
  gevent.monkey.patch_all()

Error: maximum recursion depth exceeded while calling a Python object

What is the result that you expected?

for my app to be up without the crash loop

Anyone faced this in falcon? I've seen the Django and other python solutions and followed those, so far the common theme is to monkey_patch all first however I have tried that but the crash still happens.

[open-collective-bot[bot]]

Hi :wave:,

Thanks for using Falcon. The large amount of time and effort needed to
maintain the project and develop new features is not sustainable without
the generous financial support of community members like you.

Please consider helping us secure the future of the Falcon framework with a
one-time or recurring donation.

Thank you for your support!

[vytas7]

Hello @guongle-ssense ,
FWIW I have seen this recently in both a Falcon service, and a service using another framework, and indeed, monkey-patching as early as possible helped in both cases.

Unfortunately I don't think Falcon can do much more here... Probably the issue happens well before falcon is even imported.

Could you try peeling off any other modules, arriving at a some sort of MRE, or alternatively, start from a super minimal Falcon hello world, and try adding stuff until you see issues arising?

[vytas7]

@guongle-ssense ,
Adding to my previous answer, have you checked related Gunicorn issues like https://github.com/benoitc/gunicorn/issues/1056 (an old one but still has interesting Gevent discussions regarding patching as early as possible); see also an idea by @jamadden within: https://github.com/benoitc/gunicorn/issues/1056#issuecomment-115428187 , as well as worker hooks discussed in the issue: https://docs.gunicorn.org/en/stable/settings.html#server-hooks : you could try pre_fork or even on_starting.

Furthermore, you could try running Gunicorn programmatically as an application, such as, for instance, in this example by @jmvrbanac : https://github.com/jmvrbanac/falcon-example/blob/master/example/__main__.py
With this approach, you can make sure nothing else gets between the top of your script, and the monkey-patching.

Moreover, if your application does not require specifically Gunicorn, you could give the popular uWSGI a spin. uWSGI can be clumsy to configure with all the bells and whistles, but it is at least a pure C application less affected by Python import details, and what is more, you can configure uWSGI to ensure monkey-patching is even done for you before anything else: https://uwsgi-docs.readthedocs.io/en/latest/Gevent.html#monkey-patching

[guongle-ssense]

@vytas7
thanks for responding and offering some suggestions.
I will try a minimal approach to falcon and see what is the root cause. if at the root even a hello world falls we can go from there.

As far as i can describe before import falcon I have attempted to load everything (geven + monkeypatch) and the executing command is:

ddtrace-run gunicorn src.api.api:app -b 0.0.0.0:80 --workers=3 --reload --worker-class gevent

so folder structure is simple: src/api/api.py with

import falcon

from src.components.controllers.healtcheck_controller import HealthCheckController
from src.components.enums.routes import Routes
from configs.general import APP_NAME

# Disable test coverage for this file
__test__ = False

app = falcon.API()

# Load healthcheck routes
health_check_controller = HealthCheckController()
app.add_route(str(Routes.HEALTH_CHECK.value), health_check_controller)

what I will do is remove the controller file and do a simple POST and go from there.

I have attempted the patch from ddtrace-agent: http://pypi.datadoghq.com/trace/docs/web_integrations.html#falcon

but that also doesn't solve the crash loop.

right now the API is performant under gunicorn, i could go the other route of uWSGI. Is there any gotcha (or written doc) for synergie between falcon + uWSGI?

[vytas7]

Hi again @guongle-ssense ,
Both Gunicorn and uWSGI are pretty popular choices to run Falcon WSGI applications, and I'm unaware of any specific gotchas, at least nothing specific to just Falcon. uWSGI, itself, can be a bit daunting at times given the number of available configuration options, but you should be fine by sticking to the basics. In fact, we even have an NGINX+uWSGI deployment guide by @nZac out there: https://falcon.readthedocs.io/en/stable/deploy/nginx-uwsgi.html .

Performance wise, uWSGI should beat Gunicorn handily, unless you are using PyPy. Regarding PyPy, IIRC uWSGI+PyPy+Gevent combo is unsupported (I may be out of date though), so you'd have to stick to Gunicorn if that was the case.

Or, if you liked to stay on Gunicorn for any other reason, I would, as said, try prefork hooks and/or running Gunicorn programmatically by importing gunicorn.app.base.BaseApplication and other needed gunicorn bits. In the latter approach, you could literally make sure that the gevent.monkey stuff is done prior to any gunicorn import.

[jmvrbanac]

Running Gunicorn programmatically via it's own entrypoint does solve this issue as this is what my team does; The only real downside to this approach is that you also will need to wrap the WSGI app with your monitoring tool programmatically as well.

[guongle-ssense]

@vytas7
thanks for the links. No PyPy here.

@jmvrbanac
So basically something of this setup:
http://pypi.datadoghq.com/trace/docs/web_integrations.html#falcon

instead of letting ddtrace-run ... right? (or is there more...)

[vytas7]

@guongle-ssense That's right; regardless of the chosen solution, you simply cannot use ddtrace-run because it may execute Python code before you even have a chance to monkey-patch anything.

[Kyle-Verhoog]

:wave: @guongle-ssense @vytas7 this appears to indeed be an issue with ddtrace-run importing a module which ultimately results in the ssl module being imported before any gevent monkey patching can be done, which was a known issue with ddtrace-run. I've put up a fix here: https://github.com/DataDog/dd-trace-py/pull/1629.

Feel free to close this issue 🙂

[vytas7]

Thanks @Kyle-Verhoog :+1: :rocket:

@Kyle-Verhoog , needless to say, should DataDog need any advice, tips or opinions on the Falcon integration, do not hesitate to reach out to us either by creating an issue in this tracker, or simply dropping a message on our Gitter channels!