Sentry-python: Sentry with eventlet

[Randomneo]

Environment

self-hosted sentry 9.0.0

python 3.9/3.7
sentry_sdk version: 0.20.3
eventlet version: 0.30.1
greenlet version: 1.0.0

Steps to Reproduce

import eventlet
import sentry_sdk

eventlet.monkey_patch()

sentry_sdk.init('<sentry_sdk_dsn>')
sentry_sdk.capture_message('aoe')

Expected Result

getting message in sentry

Actual Result

Nothing happens. No errors thrown. No clear documentation for that problem. Here issue for eventlet.

If sentry_sdk can't be used with eventlet and there no possible way to combine them it should be in documentation.
Please correct me if i'm missing something.

[untitaker]

Can you add sentry_sdk.flush() as last line?

[Randomneo]

Same. Nothing happens. Without flush I got.

Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

Program ends by itself instantly.

With flush. No output and no message in sentry

[Randomneo]

Problem definitely in eventlet.monkey_patch() if that line is commented everything works fine.

[temoto]

Just for record, copying workaround from our tracker.

Calling monkey_patch before even importing sentry_sdk works.

import eventlet
eventlet.monkey_patch()

import sentry_sdk
...

I understand it's well below acceptable and yet don't have a better solution right now.

Successful fix would start with analyzing what side effects are produced by import sentry_sdk

[Randomneo]

@temoto Thanks alot. I had no idea that import sentry_sdk was the problem :)

[untitaker]

only work on import is this: https://github.com/getsentry/sentry-python/blob/37105d981fb116c60df2ea3d1e58a87b9c65fc21/sentry_sdk/utils.py#L840

However, flush etc should not really break regardless of whether contextvars or thread locals are used. So I think it's more related to threading patches that would affect sentry_sdk.worker.

[untitaker]

@Randomneo can you check Hub.current.client after init? Maybe locals are broken in a really bad way.

[Randomneo]

@untitaker I not quite get how I supposed to check it. But this code outputs:

import eventlet
import sentry_sdk
from sentry_sdk import Hub

eventlet.monkey_patch()

sentry_sdk.init('dsn')
print(Hub.current.client)
sentry_sdk.capture_message('aoe')

Output:

<sentry_sdk.client._Client object at 0x7f5d8354edf0>                                                                                                                                                             
Sentry is attempting to send 0 pending error messages                                                                                                                                                            
Waiting up to 2 seconds
Press Ctrl-C to quit

It also easy to reproduce on your machine using code sample in issue description. I'm using latest stable versions of packages

[untitaker]

I'm getting the following recursionerror in addition when it's broken. As you said, if I move the SDK import after the monkeypatch, the issue disappears. You need to add debug=True to the init() call to see the stacktrace.

Traceback (most recent call last):
  File "/Users/untitaker/projects/sentry-python/sentry_sdk/transport.py", line 326, in send_event_wrapper
    self._send_event(event)
  File "/Users/untitaker/projects/sentry-python/sentry_sdk/transport.py", line 232, in _send_event
    self._send_request(
  File "/Users/untitaker/projects/sentry-python/sentry_sdk/transport.py", line 176, in _send_request
    response = self._pool.request(
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/connection.py", line 392, in connect
    self.ssl_context = create_urllib3_context(
  File "/Users/untitaker/projects/sentry-python/.tox/py3.9-flask-dev/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 303, in create_urllib3_context
    context.options |= options
  File "/Users/untitaker/.pyenv/versions/3.9.0/lib/python3.9/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/Users/untitaker/.pyenv/versions/3.9.0/lib/python3.9/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/Users/untitaker/.pyenv/versions/3.9.0/lib/python3.9/ssl.py", line 602, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 479 more times]
RecursionError: maximum recursion depth exceeded

In addition, the issue can be reproduced without Sentry. The urllib3 usage is straight from their README:

import eventlet

import urllib3

eventlet.monkey_patch()
http = urllib3.PoolManager()

r = http.request('GET', 'https://httpbin.org/robots.txt')

This will crash in the same way.

[temoto]

@untitaker

I'm getting the following recursionerror in addition when it's broken. As you said, if I move the SDK import after the monkeypatch, the issue disappears. You need to add debug=True to the init() call to see the stacktrace.

This should be fixed in just released eventlet v0.30.2 but (AFAIK) is not related to current problem.

My two cents to checking Hub.current.client:

# test.py
import sentry_sdk
sentry_sdk.init('http://1234@foo/1')
print("[before eventlet] Hub.current.client:",sentry_sdk.Hub.current.client)

import eventlet
eventlet.monkey_patch()
print("[ after eventlet] Hub.current.client:", sentry_sdk.Hub.current.client)

[before eventlet] Hub.current.client: <sentry_sdk.client._Client object at 0x10b2b6198>
[ after eventlet] Hub.current.client: <sentry_sdk.client._Client object at 0x10b2b6198>

At the very least, it refers to same object after patching, so that's something about thread locals. Still maybe thread locals are messed up in another place.

[untitaker]

0.30.2 exhibits the same problem for me. You need to both send an event via the SDK and have debug=True added