Sentry: uWSGI process got Segmentation Fault after updating to 9.1.2

[pouyana]

Important Details

How are you running Sentry?

• [ ] On-Premise docker [Version xyz]
• [ ] Saas (sentry.io)
• [x] Other On-Premise (Ubuntu 18.04)

Description

After upgrading to the 9.1.2 version, the sentry web-worker gives segmentation faults and uses all the CPU resources available.

Steps to Reproduce

1. Update the existing sentry 9.1.1 to 9.1.2

Stack Trace

Aug 01 12:55:13 wrt3015 sentry[2862]: !!! uWSGI process 2956 got Segmentation Fault !!!
Aug 01 12:55:13 wrt3015 sentry[2862]: *** backtrace of 2956 ***
Aug 01 12:55:13 wrt3015 sentry[2862]: [Sentry] uWSGI worker 3(uwsgi_backtrace+0x2a) [0x55eb57dff48a]
Aug 01 12:55:13 wrt3015 sentry[2862]: [Sentry] uWSGI worker 3(uwsgi_segfault+0x23) [0x55eb57dff873]
Aug 01 12:55:13 wrt3015 sentry[2862]: /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7fc8deaf2f20]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(ssl
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(ssl
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(tls
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_new+0x3f2) [0x7fc8dff20a22]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0x23969
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0x24fc5
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(PQconnec
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0xfa28)
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(PQconnec
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/_psycopg.so(+0x129c1) [0x7fc8d347b9c1
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/_psycopg.so(+0x1374f) [0x7fc8d347c74f
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x13ccfa) [0x7fc8df219cfa]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fc8df131443]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyObject_CallFunction_SizeT+0xa1) [0x7fc8df2493e1]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/local/src/sentry/local/lib/python2.7/site-packages/psycopg2/_psycopg.so(+0xc268) [0x7fc8d3475268]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5f82) [0x7fc8df18a412]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fc8df2bc708]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1806ed) [0x7fc8df25d6ed]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fc8df131443]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2786) [0x7fc8df186c16]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x727c) [0x7fc8df18b70c]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fc8df2bc708]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1806ed) [0x7fc8df25d6ed]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fc8df131443]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2786) [0x7fc8df186c16]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fc8df2bc708]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6364) [0x7fc8df18a7f4]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x727c) [0x7fc8df18b70c]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fc8df2bc708]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x17c108) [0x7fc8df259108]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x13ccfa) [0x7fc8df219cfa]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fc8df131443]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x208e) [0x7fc8df18651e]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x727c) [0x7fc8df18b70c]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fc8df2bc708]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x180609) [0x7fc8df25d609]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fc8df131443]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1ac13c) [0x7fc8df28913c]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fc8df131443]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x140795) [0x7fc8df21d795]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x125d12) [0x7fc8df202d12]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xb3009) [0x7fc8df190009]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8165) [0x7fc8df18c5f5]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fc8df2bc708]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1806ed) [0x7fc8df25d6ed]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fc8df131443]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2786) [0x7fc8df186c16]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fc8df2bc708]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6364) [0x7fc8df18a7f4]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fc8df2bc708]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6364) [0x7fc8df18a7f4]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7d8) [0x7fc8df2bc708]
Aug 01 12:55:13 wrt3015 sentry[2862]: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6364) [0x7fc8df18a7f4]

What you expected to happen

That update works and sentry website goes up accordingly.

Possible Solution

I think it has to do with psycopg2 library and SSL connection to the PostgreSQL sever. That is at least what is written in the stack trace.

[BYK]

Thanks for reporting! At this point, I recommend overriding psycopg2 requirement or switching to the Docker-based install as it seems to work.

You may also try updating your openssl libraries first or recompiling uwsgi as it seems like a mismatch.

As we won't be pursuing the fix for this, I'm closing the issue but feel free to continue the discussion for potential remedies over at the forums: https://forum.sentry.io/

[mikicz]

For anyone who finds this issue, I had the same issue on Debian 10, fixed it by manually updating the newest version of psycopg2-binary.