Uwsgi: segfault on 2.0.14 (Stretch Debian package)

[ilrico]

uwsgi from Debian package on Stretch
ii uwsgi-emperor 2.0.14+20161117-3 amd64
ii python3.5 3.5.3-1 amd64

!!! uWSGI process 55154 got Segmentation Fault !!!
* backtrace of 55154
/usr/bin/uwsgi-core(uwsgi_backtrace+0x35) [0x7f747e38ba65]
/usr/bin/uwsgi-core(uwsgi_segfault+0x23) [0x7f747e38be33]
/lib/x86_64-linux-gnu/libc.so.6(+0x33030) [0x7f747bf34030]
/usr/local/lib/python3.5/dist-packages/google/protobuf/pyext/_message.cpython-35m-x86_64-linux-gnu.so(+0xa34e0) [0x7f745ea604e0]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyDict_SetItem+0x23b) [0x7f747a8914ab]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(_PyModule_ClearDict+0x237) [0x7f747a84a837]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyImport_Cleanup+0x3fc) [0x7f747a9e121c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(Py_Finalize+0x64) [0x7f747a9e1434]
/usr/bin/uwsgi-core(uwsgi_plugins_atexit+0x81) [0x7f747e388991]
/lib/x86_64-linux-gnu/libc.so.6(+0x35910) [0x7f747bf36910]
/lib/x86_64-linux-gnu/libc.so.6(+0x3596a) [0x7f747bf3696a]
/usr/bin/uwsgi-core(+0x2da6f) [0x7f747e33ea6f]
/usr/bin/uwsgi-core(gracefully_kill+0xc6) [0x7f747e38a7e6]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7f747dedf0c0]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x33) [0x7f747bfea0f3]
/usr/bin/uwsgi-core(event_queue_wait+0x33) [0x7f747e37ec43]
/usr/bin/uwsgi-core(wsgi_req_accept+0xd2) [0x7f747e33c082]
/usr/bin/uwsgi-core(simple_loop_run+0xb6) [0x7f747e387756]
/usr/bin/uwsgi-core(uwsgi_ignition+0x1c3) [0x7f747e38c0b3]
/usr/bin/uwsgi-core(uwsgi_worker_run+0x28d) [0x7f747e38f7cd]
/usr/bin/uwsgi-core(+0x7edfc) [0x7f747e38fdfc]
/usr/bin/uwsgi-core(+0x2a38e) [0x7f747e33b38e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f747bf212b1]
/usr/bin/uwsgi-core(_start+0x2a) [0x7f747e33b3ba]
end of backtrace *

[xrmx]

Could you please check you are not doing something silly with protobuf? Also reproduce with latest uwsgi.

[yoloseem]

@xrmx @ilrico reproduced on latest uWSGI 2.0.17. I'm getting this segfault when I execute uwsgi --reload [pid] command. And not doing something silly with protobuf.

[pid: 1346|app: 0|req: 56730/281056] 172.31.16.77 () {52 vars in 896 bytes} [Thu Mar 15 00:32:21 2018] POST /app/api => generated 144 bytes in 33 msecs (HTTP/1.1 200) 3 headers in 121 bytes (2 switches on core 0)
...gracefully killing workers...
Gracefully killing worker 1 (pid: 1490)...
Gracefully killing worker 2 (pid: 1450)...
Gracefully killing worker 3 (pid: 1476)...
Gracefully killing worker 4 (pid: 1346)...
Gracefully killing worker 5 (pid: 1427)...
!!! uWSGI process 1427 got Segmentation Fault !!!
*** backtrace of 1427 ***
/home/service/.venv/bin/uwsgi(uwsgi_backtrace+0x2c) [0x46718c]
/home/service/.venv/bin/uwsgi(uwsgi_segfault+0x21) [0x467551]
/lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fbeec0534b0]
/home/service/.venv/lib/python3.5/site-packages/google/protobuf/pyext/_message.cpython-35m-x86_64-linux-gnu.so(+0xa34e0) [0x7fbed96874e0]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyDict_Clear+0x1b7) [0x7fbeec49a7c7]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xb2999) [0x7fbeec49a999]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x2308ba) [0x7fbeec6188ba]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(_PyGC_CollectNoFail+0x31) [0x7fbeec5bcc91]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyImport_Cleanup+0x360) [0x7fbeec5d8b20]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(Py_Finalize+0x64) [0x7fbeec5d8dd4]
/home/service/.venv/bin/uwsgi(uwsgi_plugins_atexit+0x71) [0x464531]
/lib/x86_64-linux-gnu/libc.so.6(+0x39ff8) [0x7fbeec057ff8]
/lib/x86_64-linux-gnu/libc.so.6(+0x3a045) [0x7fbeec058045]
/home/service/.venv/bin/uwsgi() [0x41e93f]
/home/service/.venv/bin/uwsgi(gracefully_kill+0xb7) [0x466037]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7fbeed3d0390]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x33) [0x7fbeec125a13]
/home/service/.venv/bin/uwsgi(event_queue_wait+0x33) [0x45a9f3]
/home/service/.venv/bin/uwsgi(wsgi_req_accept+0x152) [0x41bfa2]
/home/service/.venv/bin/uwsgi(simple_loop_run+0xb6) [0x463396]
/home/service/.venv/bin/uwsgi(simple_loop+0xe) [0x46319e]
/home/service/.venv/bin/uwsgi(uwsgi_ignition+0x22a) [0x46783a]
/home/service/.venv/bin/uwsgi(uwsgi_worker_run+0x27d) [0x46bf3d]
/home/service/.venv/bin/uwsgi(uwsgi_init_worker_mount_apps+0) [0x46c520]
/home/service/.venv/bin/uwsgi() [0x41b3be]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fbeec03e830]
/home/service/.venv/bin/uwsgi(_start+0x29) [0x41b3e9]
*** end of backtrace ***
!!! uWSGI process 1346 got Segmentation Fault !!!
(and all other processes also got same segmentations faults)
*** end of backtrace ***
worker 4 buried after 3 seconds
worker 5 buried after 3 seconds
worker 3 buried after 3 seconds
worker 1 buried after 3 seconds
worker 2 buried after 4 seconds
binary reloading uWSGI...
chdir() to /home/service/service
closing all non-uwsgi socket fds > 2 (max_fd = 65535)...
found fd 3 mapped to socket 0 (/tmp/uwsgi_sock)
found fd 4 mapped to socket 1 (0.0.0.0:8000)
running /home/service/.venv/bin/uwsgi
[uWSGI] getting INI configuration from env/uwsgi.ini
*** Starting uWSGI 2.0.17 (64bit) on [Thu Mar 15 00:32:26 2018] ***
compiled with version: 5.4.0 20160609 on 14 March 2018 05:37:59
os: Linux-4.4.0-101-generic #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017
nodename: mobile-api
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /home/service/service
detected binary path: /home/service/.venv/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 15727
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 65535
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited UNIX address /tmp/uwsgi_sock fd 3
uwsgi socket 1 inherited INET address 0.0.0.0:8000 fd 4
Python version: 3.5.2 (default, Nov 23 2017, 16:37:01)  [GCC 5.4.0 20160609]
Python main interpreter initialized at 0x237dfd0
python threads support enabled
your server socket listen backlog is limited to 512 connections
your mercy for graceful operations on workers is 60 seconds
mapped 437424 bytes (427 KB) for 5 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x237dfd0 pid: 20167 (default app)
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 20167)
spawned uWSGI worker 1 (pid: 1605, cores: 1)
spawned uWSGI worker 2 (pid: 1606, cores: 1)
spawned uWSGI worker 3 (pid: 1607, cores: 1)
spawned uWSGI worker 4 (pid: 1608, cores: 1)
spawned uWSGI worker 5 (pid: 1609, cores: 1)
Thu Mar 15 00:32:27 2018 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /ping (ip 172.31.1.250) !!!
Thu Mar 15 00:32:27 2018 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /ping (172.31.1.250)
OSError: write error
[pid: 1608|app: 0|req: 1/1] 172.31.1.250 () {34 vars in 405 bytes} [Thu Mar 15 00:32:27 2018] GET /ping => generated 0 bytes in 138 msecs (HTTP/1.1 200) 3 headers in 0 bytes (0 switches on core 0)

[ilrico]

yes, typical arrogant IT behaviour: users always doing "silly" things...

[xrmx]

@ilrico sorry?

[yoloseem]

Oh, I don't mean to nitpick that...

[unbit]

@yoloseem it looks like is the finalization phase that is breaking protobuf stuff, try adding

--skip-atexit-teardown

this should skip the Py_Finalize() call

[yoloseem]

Segfaults have disappeared after adding --skip-atexit-teardown. Thank you for quick help! :cake: