Protobuf: Python finalize will cause infinite loop

[Falldog]

callstack from gdb

#0  erase (__k=@0x7ffd59a53260: 0x2158450, this=0x7f86d4d2dca0 <google::protobuf::python::descriptor_pool_map>) at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/tr1/hashtable.h:1041
#1  google::protobuf::python::cdescriptor_pool::Dealloc (self=0x7f86d49178f0) at google/protobuf/pyext/descriptor_pool.cc:153
#2  0x00007f86dc7f671f in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3  0x00007f86dc7b4b46 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4  0x00007f86dc7de2bf in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f86dc7bd85b in PyDict_SetItem () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f86dc775ca6 in _PyModule_Clear () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f86dc77614d in PyImport_Cleanup () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f86dc7b388e in Py_Finalize () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x0000000000467451 in uwsgi_plugins_atexit ()
#10 0x00007f86dc2cb1a9 in __run_exit_handlers (status=30, listp=0x7f86dc6516c8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#11 0x00007f86dc2cb1f5 in __GI_exit (status=<optimized out>) at exit.c:104
#12 0x00000000004218cf in uwsgi_exit ()
#13 0x00000000004674a5 in end_me ()
#14 <signal handler called>
#15 0x00007f86dc38d693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#16 0x000000000045da33 in event_queue_wait ()
#17 0x000000000041f1be in wsgi_req_accept ()
#18 0x0000000000466496 in simple_loop_run ()
#19 0x000000000046a654 in uwsgi_ignition ()
#20 0x000000000046ee4d in uwsgi_worker_run ()
#21 0x000000000046f43f in uwsgi_run ()
#22 0x000000000041e62e in main ()

Environment:

Ubuntu 14.04 x64
Python 2.7.6
Nginx + uWsgi + Django + protobuf 3.3.0

Result:

Infinite loop and 100% CPU usage

Steps

• Stop uwsgi
• Can't reproduce it every time when stopping uwsgi

[pcheliniy]

We have the same problem.
centos 7.3.1611, python 2.7.5, Nginx 1.10.2
protobuf 3.0.0, uWSGI 2.0.14, Flask app

#0  erase (__k=@0x7ffd0bb1ae00: 0x28d8320, this=0x7f59a6e6e260 <google::protobuf::python::descriptor_pool_map>) at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/tr1/hashtable.h:1042
#1  google::protobuf::python::cdescriptor_pool::Dealloc (self=0x331f070) at google/protobuf/pyext/descriptor_pool.cc:155
#2  0x00007f59b83b5fcf in dict_dealloc () from /usr/lib64/libpython2.7.so.1.0
#3  0x00007f59b83d2a44 in subtype_dealloc () from /usr/lib64/libpython2.7.so.1.0
#4  0x00007f59b83b50ef in insertdict_by_entry () from /usr/lib64/libpython2.7.so.1.0
#5  0x00007f59b83b6ab0 in dict_set_item_by_hash_or_entry () from /usr/lib64/libpython2.7.so.1.0
#6  0x00007f59b83bb506 in _PyModule_Clear () from /usr/lib64/libpython2.7.so.1.0
#7  0x00007f59b842b6cf in PyImport_Cleanup () from /usr/lib64/libpython2.7.so.1.0
#8  0x00007f59b8436fce in Py_Finalize () from /usr/lib64/libpython2.7.so.1.0
#9  0x0000000000462cc1 in uwsgi_plugins_atexit ()
#10 0x00007f59b7d75a49 in __run_exit_handlers (status=0, listp=0x7f59b80f76c8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:77
#11 0x00007f59b7d75a95 in __GI_exit (status=<optimized out>) at exit.c:99
#12 0x000000000041e45f in uwsgi_exit ()
#13 0x0000000000465096 in simple_goodbye_cruel_world ()
#14 0x00000000004650c8 in goodbye_cruel_world ()
#15 0x000000000041f222 in uwsgi_close_request ()
#16 0x0000000000461d2d in simple_loop_run ()
#17 0x0000000000465cf4 in uwsgi_ignition ()
#18 0x000000000046a4fd in uwsgi_worker_run ()
#19 0x000000000046aaef in uwsgi_run ()
#20 0x000000000041b1fe in main ()

[aukaost]

Are you guys using Python wheels or building from source?

[Falldog]

I installed protobuf by pip

[Smirl]

+100

Environment

• Python 2.7.10
• nginx==1.12.1
• protobuf==3.0.0
• uWSGI==2.0.12
• Amazon Linux

Info

We are seeing the same issue when we upgraded from pip==6.0.8 to pip==9.0.1. It looks like this later version of pip prefers the Platform Wheel protobuf-3.0.0-cp27-cp27mu-manylinux1_x86_64.whl where pip==6.0.8 prefers the Universal Wheel protobuf-3.0.0-py2.py3-none-any.whl.

The universal wheel doesn't have the *.so files where the platform wheel does. We are seeing this issue even without the PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION environment variable set.

When we use pip install --no-binary protobuf (i.e. install from source dist) we do not see the high CPU issue, nor did we see the issue using older version of pip (i.e. install from universal wheel).

[anandolee]

The problem is: descriptor_pool_map is defined as a static object but atxit handlers are called AFTER C++ destructors of static objects.
We have changed descriptor_pool_map to pointer in internal code which should fix this issue. Expect to see the fix in next relase.

Will close this issue for clean up. Feel free to reopen if it is still an issue after next release.