Protobuf: segfault during uwsgi reload/restart/stop using protobuf 3.6.1

[vpassama]

What version of protobuf and what language are you using?
Version: v3.6.1
Language: Python (C++ implementation)

What operating system (Linux, Windows, ...) and version?
Debian 8

* Extra info about env *
uwsgi 2.0.7-debian
nginx/1.10.3

What runtime / compiler are you using (e.g., python version or gcc version)
gcc (Debian 4.9.2-10+deb8u1) 4.9.2
Python v2.7.9

What did you do?
Steps to reproduce the behavior:

1. Get protobuf from github then use the tag 3.6.1
2. Compilation (OK + tests OK)
3. Installation OK
4. Build for python with cpp implementation python setup.py build --cpp_implementation
   In this step I had to add 1 line in setup.py in order to make it work:

extra_compile_args.append('-std=c++11')

Otherwise I get this error:

building 'google.protobuf.pyext._message' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/google
creating build/temp.linux-x86_64-2.7/google/protobuf
creating build/temp.linux-x86_64-2.7/google/protobuf/pyext
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I. -I../src -I/usr/include/python2.7 -c google/protobuf/pyext/repeated_composite_container.cc -o build/temp.linux-x86_64-2.7/google/protobuf/pyext/repeated_composite_container.o -Wno-write-strings -Wno-invalid-offsetof -Wno-sign-compare
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /usr/include/c++/4.9/mutex:35:0,
                 from ../src/google/protobuf/stubs/mutex.h:33,
                 from ../src/google/protobuf/stubs/common.h:52,
                 from ./google/protobuf/pyext/message.h:42,
                 from ./google/protobuf/pyext/repeated_composite_container.h:43,
                 from google/protobuf/pyext/repeated_composite_container.cc:34:
/usr/include/c++/4.9/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
 #error This file requires compiler and library support for the \
  ^
In file included from ./google/protobuf/pyext/message.h:43:0,
                 from ./google/protobuf/pyext/repeated_composite_container.h:43,
                 from google/protobuf/pyext/repeated_composite_container.cc:34:
./google/protobuf/pyext/thread_unsafe_shared_ptr.h:51:7: warning: identifier ‘nullptr’ is a keyword in C++11 [-Wc++0x-compat]
       : ptr_(ptr), refcount_(ptr ? new RefcountT(1) : nullptr) {
       ^
In file included from ../src/google/protobuf/arena.h:51:0,
                 from ../src/google/protobuf/message.h:118,
                 from ../src/google/protobuf/dynamic_message.h:45,
                 from google/protobuf/pyext/repeated_composite_container.cc:41:
../src/google/protobuf/arena_impl.h:311:3: warning: identifier ‘static_assert’ is a keyword in C++11 [-Wc++0x-compat]
   static_assert(kBlockHeaderSize % 8 == 0,
   ^
In file included from ../src/google/protobuf/stubs/common.h:52:0,
                 from ./google/protobuf/pyext/message.h:42,
                 from ./google/protobuf/pyext/repeated_composite_container.h:43,
                 from google/protobuf/pyext/repeated_composite_container.cc:34:
../src/google/protobuf/stubs/mutex.h:50:20: warning: defaulted and deleted functions only available with -std=c++11 or -std=gnu++11
   WrappedMutex() = default;
                    ^
../src/google/protobuf/stubs/mutex.h:58:8: error: ‘mutex’ in namespace ‘std’ does not name a type
   std::mutex mu_;
        ^
../src/google/protobuf/stubs/mutex.h: In member function ‘void google::protobuf::internal::WrappedMutex::Lock()’:
../src/google/protobuf/stubs/mutex.h:51:17: error: ‘mu_’ was not declared in this scope
   void Lock() { mu_.lock(); }
                 ^
../src/google/protobuf/stubs/mutex.h: In member function ‘void google::protobuf::internal::WrappedMutex::Unlock()’:
../src/google/protobuf/stubs/mutex.h:52:19: error: ‘mu_’ was not declared in this scope
   void Unlock() { mu_.unlock(); }
                   ^
../src/google/protobuf/stubs/mutex.h: At global scope:
../src/google/protobuf/stubs/mutex.h:61:7: error: expected nested-name-specifier before ‘Mutex’
 using Mutex = WrappedMutex;
       ^
../src/google/protobuf/stubs/mutex.h:66:28: error: expected ‘)’ before ‘*’ token
   explicit MutexLock(Mutex *mu) : mu_(mu) { this->mu_->Lock(); }
                            ^
../src/google/protobuf/stubs/mutex.h:69:3: error: ‘Mutex’ does not name a type
   Mutex *const mu_;
   ^
../src/google/protobuf/stubs/mutex.h: In destructor ‘google::protobuf::internal::MutexLock::~MutexLock()’:
../src/google/protobuf/stubs/mutex.h:67:24: error: ‘class google::protobuf::internal::MutexLock’ has no member named ‘mu_’
   ~MutexLock() { this->mu_->Unlock(); }
                        ^
../src/google/protobuf/stubs/mutex.h: At global scope:
../src/google/protobuf/stubs/mutex.h:80:33: error: expected ‘)’ before ‘*’ token
   explicit MutexLockMaybe(Mutex *mu) :
                                 ^
In file included from ../src/google/protobuf/arena.h:48:0,
                 from ../src/google/protobuf/message.h:118,
                 from ../src/google/protobuf/dynamic_message.h:45,
                 from google/protobuf/pyext/repeated_composite_container.cc:41:
/usr/include/c++/4.9/typeinfo:39:37: error: expected ‘}’ before end of line
 #pragma GCC visibility push(default)
                                     ^
/usr/include/c++/4.9/typeinfo:39:37: error: expected unqualified-id before end of line
/usr/include/c++/4.9/typeinfo:39:37: error: expected ‘}’ before end of line
/usr/include/c++/4.9/typeinfo:39:37: error: expected ‘}’ before end of line
/usr/include/c++/4.9/typeinfo:39:37: error: expected ‘}’ before end of line
/usr/include/c++/4.9/typeinfo:39:37: error: expected declaration before end of line
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

1. Test build OK python setup.py test --cpp_implementation

Ran 718 tests in 1.314s

OK (skipped=9)

1. Install protobuf in my application

source /myapp/.venv/bin/activate
python setup.py install --cpp_implementation

1. Reload my app

service uwsgi reload

What did you expect to see

Tue Sep  4 16:17:38 2018 - ...gracefully killing workers...
Tue Sep  4 16:17:38 2018 - Gracefully killing worker 1 (pid: 18114)...
Tue Sep  4 16:17:39 2018 - worker 1 buried after 1 seconds
Tue Sep  4 16:17:39 2018 - binary reloading uWSGI...

What did you see instead?

Tue Sep  4 16:17:38 2018 - ...gracefully killing workers...
Tue Sep  4 16:17:38 2018 - Gracefully killing worker 1 (pid: 18114)...
Tue Sep  4 16:17:38 2018 - !!! uWSGI process 18114 got Segmentation Fault !!!
Tue Sep  4 16:17:38 2018 - *** backtrace of 18114 ***
/usr/bin/uwsgi(uwsgi_backtrace+0x30) [0x4635f0]
/usr/bin/uwsgi(uwsgi_segfault+0x21) [0x4639b1]
/lib/x86_64-linux-gnu/libc.so.6(+0x350e0) [0x7fa557d170e0]
/myapp/.venv/local/lib/python2.7/site-packages/protobuf-3.6.1-py2.7-linux-x86_64.egg/google/protobuf/pyext/_message.so(_ZNKSt10_HashtableIPKN6google8protobuf14DescriptorPoolESt4pairIKS4_PNS1_6python16PyDescriptorPoolEESaISA_ENSt8__detail10_Select1stESt8equal_toIS4_ENS1_4hashIS4_EENSC_18_Mod_range_hashingENSC_20_Default_ranged_hashENSC_20_Prime_rehash_policyENSC_17_Hashtable_traitsILb1ELb0ELb1EEEE19_M_find_before_nodeEmRS6_m+0x13) [0x7fa5534701b3]
/myapp/.venv/local/lib/python2.7/site-packages/protobuf-3.6.1-py2.7-linux-x86_64.egg/google/protobuf/pyext/_message.so(+0x18317) [0x7fa55346f317]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xfcb9b) [0x7fa5567cab9b]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xbb08e) [0x7fa55678908e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xf9747) [0x7fa5567c7747]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyDict_SetItem+0x67) [0x7fa5567cb977]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyModule_Clear+0xb6) [0x7fa55682e0a6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_Cleanup+0x41f) [0x7fa556873e9f]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(Py_Finalize+0xfe) [0x7fa55685de4e]
/usr/bin/uwsgi(uwsgi_plugins_atexit+0x71) [0x4607c1]
/lib/x86_64-linux-gnu/libc.so.6(+0x37b29) [0x7fa557d19b29]
/lib/x86_64-linux-gnu/libc.so.6(+0x37b75) [0x7fa557d19b75]
/usr/bin/uwsgi() [0x419dff]
/usr/bin/uwsgi(gracefully_kill+0xbe) [0x46242e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890) [0x7fa559c9b890]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x33) [0x7fa557dcac03]
/usr/bin/uwsgi(event_queue_wait+0x33) [0x456ed3]
/usr/bin/uwsgi(wsgi_req_accept+0xd2) [0x417602]
/usr/bin/uwsgi(simple_loop_run+0xb6) [0x45f7a6]
/usr/bin/uwsgi(uwsgi_ignition+0x208) [0x463c78]
/usr/bin/uwsgi(uwsgi_worker_run+0x25d) [0x466fbd]
/usr/bin/uwsgi(uwsgi_run+0x3ae) [0x4674de]
/usr/bin/uwsgi(_start+0) [0x41699e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fa557d03b45]
/usr/bin/uwsgi() [0x4169c7]
*** end of backtrace ***
Tue Sep  4 16:17:39 2018 - worker 1 buried after 1 seconds
Tue Sep  4 16:17:39 2018 - binary reloading uWSGI...

Anything else we should know about your project / environment

Before today, I was using protobuf v3.3.2 and because I had this issue and my colleague saw this issue, we decided to try protobuf v3.6. And now I don't have any problem with cpu 100% as all the workers segfault :3 The application itself is still working. Thank you for reading.

[TeBoring]

Could you help send a pull request to fix this issue?
On Tue, Sep 4, 2018 at 08:33 Vincent Passama [e-mail]
wrote:

What version of protobuf and what language are you using?
Version: v3.6.1
Language: Python (C++ implementation)

What operating system (Linux, Windows, ...) and version?
Debian 8

* Extra info about env *
uwsgi 2.0.7-debian
nginx/1.10.3

What runtime / compiler are you using (e.g., python version or gcc
version)
gcc (Debian 4.9.2-10+deb8u1) 4.9.2
Python v2.7.9

What did you do?
Steps to reproduce the behavior:

1. Get protobuf from github then use the tag 3.6.1
2. Compilation (OK + tests OK)
3. Installation OK
4. Build for python with cpp implementation python setup.py build
   --cpp_implementation
   In this step I had to add 1 line in setup.py in order to make it work:

extra_compile_args.append('-std=c++11')

Otherwise I get this error:

building 'google.protobuf.pyext._message' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/google
creating build/temp.linux-x86_64-2.7/google/protobuf
creating build/temp.linux-x86_64-2.7/google/protobuf/pyext
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I. -I../src -I/usr/include/python2.7 -c google/protobuf/pyext/repeated_composite_container.cc -o build/temp.linux-x86_64-2.7/google/protobuf/pyext/repeated_composite_container.o -Wno-write-strings -Wno-invalid-offsetof -Wno-sign-compare
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /usr/include/c++/4.9/mutex:35:0,
from ../src/google/protobuf/stubs/mutex.h:33,
from ../src/google/protobuf/stubs/common.h:52,
from ./google/protobuf/pyext/message.h:42,
from ./google/protobuf/pyext/repeated_composite_container.h:43,
from google/protobuf/pyext/repeated_composite_container.cc:34:
/usr/include/c++/4.9/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support for the \
^
In file included from ./google/protobuf/pyext/message.h:43:0,
from ./google/protobuf/pyext/repeated_composite_container.h:43,
from google/protobuf/pyext/repeated_composite_container.cc:34:
./google/protobuf/pyext/thread_unsafe_shared_ptr.h:51:7: warning: identifier ‘nullptr’ is a keyword in C++11 [-Wc++0x-compat]
: ptr_(ptr), refcount_(ptr ? new RefcountT(1) : nullptr) {
^
In file included from ../src/google/protobuf/arena.h:51:0,
from ../src/google/protobuf/message.h:118,
from ../src/google/protobuf/dynamic_message.h:45,
from google/protobuf/pyext/repeated_composite_container.cc:41:
../src/google/protobuf/arena_impl.h:311:3: warning: identifier ‘static_assert’ is a keyword in C++11 [-Wc++0x-compat]
static_assert(kBlockHeaderSize % 8 == 0,
^
In file included from ../src/google/protobuf/stubs/common.h:52:0,
from ./google/protobuf/pyext/message.h:42,
from ./google/protobuf/pyext/repeated_composite_container.h:43,
from google/protobuf/pyext/repeated_composite_container.cc:34:
../src/google/protobuf/stubs/mutex.h:50:20: warning: defaulted and deleted functions only available with -std=c++11 or -std=gnu++11
WrappedMutex() = default;
^
../src/google/protobuf/stubs/mutex.h:58:8: error: ‘mutex’ in namespace ‘std’ does not name a type
std::mutex mu_;
^
../src/google/protobuf/stubs/mutex.h: In member function ‘void google::protobuf::internal::WrappedMutex::Lock()’:
../src/google/protobuf/stubs/mutex.h:51:17: error: ‘mu_’ was not declared in this scope
void Lock() { mu_.lock(); }
^
../src/google/protobuf/stubs/mutex.h: In member function ‘void google::protobuf::internal::WrappedMutex::Unlock()’:
../src/google/protobuf/stubs/mutex.h:52:19: error: ‘mu_’ was not declared in this scope
void Unlock() { mu_.unlock(); }
^
../src/google/protobuf/stubs/mutex.h: At global scope:
../src/google/protobuf/stubs/mutex.h:61:7: error: expected nested-name-specifier before ‘Mutex’
using Mutex = WrappedMutex;
^
../src/google/protobuf/stubs/mutex.h:66:28: error: expected ‘)’ before ‘’ token
explicit MutexLock(Mutex *mu) : mu_(mu) { this->mu_->Lock(); }
^
../src/google/protobuf/stubs/mutex.h:69:3: error: ‘Mutex’ does not name a type
Mutex *const mu_;
^
../src/google/protobuf/stubs/mutex.h: In destructor ‘google::protobuf::internal::MutexLock::~MutexLock()’:
../src/google/protobuf/stubs/mutex.h:67:24: error: ‘class google::protobuf::internal::MutexLock’ has no member named ‘mu_’
~MutexLock() { this->mu_->Unlock(); }
^
../src/google/protobuf/stubs/mutex.h: At global scope:
../src/google/protobuf/stubs/mutex.h:80:33: error: expected ‘)’ before ‘’ token
explicit MutexLockMaybe(Mutex *mu) :
^
In file included from ../src/google/protobuf/arena.h:48:0,
from ../src/google/protobuf/message.h:118,
from ../src/google/protobuf/dynamic_message.h:45,
from google/protobuf/pyext/repeated_composite_container.cc:41:
/usr/include/c++/4.9/typeinfo:39:37: error: expected ‘}’ before end of line
#pragma GCC visibility push(default)
^
/usr/include/c++/4.9/typeinfo:39:37: error: expected unqualified-id before end of line
/usr/include/c++/4.9/typeinfo:39:37: error: expected ‘}’ before end of line
/usr/include/c++/4.9/typeinfo:39:37: error: expected ‘}’ before end of line
/usr/include/c++/4.9/typeinfo:39:37: error: expected ‘}’ before end of line
/usr/include/c++/4.9/typeinfo:39:37: error: expected declaration before end of line
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

1. Test build OK python setup.py test --cpp_implementation

Ran 718 tests in 1.314s

OK (skipped=9)

1. Install protobuf in my application

source /myapp/.venv/bin/activate
python setup.py install --cpp_implementation

1. Reload my app

service uwsgi reload

What did you expect to see

Tue Sep 4 16:17:38 2018 - ...gracefully killing workers...
Tue Sep 4 16:17:38 2018 - Gracefully killing worker 1 (pid: 18114)...
Tue Sep 4 16:17:39 2018 - worker 1 buried after 1 seconds
Tue Sep 4 16:17:39 2018 - binary reloading uWSGI...

What did you see instead?

Tue Sep 4 16:17:38 2018 - ...gracefully killing workers...
Tue Sep 4 16:17:38 2018 - Gracefully killing worker 1 (pid: 18114)...
Tue Sep 4 16:17:38 2018 - !!! uWSGI process 18114 got Segmentation Fault !!!
Tue Sep 4 16:17:38 2018 - * backtrace of 18114
/usr/bin/uwsgi(uwsgi_backtrace+0x30) [0x4635f0]
/usr/bin/uwsgi(uwsgi_segfault+0x21) [0x4639b1]
/lib/x86_64-linux-gnu/libc.so.6(+0x350e0) [0x7fa557d170e0]
/myapp/.venv/local/lib/python2.7/site-packages/protobuf-3.6.1-py2.7-linux-x86_64.egg/google/protobuf/pyext/_message.so(_ZNKSt10_HashtableIPKN6google8protobuf14DescriptorPoolESt4pairIKS4_PNS1_6python16PyDescriptorPoolEESaISA_ENSt8__detail10_Select1stESt8equal_toIS4_ENS1_4hashIS4_EENSC_18_Mod_range_hashingENSC_20_Default_ranged_hashENSC_20_Prime_rehash_policyENSC_17_Hashtable_traitsILb1ELb0ELb1EEEE19_M_find_before_nodeEmRS6_m+0x13) [0x7fa5534701b3]
/myapp/.venv/local/lib/python2.7/site-packages/protobuf-3.6.1-py2.7-linux-x86_64.egg/google/protobuf/pyext/_message.so(+0x18317) [0x7fa55346f317]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xfcb9b) [0x7fa5567cab9b]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xbb08e) [0x7fa55678908e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xf9747) [0x7fa5567c7747]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyDict_SetItem+0x67) [0x7fa5567cb977]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyModule_Clear+0xb6) [0x7fa55682e0a6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyImport_Cleanup+0x41f) [0x7fa556873e9f]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(Py_Finalize+0xfe) [0x7fa55685de4e]
/usr/bin/uwsgi(uwsgi_plugins_atexit+0x71) [0x4607c1]
/lib/x86_64-linux-gnu/libc.so.6(+0x37b29) [0x7fa557d19b29]
/lib/x86_64-linux-gnu/libc.so.6(+0x37b75) [0x7fa557d19b75]
/usr/bin/uwsgi() [0x419dff]
/usr/bin/uwsgi(gracefully_kill+0xbe) [0x46242e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890) [0x7fa559c9b890]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x33) [0x7fa557dcac03]
/usr/bin/uwsgi(event_queue_wait+0x33) [0x456ed3]
/usr/bin/uwsgi(wsgi_req_accept+0xd2) [0x417602]
/usr/bin/uwsgi(simple_loop_run+0xb6) [0x45f7a6]
/usr/bin/uwsgi(uwsgi_ignition+0x208) [0x463c78]
/usr/bin/uwsgi(uwsgi_worker_run+0x25d) [0x466fbd]
/usr/bin/uwsgi(uwsgi_run+0x3ae) [0x4674de]
/usr/bin/uwsgi(_start+0) [0x41699e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fa557d03b45]
/usr/bin/uwsgi() [0x4169c7]
end of backtrace *
Tue Sep 4 16:17:39 2018 - worker 1 buried after 1 seconds
Tue Sep 4 16:17:39 2018 - binary reloading uWSGI...

Anything else we should know about your project / environment

Before today, I was using protobuf v3.3.2 and because I had this issue
https://github.com/unbit/uwsgi/issues/1801 and my colleague saw this
issue https://github.com/protocolbuffers/protobuf/issues/3607, we
decided to try protobuf v3.6. And now I don't have any problem with cpu
100% as all the workers segfault :3 The application itself is still
working. Thank you for reading.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/protocolbuffers/protobuf/issues/5108, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AE9H5d9GVfmhI9vK6HhaYC5psqxQx-5rks5uXp0zgaJpZM4WZO3G
.

[vpassama]

Hi, I'm not sure I can for now as I don't know anything about what could be the cause of the segfault.

[TeBoring]

In your first post, did you say you added one line and make it work?
On Tue, Sep 4, 2018 at 08:40 Vincent Passama [e-mail]
wrote:

Hi, I'm not sure I can for now as I don't know anything about what could
be the cause of the segfault.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/protocolbuffers/protobuf/issues/5108#issuecomment-418415909,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5bhE5KFArB-XcreUhgZJuM4bUdEyks5uXp7igaJpZM4WZO3G
.

[vpassama]

Ho, I was speaking about the build step (not the segfault described at the end of the issue) I thought it could be helpful to add this details. But I can make another issue/PR about this if you want ?

[TeBoring]

Yeah, that is needed. Thanks.
What’s the version of python you are using?
On Tue, Sep 4, 2018 at 08:53 Vincent Passama [e-mail]
wrote:

Ho, I was speaking about the build step (not the segfault described at the
end of the issue) I thought it could be helpful to add this details. But I
can make another issue/PR about this if you want ?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/protocolbuffers/protobuf/issues/5108#issuecomment-418420707,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5cLG53QkeqMcsBzkj3TlZZqwTGtsks5uXqIAgaJpZM4WZO3G
.

[vpassama]

Python v2.7.9

About the line : extra_compile_args.append('-std=c++11') I didn't check anything about how to add this without breaking something for other environments. So it is safer that I just make an issue and not a PR.

[TeBoring]

Could you help do more debug? Right now, the error log only shows seg fault
is in protobuf. And we need to understand your code in order to debug.
Could you simplify the step to debug?
On Tue, Sep 4, 2018 at 09:11 Vincent Passama [e-mail]
wrote:

Python v2.7.9

About the line : extra_compile_args.append('-std=c++11') I didn't check
anything about how to add this without breaking something for other
environments. So it is safer that I just make an issue and not a PR.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/protocolbuffers/protobuf/issues/5108#issuecomment-418427149,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5eIHTAjYz9JxrvCeaZ80qk6GFAD0ks5uXqYegaJpZM4WZO3G
.

[vpassama]

Ok. I tried to get some details with gdb and it seems the segfault happens when a protobuf object is deleted. The segfault is random by the way (like it was before when I had 100% cpu). It may be related to uwsgi. I tried to reproduce the bug by isolating the code, and the segfault happens only if I import some auto generated protobuf files (it seems normal to me as it is a problem of dealloc on protobuf objects).

For now I give you the gdb backtrace :

Program received signal SIGSEGV, Segmentation fault.
std::_Hashtable<void const*, std::pair<void const* const, _object*>, std::allocator<std::pair<void const* const, _object*> >, std::__detail::_Select1st, std::equal_to<void const*>, google::protobuf::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node (
    this=this@entry=0x7f640f791c00 <google::protobuf::python::interned_descriptors>, __n=__n@entry=198, 
    __k=@0x7f6410b1edc0: 0x1df6640, __code=31417920) at /usr/include/c++/4.9/bits/hashtable.h:1442
1442          for (__node_type* __p = static_cast<__node_type*>(__prev_p->_M_nxt);;
(gdb) bt
#0  std::_Hashtable<void const*, std::pair<void const* const, _object*>, std::allocator<std::pair<void const* const, _object*> >, std::__detail::_Select1st, std::equal_to<void const*>, google::protobuf::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node (
    this=this@entry=0x7f640f791c00 <google::protobuf::python::interned_descriptors>, __n=__n@entry=198, 
    __k=@0x7f6410b1edc0: 0x1df6640, __code=31417920) at /usr/include/c++/4.9/bits/hashtable.h:1442
#1  0x00007f640f5694b6 in _M_erase (this=0x7f640f791c00 <google::protobuf::python::interned_descriptors>, 
    __k=@0x7f6410b1edc0: 0x1df6640) at /usr/include/c++/4.9/bits/hashtable.h:1817
#2  erase (__k=@0x7f6410b1edc0: 0x1df6640, this=0x7f640f791c00 <google::protobuf::python::interned_descriptors>)
    at /usr/include/c++/4.9/bits/hashtable.h:741
#3  erase (__x=@0x7f6410b1edc0: 0x1df6640, this=0x7f640f791c00 <google::protobuf::python::interned_descriptors>)
    at /usr/include/c++/4.9/bits/unordered_map.h:500
#4  google::protobuf::python::descriptor::Dealloc (self=0x7f6410b1edb0) at google/protobuf/pyext/descriptor.cc:373
#5  0x00007f6411b1bb9b in dict_dealloc.lto_priv.375 (mp=0x7f640fed4a28) at ../Objects/dictobject.c:1010
#6  0x00007f6411ada08e in subtype_dealloc (self=0x7f640e6a8510) at ../Objects/typeobject.c:1015
#7  0x00007f6411b18747 in insertdict_by_entry (value=0x7f6411f89390 <_Py_NoneStruct>, ep=0x23fa3b0, hash=5226084808037340962, 
    key=0x7f640f099c30, mp=0x7f64113306e0) at ../Objects/dictobject.c:519
#8  insertdict (mp=0x7f64113306e0, key=0x7f640f099c30, hash=5226084808037340962, value=0x7f6411f89390 <_Py_NoneStruct>)
    at ../Objects/dictobject.c:556
#9  0x00007f6411b1c977 in dict_set_item_by_hash_or_entry (ep=0x0, value=0x7f6411f89390 <_Py_NoneStruct>, hash=<optimized out>, 
    key=<optimized out>, op=0x7f64113306e0) at ../Objects/dictobject.c:765
#10 PyDict_SetItem (op=op@entry=0x7f64113306e0, key=<optimized out>, value=value@entry=0x7f6411f89390 <_Py_NoneStruct>)
    at ../Objects/dictobject.c:818
#11 0x00007f6411b7f15c in _PyModule_Clear (m=<optimized out>) at ../Objects/moduleobject.c:139
#12 0x00007f6411bc4e9f in PyImport_Cleanup () at ../Python/import.c:526
#13 0x00007f6411baee4e in Py_Finalize () at ../Python/pythonrun.c:459
#14 0x00000000004607c1 in uwsgi_plugins_atexit ()
#15 0x00007f641306ab29 in __run_exit_handlers (status=30, listp=0x7f64133d85a8 <__exit_funcs>, 
    run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#16 0x00007f641306ab75 in __GI_exit (status=<optimized out>) at exit.c:104
#17 0x0000000000419dff in uwsgi_exit ()
#18 0x0000000000460815 in end_me ()
#19 <signal handler called>
#20 0x00007f641311bc03 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#21 0x0000000000456ed3 in event_queue_wait ()
#22 0x0000000000417602 in wsgi_req_accept ()
#23 0x000000000045f7a6 in simple_loop_run ()
#24 0x0000000000463c78 in uwsgi_ignition ()
#25 0x0000000000466fbd in uwsgi_worker_run ()
#26 0x00000000004674de in uwsgi_run ()
#27 0x000000000041699e in main ()

I will give a try to reproduce this with some autogenerated protobuf (as the original code is not mine) in order to give you some example code.

[vpassama]

I reproduced the bug with an example; I hope it will help.

https://github.com/vpassama/protobuf_issue_5108

[TeBoring]

Are you able to simplify it without using uwsgi?
On Mon, Sep 10, 2018 at 15:06 Jie Luo [e-mail] wrote:

Assigned #5108 https://github.com/protocolbuffers/protobuf/issues/5108
to @TeBoring https://github.com/TeBoring.

—
You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub
https://github.com/protocolbuffers/protobuf/issues/5108#event-1837019253,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5cDltnXVHWbnP2zT-3cfnJaBZZOQks5uZs9zgaJpZM4WZO3G
.

[vpassama]

Hi, I think the bug is related to the use of uwsgi + protobuf, I don't know how to reproduce it outside of it.

[knaoe]

It's the same with me.

Info about my env:

• protobuf:3.6.1
• uWSGI: 2.0.17
• Python: 2.7.15
• Debian: 9.4
• Nginx: 1.13.12

Some time I got segmentation faults like following:

!!! uWSGI process 42 got Segmentation Fault !!!
*** backtrace of 42 ***
/usr/local/bin/uwsgi(uwsgi_backtrace+0x35) [0x555ec6e86025]
/usr/local/bin/uwsgi(uwsgi_segfault+0x23) [0x555ec6e863d3]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060) [0x7f9022a3a060]
/usr/local/lib/python2.7/site-packages/google/protobuf/pyext/_message.so(+0xa5b1a) [0x7f9016a86b1a]
/usr/local/lib/libpython2.7.so.1.0(+0x9c177) [0x7f902307a177]
/usr/local/lib/libpython2.7.so.1.0(PyDict_SetItem+0x67) [0x7f902307bc17]
/usr/local/lib/libpython2.7.so.1.0(_PyModule_Clear+0x14c) [0x7f902307ff7c]
/usr/local/lib/libpython2.7.so.1.0(PyImport_Cleanup+0x40f) [0x7f9023101aef]
/usr/local/lib/libpython2.7.so.1.0(Py_Finalize+0xfe) [0x7f902311381e]
/usr/local/bin/uwsgi(uwsgi_plugins_atexit+0x81) [0x555ec6e83091]
/lib/x86_64-linux-gnu/libc.so.6(+0x35940) [0x7f9022a3c940]
/lib/x86_64-linux-gnu/libc.so.6(+0x3599a) [0x7f9022a3c99a]
/usr/local/bin/uwsgi(+0x3f1af) [0x555ec6e3b1af]
/usr/local/bin/uwsgi(end_me+0x25) [0x555ec6e830d5]
/usr/local/bin/uwsgi(uwsgi_ignition+0x147) [0x555ec6e865d7]
/usr/local/bin/uwsgi(uwsgi_worker_run+0x28d) [0x555ec6e8b00d]
/usr/local/bin/uwsgi(+0x8f60c) [0x555ec6e8b60c]
/usr/local/bin/uwsgi(+0x3bcce) [0x555ec6e37cce]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f9022a272e1]
/usr/local/bin/uwsgi(_start+0x2a) [0x555ec6e37cfa]
*** end of backtrace ***
worker 1 killed successfully (pid: 42)

[manuel-koch]

This simple snippet trigger the same crash on Python 3.7

Python 3.7.1 (default, Oct 24 2018, 22:38:59)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import google.protobuf
>>> google.protobuf.__version__
'3.6.1'
>>> import google.protobuf.pyext._message
__main__:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
>>> google.protobuf.pyext._message.Message._extensions_by_name
Segmentation fault

I already reported it here for a similar issue https://github.com/protocolbuffers/protobuf/issues/2974#issuecomment-399078600

[vpassama]

@manuel-koch Are you sure this is the same crash ? Because in my case, the segmentation fault occurs on reloading/restarting not on imports.

[manuel-koch]

Looking at #2974 I think it is the same problem.

[manuel-koch]

my snippet crashes when accessing a symbol/property that looks very similar to the callstack of uwsgi.

[manuel-koch]

I have a similar problem in my docker image that is running uwsgi/python :

*** Starting uWSGI 2.0.17.1 (64bit) on [Thu Nov 22 11:12:51 2018] ***
compiled with version: 6.3.0 20170516 on 13 November 2018 15:23:09
os: Linux-4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018
...
Python version: 3.7.1 (default, Oct 24 2018, 22:38:59)  [GCC 6.3.0 20170516]
...

I.e. _not_ installing/using python packages that include "protobuf" makes my uwsgi instance run fine without problems.

Once I have "protobuf" in my docker image after an install of the following python package
via e.g. pip install google-cloud-storage ( google-cloud-storage==1.13.0 )

Successfully installed cachetools-3.0.0 google-api-core-1.5.2 google-auth-1.6.1 google-cloud-core-0.28.1 google-cloud-storage-1.13.0 google-resumable-media-0.3.1 googleapis-common-protos-1.5.5 protobuf-3.6.1 pyasn1-modules-0.2.2 rsa-4.0

and importing it in my code like

import boto3
from google.cloud import storage as google_storage
from google.oauth2 import service_account as google_service_account

my uwsgi/python fails to startup and constantly crashes with segfault !?

*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x55feccb090b0 pid: 49 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 49)
spawned uWSGI worker 1 (pid: 64, cores: 1)
Python auto-reloader enabled
spawned uWSGI worker 2 (pid: 66, cores: 1)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
running "unix_signal:3 gracefully_kill_them_all" (master-start)...
running "unix_signal:5 gracefully_kill_them_all" (master-start)...
running "unix_signal:15 gracefully_kill_them_all" (master-start)...
!!! uWSGI process 66 got Segmentation Fault !!!
!!! uWSGI process 64 got Segmentation Fault !!!
*** backtrace of 66 ***
uwsgi(uwsgi_backtrace+0x35) [0x55fecaa0cca5]
uwsgi(uwsgi_segfault+0x23) [0x55fecaa0d053]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060) [0x7f8ae1b06060]
/usr/local/lib/libpython3.7m.so.1.0(PyBytes_AsString+0) [0x7f8ae213dbb0]
uwsgi(uwsgi_python_autoreloader_thread+0x166) [0x55fecaa24416]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f8ae3e63494]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f8ae1bbbacf]
*** end of backtrace ***
*** backtrace of 64 ***
uwsgi(uwsgi_backtrace+0x35) [0x55fecaa0cca5]
uwsgi(uwsgi_segfault+0x23) [0x55fecaa0d053]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060) [0x7f8ae1b06060]
/usr/local/lib/libpython3.7m.so.1.0(PyBytes_AsString+0) [0x7f8ae213dbb0]
uwsgi(uwsgi_python_autoreloader_thread+0x166) [0x55fecaa24416]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f8ae3e63494]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f8ae1bbbacf]
*** end of backtrace ***
DAMN ! worker 1 (pid: 64) died, killed by signal 11 :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 68)
DAMN ! worker 2 (pid: 66) died, killed by signal 11 :( trying respawn ...
Python auto-reloader enabled
Respawned uWSGI worker 2 (new pid: 70)
!!! uWSGI process 68 got Segmentation Fault !!!
...

[manuel-koch]

I downgraded my docker image to use python-3.6 and forced protobuf==3.6.0
and the repeated crash on startup disappeared !?

*** Starting uWSGI 2.0.17.1 (64bit) on [Fri Nov 23 13:33:45 2018] ***
compiled with version: 6.3.0 20170516 on 23 November 2018 12:35:17
os: Linux-4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018
Python version: 3.6.7 (default, Nov 16 2018, 06:39:52)  [GCC 6.3.0 20170516]

But it still segfaults when uwsgi workers are recycled after selected number of requests

!!! uWSGI process 41 got Segmentation Fault !!!
*** backtrace of 41 ***
uwsgi(uwsgi_backtrace+0x35) [0x5605c2b776d5]
uwsgi(uwsgi_segfault+0x23) [0x5605c2b77a83]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060) [0x7f60a2198060]
/usr/local/lib/python3.6/site-packages/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so(+0xb6a4d) [0x7f60976d1a4d]
/usr/local/lib/libpython3.6m.so.1.0(+0xc6517) [0x7f60a2802517]
/usr/local/lib/libpython3.6m.so.1.0(+0xec741) [0x7f60a2828741]
/usr/local/lib/libpython3.6m.so.1.0(+0xc5697) [0x7f60a2801697]
/usr/local/lib/libpython3.6m.so.1.0(+0xcb819) [0x7f60a2807819]
/usr/local/lib/libpython3.6m.so.1.0(+0x1bd212) [0x7f60a28f9212]
/usr/local/lib/libpython3.6m.so.1.0(_PyGC_CollectNoFail+0x31) [0x7f60a28f9f01]
/usr/local/lib/libpython3.6m.so.1.0(PyImport_Cleanup+0x1b8) [0x7f60a28c71f8]
/usr/local/lib/libpython3.6m.so.1.0(Py_FinalizeEx+0x6b) [0x7f60a28d62eb]
uwsgi(uwsgi_plugins_atexit+0x81) [0x5605c2b747f1]
/lib/x86_64-linux-gnu/libc.so.6(+0x35940) [0x7f60a219a940]
/lib/x86_64-linux-gnu/libc.so.6(+0x3599a) [0x7f60a219a99a]
uwsgi(+0x3ef6f) [0x5605c2b2cf6f]
uwsgi(end_me+0x25) [0x5605c2b74835]
uwsgi(uwsgi_ignition+0x14a) [0x5605c2b77c8a]
uwsgi(uwsgi_worker_run+0x275) [0x5605c2b7c615]
uwsgi(+0x8ec0c) [0x5605c2b7cc0c]
uwsgi(+0x3bbde) [0x5605c2b29bde]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f60a21852e1]
uwsgi(_start+0x2a) [0x5605c2b29c0a]
*** end of backtrace ***

[petee-d]

Had the same issue with protobuf==3.6.0 with CPP implementation, uWSGI==2.0.13.1 (also tried uWSGI==2.0.18, same) and python 2.7.15. The backtrace after the segfault was basically the same as above.

For anyone trying to get rid of this problem somehow, one option is not using the CPP extension and using native Python implementation, this can be done by running the Python process with environment variablePROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
https://developers.google.com/protocol-buffers/docs/reference/python-generated#cpp_impl

[vpassama]

fixed by release 3.7.0