Caffe: Segmentation fault in using python layer in caffe

[prateek-kacker]

Hi,

I am unable to use Python as a layer in caffe. I am receiving a segmentation fault. It occurs only when I try use a python layer in my caffe prototxt file.
However I checked that python and pycaffe installation by running importing caffe in python and there is no error in that.

My system configuration is:-
Operating system: Ubuntu 16.04.2 LTS
Compiler: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)
CUDA version (if applicable): Cuda compilation tools, release 8.0, V8.0.61
Python :2.7 Version

The error is as follows:

I0304 15:56:22.749100 25343 layer_factory.hpp:77] Creating layer data
* Aborted at 1488642983 (unix time) try "date -d @1488642983" if you are using GNU date
PC: @ 0x7f0784fb4873 std::_Hashtable<>::clear()
SIGSEGV (@0x9) received by PID 25343 (TID 0x7f07c0e80ac0) from PID 9; stack trace: *
@ 0x7f07be2fe4b0 (unknown)
@ 0x7f0784fb4873 std::_Hashtable<>::clear()
@ 0x7f0784fa6346 google::protobuf::DescriptorPool::FindFileByName()
@ 0x7f0784f84ac8 google::protobuf::python::cdescriptor_pool::AddSerializedFile()
@ 0x7f07be9677d0 PyEval_EvalFrameEx
@ 0x7f07bea9001c PyEval_EvalCodeEx
@ 0x7f07be9e63dd (unknown)
@ 0x7f07be9b91e3 PyObject_Call
@ 0x7f07be9d9ae5 (unknown)
@ 0x7f07be970123 (unknown)
@ 0x7f07be9b91e3 PyObject_Call
@ 0x7f07be96413c PyEval_EvalFrameEx
@ 0x7f07bea9001c PyEval_EvalCodeEx
@ 0x7f07be95eb89 PyEval_EvalCode
@ 0x7f07be9f31b4 PyImport_ExecCodeModuleEx
@ 0x7f07be9f3b8f (unknown)
@ 0x7f07be9f5300 (unknown)
@ 0x7f07be9f55c8 (unknown)
@ 0x7f07be9f66db PyImport_ImportModuleLevel
@ 0x7f07be96d698 (unknown)
@ 0x7f07be9b91e3 PyObject_Call
@ 0x7f07bea8f447 PyEval_CallObjectWithKeywords
@ 0x7f07be9625c6 PyEval_EvalFrameEx
@ 0x7f07bea9001c PyEval_EvalCodeEx
@ 0x7f07be95eb89 PyEval_EvalCode
@ 0x7f07be9f31b4 PyImport_ExecCodeModuleEx
@ 0x7f07be9f3b8f (unknown)
@ 0x7f07be9f5300 (unknown)
@ 0x7f07be9f55c8 (unknown)
@ 0x7f07be9f66db PyImport_ImportModuleLevel
@ 0x7f07be96d698 (unknown)
@ 0x7f07be9b91e3 PyObject_Call
./run_lstm_HAR_RGB.sh: line 6: 25343 Segmentation fault (core dumped) GLOG_logtostderr=1 ~/caffe/build/tools/caffe train -solver lstm_solver_HAR_RGB.prototxt -weights single_frame_all_layers_hyb_RGB_iter_5000.caffemodel -gpu 0
Done.

Please help..

[Fang-Haoshu]

I meet exactly the same error. Any help?

[nbarriga]

I ran into this same problem today. After hours of recompilations and debugging, downgrading protobuf python package fixed it for me:
pip install --user --upgrade protobuf==3.1.0.post1

[piaozhx]

@nbarriga very nice! it worked!

[gxstudy]

Hi, @nbarriga ,
I got the same error, but there isn't any segmentation fault error message afterward:
I0324 21:41:09.567148 8061 layer_factory.hpp:77] Creating layer relu_sqrt_1
* Aborted at 1490406070 (unix time) try "date -d @1490406070" if you are using GNU date
PC: @ 0x7f05e9a70873 std::_Hashtable<>::clear()
SIGSEGV (@0x9) received by PID 8061 (TID 0x7f061bef5ac0) from PID 9; stack trace: *
@ 0x7f06191474b0 (unknown)
@ 0x7f05e9a70873 std::_Hashtable<>::clear()
@ 0x7f05e9a62346 google::protobuf::DescriptorPool::FindFileByName()
@ 0x7f05e9a40ac8 google::protobuf::python::cdescriptor_pool::AddSerializedFile()
@ 0x7f06197b07d0 PyEval_EvalFrameEx
@ 0x7f06198d901c PyEval_EvalCodeEx
@ 0x7f061982f3dd (unknown)
@ 0x7f06198021e3 PyObject_Call
@ 0x7f0619822ae5 (unknown)
@ 0x7f06197b9123 (unknown)
@ 0x7f06198021e3 PyObject_Call
@ 0x7f06197ad13c PyEval_EvalFrameEx
@ 0x7f06198d901c PyEval_EvalCodeEx
@ 0x7f06197a7b89 PyEval_EvalCode
@ 0x7f061983c1b4 PyImport_ExecCodeModuleEx
@ 0x7f061983cb8f (unknown)
@ 0x7f061983e300 (unknown)
@ 0x7f061983e5c8 (unknown)
@ 0x7f061983f6db PyImport_ImportModuleLevel
@ 0x7f06197b6698 (unknown)
@ 0x7f06198021e3 PyObject_Call
@ 0x7f06198d8447 PyEval_CallObjectWithKeywords
@ 0x7f06197ab5c6 PyEval_EvalFrameEx
@ 0x7f06198d901c PyEval_EvalCodeEx
@ 0x7f06197a7b89 PyEval_EvalCode
@ 0x7f061983c1b4 PyImport_ExecCodeModuleEx
@ 0x7f061983cb8f (unknown)
@ 0x7f061983e300 (unknown)
@ 0x7f061983e5c8 (unknown)
@ 0x7f061983f6db PyImport_ImportModuleLevel
@ 0x7f06197b6698 (unknown)
@ 0x7f06198021e3 PyObject_Call
Done.

And simply run "pip install --user --upgrade protobuf==3.1.0.post1" didn't solve my problem, I found that after running that, when I check the protoc version using "protoc --version", it's still said it's libprotoc 3.2.0, could it be the reason? Do you have any suggestion on how to fix the error? Thank you so much for the help. I will really appreciate if you could give me some suggestion.

[nbarriga]

I also have protoc 3.2.0. Are you using python 2? If you are using python3, then you need to run pip3.

I can't think of anything else.

[gxstudy]

@nbarriga Thank you so much for your reply. I am using python 2. May I ask what's your previous version of protobuf before you use "pip install --user --upgrade protobuf==3.1.0.post1"? Is it the protobuf 3.2.0 you downloaded and built from https://github.com/google/protobuf? And what's the version of protobuf after you upgrade it? Is it still 3.2.0 or it says 3.1.0 now. Do you need to rebuild caffe after changing the protobuf version?

[nbarriga]

The protoc executable I have is 3.2.0, from https://github.com/google/protobuf

The protobuf python package I had was 3.2.0, and that is the one I downgraded using pip.
protoc version is still 3.2.0.

I believe I had to recompile after installing the protobuf executable (protoc) and library, but not after downgrading the python package.

[gxstudy]

Thank you so much! The problem has been solved using pip install --user --upgrade protobuf==3.1.0rc1.post1. I really really appreciated your help. Thank you very much.

[aznshodan]

I'm experiencing the same issue.
I tried both
pip install --user --upgrade protobuf==3.2.0rc1.post1
and
pip install --user --upgrade protobuf==3.1.0.post1

and it still doesn't work.

This is the error message I get.

*** Aborted at 1495447102 (unix time) try "date -d @1495447102" if you are using GNU date ***
PC: @     0x7fac885f7873 std::_Hashtable<>::clear()
*** SIGSEGV (@0x9) received by PID 26942 (TID 0x7fb0c08ff740) from PID 9; stack trace: ***
    @     0x7fb0bdae64b0 (unknown)
    @     0x7fac885f7873 std::_Hashtable<>::clear()
    @     0x7fac885e9346 google::protobuf::DescriptorPool::FindFileByName()
    @     0x7fac885c7ac8 google::protobuf::python::cdescriptor_pool::AddSerializedFile()
    @     0x7fb0be14f7d0 PyEval_EvalFrameEx
    @     0x7fb0be27801c PyEval_EvalCodeEx
    @     0x7fb0be1ce3dd (unknown)
    @     0x7fb0be1a11e3 PyObject_Call
    @     0x7fb0be1c1ae5 (unknown)
    @     0x7fb0be158123 (unknown)
    @     0x7fb0be1a11e3 PyObject_Call
    @     0x7fb0be14c13c PyEval_EvalFrameEx
    @     0x7fb0be27801c PyEval_EvalCodeEx
    @     0x7fb0be146b89 PyEval_EvalCode
    @     0x7fb0be1db1b4 PyImport_ExecCodeModuleEx
    @     0x7fb0be1dbb8f (unknown)
    @     0x7fb0be1dd300 (unknown)
    @     0x7fb0be1dd5c8 (unknown)
    @     0x7fb0be1de6db PyImport_ImportModuleLevel
    @     0x7fb0be155698 (unknown)
    @     0x7fb0be1a11e3 PyObject_Call
    @     0x7fb0be277447 PyEval_CallObjectWithKeywords
    @     0x7fb0be14a5c6 PyEval_EvalFrameEx
    @     0x7fb0be27801c PyEval_EvalCodeEx
    @     0x7fb0be146b89 PyEval_EvalCode
    @     0x7fb0be1db1b4 PyImport_ExecCodeModuleEx
    @     0x7fb0be1dbb8f (unknown)
    @     0x7fb0be1dd300 (unknown)
    @     0x7fb0be1dd5c8 (unknown)
    @     0x7fb0be1de6db PyImport_ImportModuleLevel
    @     0x7fb0be155698 (unknown)
    @     0x7fb0be1a11e3 PyObject_Call

[predictori]

Hi, @aznshodan,

I faced same error and I tried both
pip install --user --upgrade protobuf==3.2.0rc1.post1
and
pip install --user --upgrade protobuf==3.1.0.post1

and it not worked same.

In my case, executed following command, and it worked.

sudo -H pip install --upgrade protobuf==3.1.0.post1

[bagaabu]

This works for me!
sudo -H pip install --upgrade protobuf==3.1.0.post1
It uninstall protobuf 3.2.0 and reinstall 3.1.0, then it worked!
Thanks all you guys!

[bmiwcy]

@predictori
sudo -H pip install --upgrade protobuf==3.1.0.post1
its worked! thank you!

[ZeitgeberH]

system has "libprotobuf 3.0.0", but install python protobuf=3.0.0 doesn't work. Instead,
sudo -H pip install --upgrade protobuf==3.1.0.post1
works. Thank you! @predictori

[maingoh]

Got the same segfault but I need to use protobuf 3.4.0.
Does it mean caffe python layers will never support higher versions of protobuf ?

[wqvbjhc]

Got the same segfault but I need to use protobuf 3.4.0.
Does it mean caffe python layers will never support higher versions of protobuf ?

I got the segfault too, protobuf=3.6.0

[zhaokai5]

Got the same segfault but I need to use protobuf 3.4.0.
Does it mean caffe python layers will never support higher versions of protobuf ?

I got the segfault too, protobuf=3.6.0

Do you resolved this problem?
I encounter this issue too. My protoc --version shown that libprotoc 3.6.1,and pip install protobuf==3.6.1. It still have this issue. Any suggestion for me? Thanks!

[maingoh]

We had to rebuild the protobuf python package to use the current dynamic library on the system.
You have to clone the protobuf repository and install the package from the setup.py.
Something like this:

git clone https://github.com/protocolbuffers/protobuf.git
git checkout v3.6.1
cd protobuf/python
sed -i "s/libprotobuf.a/libprotobuf.so/g" setup.py
sed -i "s/libprotobuf-lite.a/libprotobuf-lite.so/g" setup.py
python3 setup.py install --cpp_implementation

Let me know if it works for you !

[zhaokai5]

We had to rebuild the protobuf python package to use the current dynamic library on the system.
You have to clone the protobuf repository and install the package from the setup.py.
Something like this:

git clone https://github.com/protocolbuffers/protobuf.git
git checkout v3.6.1
cd protobuf/python
sed -i "s/libprotobuf.a/libprotobuf.so/g" setup.py
sed -i "s/libprotobuf-lite.a/libprotobuf-lite.so/g" setup.py
python3 setup.py install --cpp_implementation

Let me know if it works for you !

Thanks! It worked now.
but after python3 setup.py install --cpp_implementation, I encounter the the issue of undefined symbol: _ZN6google8protobuf4util18MessageDifferencer6EqualsERKNS0_7MessageES5_,
I fixed the problem according to below links .https://github.com/szagoruyko/loadcaffe/issues/45
And now it worked well.Thanks!

[Time0o]

We had to rebuild the protobuf python package to use the current dynamic library on the system.
You have to clone the protobuf repository and install the package from the setup.py.
Something like this:

git clone https://github.com/protocolbuffers/protobuf.git
git checkout v3.6.1
cd protobuf/python
sed -i "s/libprotobuf.a/libprotobuf.so/g" setup.py
sed -i "s/libprotobuf-lite.a/libprotobuf-lite.so/g" setup.py
python3 setup.py install --cpp_implementation

Let me know if it works for you !

This did not work for me using Python 2.7.13 and protobuf 3.7.0. Is there any way to get caffe to produce more verbose debug output to trace what is going wrong here?

EDIT: I'm also not sure what your sed commands do, the only occurences in setup.py that are replaced by this seem to be in a branch that is not run unless --compile_static_extension is specified. Could you help me out here @maingoh?.

ANOTHER EDIT: I think it got it working now, I completely removed any remaining protobuf files on my system, installed both protobuf and the protobuf python bindings from source from the 3.7.x branch (without modifying any source files) and then built caffe again from scratch. The segfault does not occur anymore.

[cjolivier01]

Using the shared protobuf lib for 3.8.0 from the python protobuf module worked for me as well, even with python-protobuf package building with the cpp extension (obviously no problem without the c++ extension). This was crashing in exactly the same place for me when launching a c++ program which, in turn, imported tensorflow via pybind (note that libtensorflow_framework.so exports its own build of protobuf symbols).
For whatever reason, TF can have its protobuf shared symbols exposed, and I can link my own protobuf shared object (exact same version, mind you, can verify via ldd), but load the python protobuf module with its static version of protobuf, and everything just goes to pieces.

[York1996OutLook]

i find when i use python debug layer ,i can only use the first GPU !
Dose anybody know why?