Caffe: Segfault on PyCaffe import

[ericye16]

   Python 2.7.10 (default, Jun 10 2015, 19:42:47)
   [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import caffe
   Segmentation fault: 11

Configuration is CPU only. I tried to debug it with LLDB but didn't get very far. Running on Mac OS X 10.10.4. Let me know if you need more information!

[rayz0620]

Try checking your numpy and BLAS linkage? Maybe something is wrong with them.

[ericye16]

>>> np.__version__
'1.9.2'
>>> np.__config__.show()
lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH']
    define_macros = [('NO_ATLAS_INFO', 3)]
openblas_lapack_info:
  NOT AVAILABLE
atlas_3_10_blas_threads_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_3_10_threads_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
atlas_3_10_blas_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3)]
atlas_info:
  NOT AVAILABLE
atlas_3_10_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

I'm running it in a virtualenv, could that mess with it?

[seanbell]

I've had it segfault when importing if it wasn't built properly, or if I rebuilt the library but didn't rebuild pycaffe as well. Have you tried doing a full make clean followed by make all -jN && make pycaffe -jN (where N is the number of processors)?

If you're in a virtualenv, you might need to make sure that all the library versions that get used to build caffe match those in the virtualenv -- not sure about that though.

[rayz0620]

Seems info of ATLAS, OPENBLAS and MKL are all unavailable, which may just indicate that you have none of them linked correctly. Numpy requires at least one of them to work properly.
You may refer to this stackoverflow question:Compiling numpy with OpenBLAS integration

[ericye16]

Still nope.

>>> np.__version__
'1.10.0.dev0+7a3926c'
>>> np.__config__.show()
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    runtime_library_dirs = ['/opt/OpenBLAS/lib']
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    runtime_library_dirs = ['/opt/OpenBLAS/lib']
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    runtime_library_dirs = ['/opt/OpenBLAS/lib']
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/OpenBLAS/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    runtime_library_dirs = ['/opt/OpenBLAS/lib']
blas_mkl_info:
  NOT AVAILABLE
>>> import caffe
Segmentation fault: 11

[ericye16]

More info:

lldb python
(lldb) target create "python"
Current executable set to 'python' (x86_64).
(lldb) r
Process 49253 launched: '/Users/eric/numpy/.env/bin/python' (x86_64)
Python 2.7.10 (default, Jun 10 2015, 19:42:47)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
Process 49253 stopped
* thread #1: tid = 0x49604, 0x0000000000000000, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
(lldb) list
(lldb) bt
* thread #1: tid = 0x49604, 0x0000000000000000, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x0000000103fcd0dd Python`PyEval_GetGlobals + 23
    frame #2: 0x0000000103fdc62b Python`PyImport_Import + 137
    frame #3: 0x0000000103fdad27 Python`PyImport_ImportModule + 31
    frame #4: 0x000000010323acc8 _caffe.so`caffe::init_module__caffe() + 4328
    frame #5: 0x0000000103eee391 libboost_python.dylib`boost::python::handle_exception_impl(boost::function0<void>) + 81
    frame #6: 0x0000000103eef3b9 libboost_python.dylib`boost::python::detail::init_module(char const*, void (*)()) + 121
    frame #7: 0x00000001000a0327 Python`_PyImport_LoadDynamicModule + 140
    frame #8: 0x00000001000a0003 Python`import_submodule + 270
    frame #9: 0x000000010009fbc6 Python`load_next + 284
    frame #10: 0x000000010009db5d Python`PyImport_ImportModuleLevel + 453
    frame #11: 0x0000000100083641 Python`builtin___import__ + 135
    frame #12: 0x000000010000ead7 Python`PyObject_Call + 99
    frame #13: 0x000000010008de2e Python`PyEval_CallObjectWithKeywords + 93
    frame #14: 0x000000010008a274 Python`PyEval_EvalFrameEx + 8334
    frame #15: 0x0000000100087fb4 Python`PyEval_EvalCodeEx + 1387
    frame #16: 0x0000000100087a43 Python`PyEval_EvalCode + 54
    frame #17: 0x000000010009c8c3 Python`PyImport_ExecCodeModuleEx + 241
    frame #18: 0x000000010009f557 Python`load_source_module + 1089
    frame #19: 0x00000001000a0003 Python`import_submodule + 270
    frame #20: 0x000000010009fbc6 Python`load_next + 284
    frame #21: 0x000000010009db5d Python`PyImport_ImportModuleLevel + 453
    frame #22: 0x0000000100083641 Python`builtin___import__ + 135
    frame #23: 0x000000010000ead7 Python`PyObject_Call + 99
    frame #24: 0x000000010008de2e Python`PyEval_CallObjectWithKeywords + 93
    frame #25: 0x000000010008a274 Python`PyEval_EvalFrameEx + 8334
    frame #26: 0x0000000100087fb4 Python`PyEval_EvalCodeEx + 1387
    frame #27: 0x0000000100087a43 Python`PyEval_EvalCode + 54
    frame #28: 0x000000010009c8c3 Python`PyImport_ExecCodeModuleEx + 241
    frame #29: 0x000000010009f557 Python`load_source_module + 1089
    frame #30: 0x000000010009f818 Python`load_package + 303
    frame #31: 0x00000001000a0003 Python`import_submodule + 270
    frame #32: 0x000000010009fbc6 Python`load_next + 284
    frame #33: 0x000000010009db5d Python`PyImport_ImportModuleLevel + 453
    frame #34: 0x0000000100083641 Python`builtin___import__ + 135
    frame #35: 0x000000010000ead7 Python`PyObject_Call + 99
    frame #36: 0x000000010008de2e Python`PyEval_CallObjectWithKeywords + 93
    frame #37: 0x000000010008a274 Python`PyEval_EvalFrameEx + 8334
    frame #38: 0x0000000100087fb4 Python`PyEval_EvalCodeEx + 1387
    frame #39: 0x0000000100087a43 Python`PyEval_EvalCode + 54
    frame #40: 0x00000001000a7816 Python`run_mod + 53
    frame #41: 0x00000001000a7632 Python`PyRun_InteractiveOneFlags + 377
    frame #42: 0x00000001000a711e Python`PyRun_InteractiveLoopFlags + 192
    frame #43: 0x00000001000a6fc8 Python`PyRun_AnyFileExFlags + 60
    frame #44: 0x00000001000b8e09 Python`Py_Main + 3057
    frame #45: 0x00007fff86e6b5c9 libdyld.dylib`start + 1
(lldb)

(Is there a debug build of caffe I can use so I can see the symbols?)

[ericye16]

Looks like a bug in either python or numpy when importing numpy.multicore.array. I'll close if this means it's not a bug in PyCaffe:

lldb python
(lldb) target create "python"
Current executable set to 'python' (x86_64).
(lldb) r
Process 66436 launched: '/Users/eric/numpy/.env/bin/python' (x86_64)
Python 2.7.10 (default, Jun 10 2015, 19:42:47)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
Process 66436 stopped
* thread #1: tid = 0xb355c, 0x0000000000000000, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
(lldb) list
(lldb) bt
* thread #1: tid = 0xb355c, 0x0000000000000000, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x0000000105e3b0dd Python`PyEval_GetGlobals + 23
    frame #2: 0x0000000105e4a62b Python`PyImport_Import + 137
    frame #3: 0x0000000105e48d27 Python`PyImport_ImportModule + 31
    frame #4: 0x0000000103238304 _caffe.so`_import_array() + 20 at __multiarray_api.h:1632
    frame #5: 0x0000000103237f55 _caffe.so`caffe::init_module__caffe() + 3909 at _caffe.cpp:298
    frame #6: 0x0000000105d5c391 libboost_python.dylib`boost::python::handle_exception_impl(boost::function0<void>) + 81
    frame #7: 0x0000000105d5d3b9 libboost_python.dylib`boost::python::detail::init_module(char const*, void (*)()) + 121
    frame #8: 0x0000000103236ffb _caffe.so`init_caffe + 27 at _caffe.cpp:195
    frame #9: 0x00000001000a0327 Python`_PyImport_LoadDynamicModule + 140
    frame #10: 0x00000001000a0003 Python`import_submodule + 270
    frame #11: 0x000000010009fbc6 Python`load_next + 284
    frame #12: 0x000000010009db5d Python`PyImport_ImportModuleLevel + 453
    frame #13: 0x0000000100083641 Python`builtin___import__ + 135
    frame #14: 0x000000010000ead7 Python`PyObject_Call + 99
    frame #15: 0x000000010008de2e Python`PyEval_CallObjectWithKeywords + 93
    frame #16: 0x000000010008a274 Python`PyEval_EvalFrameEx + 8334
    frame #17: 0x0000000100087fb4 Python`PyEval_EvalCodeEx + 1387
    frame #18: 0x0000000100087a43 Python`PyEval_EvalCode + 54
    frame #19: 0x000000010009c8c3 Python`PyImport_ExecCodeModuleEx + 241
    frame #20: 0x000000010009f557 Python`load_source_module + 1089
    frame #21: 0x00000001000a0003 Python`import_submodule + 270
    frame #22: 0x000000010009fbc6 Python`load_next + 284
    frame #23: 0x000000010009db5d Python`PyImport_ImportModuleLevel + 453
    frame #24: 0x0000000100083641 Python`builtin___import__ + 135
    frame #25: 0x000000010000ead7 Python`PyObject_Call + 99
    frame #26: 0x000000010008de2e Python`PyEval_CallObjectWithKeywords + 93
    frame #27: 0x000000010008a274 Python`PyEval_EvalFrameEx + 8334
    frame #28: 0x0000000100087fb4 Python`PyEval_EvalCodeEx + 1387
    frame #29: 0x0000000100087a43 Python`PyEval_EvalCode + 54
    frame #30: 0x000000010009c8c3 Python`PyImport_ExecCodeModuleEx + 241
    frame #31: 0x000000010009f557 Python`load_source_module + 1089
    frame #32: 0x000000010009f818 Python`load_package + 303
    frame #33: 0x00000001000a0003 Python`import_submodule + 270
    frame #34: 0x000000010009fbc6 Python`load_next + 284
    frame #35: 0x000000010009db5d Python`PyImport_ImportModuleLevel + 453
    frame #36: 0x0000000100083641 Python`builtin___import__ + 135
    frame #37: 0x000000010000ead7 Python`PyObject_Call + 99
    frame #38: 0x000000010008de2e Python`PyEval_CallObjectWithKeywords + 93
    frame #39: 0x000000010008a274 Python`PyEval_EvalFrameEx + 8334
    frame #40: 0x0000000100087fb4 Python`PyEval_EvalCodeEx + 1387
    frame #41: 0x0000000100087a43 Python`PyEval_EvalCode + 54
    frame #42: 0x00000001000a7816 Python`run_mod + 53
    frame #43: 0x00000001000a7632 Python`PyRun_InteractiveOneFlags + 377
    frame #44: 0x00000001000a711e Python`PyRun_InteractiveLoopFlags + 192
    frame #45: 0x00000001000a6fc8 Python`PyRun_AnyFileExFlags + 60
    frame #46: 0x00000001000b8e09 Python`Py_Main + 3057
    frame #47: 0x00007fff86e6b5c9 libdyld.dylib`start + 1
(lldb) up
frame #1: 0x0000000105e3b0dd Python`PyEval_GetGlobals + 23
Python`PyEval_GetGlobals:
    0x105e3b0dd <+23>: movq   %rax, %rcx
    0x105e3b0e0 <+26>: xorl   %eax, %eax
    0x105e3b0e2 <+28>: testq  %rcx, %rcx
    0x105e3b0e5 <+31>: je     0x105e3b0eb               ; <+37>
(lldb) up
frame #2: 0x0000000105e4a62b Python`PyImport_Import + 137
Python`PyImport_Import:
    0x105e4a62b <+137>: movq   %rax, %rbx
    0x105e4a62e <+140>: testq  %rbx, %rbx
    0x105e4a631 <+143>: je     0x105e4a665               ; <+195>
    0x105e4a633 <+145>: movq   %r12, -0x30(%rbp)
(lldb) up
frame #3: 0x0000000105e48d27 Python`PyImport_ImportModule + 31
Python`PyImport_ImportModule:
    0x105e48d27 <+31>: movq   %rax, %r14
    0x105e48d2a <+34>: decq   (%rbx)
    0x105e48d2d <+37>: jne    0x105e48d39               ; <+49>
    0x105e48d2f <+39>: movq   0x8(%rbx), %rax
(lldb) up
frame #4: 0x0000000103238304 _caffe.so`_import_array() + 20 at __multiarray_api.h:1632
   1629 _import_array(void)
   1630 {
   1631   int st;
-> 1632   PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray");
   1633   PyObject *c_api = NULL;
   1634
   1635   if (numpy == NULL) {
(lldb) p numpy
(PyObject *) $0 = 0x00007fff5fbfd968
(lldb) up
frame #5: 0x0000000103237f55 _caffe.so`caffe::init_module__caffe() + 3909 at _caffe.cpp:298
   295
   296    // boost python expects a void (missing) return value, while import_array
   297    // returns NULL for python3. import_array1() forces a void return value.
-> 298    import_array1();
   299  }
   300
   301  }  // namespace caffe

[forresto]

Let me know if you figure this out. I'm getting the same Segmentation fault: 11 with the anaconda mkl version of numpy. Thought it might be my old GPU (NVIDIA GeForce 9600M GT), but getting the same with CPU-only build.

[ericye16]

I tried building Python myself and I noticed that I couldn't import numpy due to lacking a PyInit_64 function or something in numpy multiarray. This leads me to think that the issues are because the builds of numpy and the build of caffe maybe aren't the same on my system. Right now the only way I can think of ameliorating this is by building Python, numpy and caffe all from scratch so I can be sure they work together but that's a _lot_ of work...

[artemyk]

For me this was resolved by making sure PYTHON_INCLUDE and PYTHON_LIB (in Makefile.config) pointed to the appropriate brew installed places (/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/include/python2.7 and /usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/)

[ericye16]

Thanks! That worked for me :smile:

[christopher5106]

yep, i created my tutorial with this instruction
http://christopher5106.github.io/big/data/2015/07/16/deep-learning-install-caffe-cudnn-cuda-for-digits-python-on-mac-osx.html

[yi-ji]

The problem is actually caused by /System/Library/Frameworks/Python.framework sneaking into pycaffe compiling somehow. This happens because homebrew-installed python and system python are both on your computer.

To solve this problem, a simple way is:
sudo mv /System/Library/Frameworks/Python.framework /System/Library/Frameworks/Python.framework.bak sudo ln -s /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework /System/Library/Frameworks/Python.framework

It replaces system's python framework with homebrew one. Since your pycaffe is made with homebrew python, this will solve the segmentation fault caused by system's python sneaking,

[anatoliykmetyuk]

The solution with replacement of the system framework with the homebrew's one doesn't work in case if you use Anaconda's python, since it doesn't come in the form of a framework. Also, you need to disable the system integrity check on El Capitan (maybe also some earlier systems) as described here in order to manipulate its frameworks.

Here is what to do in case of Anaconda. From python/caffe, do the following (substitute your own path to Anaconda!):

install_name_tool -change libpython2.7.dylib /path/to/anaconda/lib/libpython2.7.dylib _caffe.so

It will relink _caffe.so to the Anaconda's python library (I think same trick is possible with homebrew if you locate its libpython2.7.dylib). See this for more information.

[shelhamer]

It should not be necessary to disable system integrity protection if compiling since #3634 and #3696 (with the possible exception of MKL BLAS) -- I am running Caffe on El Cap without tampering with system integrity protection, although it was broken until the linking was adjusted in those PRs.

[anatoliykmetyuk]

It is not necessary to disable the integrity when compiling caffe, but it is when you try to do sudo mv /System/Library/Frameworks/Python.framework /System/Library/Frameworks/Python.framework.bak, as described in @yiisama's solution.

[anatoliykmetyuk]

And, by the way, the linking is still not perfect, I had to manually relink these two (from the otool -L _caffe.so output):

/Users/anatolii/anaconda/lib/libhdf5_hl.10.dylib (compatibility version 11.0.0, current version 11.1.0)
/Users/anatolii/anaconda/lib/libhdf5.10.dylib (compatibility version 11.0.0, current version 11.1.0)

Previously, they were under @rpath, but I installed hdf5 with Anaconda, and it was not seen. No integrity protection tampering was necessary in this case either, though.

[shelhamer]

Oh ok, any help with the linking is appreciated! Thanks for the details @anatoliykmetyuk .

[infoxiao]

I encountered the same issue and @tounatsu 's solution worked!

[whcn]

I also encountered this question, it took my a lot of time to solve

The root is brew python conflict with system python

Below is my solution:

In Makefile.config, find PYTHON_INCLUDE, we can see the comment above this line,
it says We need to be able to find Python.h and numpy/arrayobject.h.,
I want to use brew's python and brew's numpy, so i modified to

PYTHON_INCLUDE := /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7 \  

/usr/local/Cellar/numpy/1.12.0/lib/python2.7/site-packages/numpy/core/include/

Then, find PYTHON_LIB, we can see the comment above this line,
it say # We need to be able to find libpythonX.X.so or .dylib., same as above, i modified to

PYTHON_LIB := /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib

you can find those files in these locations, then you can do

<caffe_root>/python:
for req in $(cat requirements.txt); do pip install $req; done

<caffe_root>:
make all -j4
make runtest
make pycaffe

I hope this can help.

[willard-yuan]

@artemyk Awesome, it works for me.

[thundertrick]

This is due to conflict between brew python and OS X system python.
Homebrew has provided an awesome solution for OS X:

$ python -V   # system Python interpreter
$ python2 -V  # Homebrew installed Python 2 interpreter
$ python3 -V  # Homebrew installed Python 3 interpreter (if installed)

So the solution is changing all the python paths to python2. Bellowing is my Makefile.config related:

# ...
# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
#       /usr/lib/python2.7/dist-packages/numpy/core/include
# ------ For Homebrew installed python. Numpy path is added using python commands. 
PYTHON_INCLUDE := /usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/include/python2.7

# We need to be able to find libpythonX.X.so or .dylib. ------ (Update Homebrew path)
# PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib
PYTHON_LIB := /usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib

# Homebrew installs numpy in a non standard path (keg only) ------ (python2 for brew instead of python for system)
PYTHON_INCLUDE += $(dir $(shell python2 -c 'import numpy.core; print(numpy.core.__file__)'))/include
PYTHON_LIB += $(shell brew --prefix numpy)/lib
# ...

[AnkitGupta17134]

For me this was resolved by making sure PYTHON_INCLUDE and PYTHON_LIB (in Makefile.config) pointed to the appropriate brew installed places (/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/include/python2.7 and /usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/)

Worked for me too......!!! Thanks