Caffe: Make runtest failure

[deepsamal]

Mac OS 10.9, CUDA 7.0
Make all and Make test ran without errors, except some warning about unused -pthread and 1 warning in test. But $make runtest gives segmentation fault. $make sudo runtest gives the error-
.build_release/tools/caffe
dyld: Library not loaded: @rpath/libcudart.7.0.dylib
Referenced from: /Users/deepsamal/Desktop/Research/caffe/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5

Seems a problem with dynamic loading of the cuda library path.
checking the links of libcaffe.so, the paths to cuda libs are not resolved. But the libraries are present both in /usr/local/cuda/lib and in /Developer/NVIDIA.... folders.

Deep-Samals-MacBook-Pro:caffe deepsamal$ otool -L ~/Desktop/Research/caffe/build/lib/libcaffe.so
/Users/deepsamal/Desktop/Research/caffe/build/lib/libcaffe.so:
@rpath/libcaffe.so (compatibility version 0.0.0, current version 0.0.0)
@rpath/libcudart.7.0.dylib (compatibility version 0.0.0, current version 7.0.29)
@rpath/libcublas.7.0.dylib (compatibility version 0.0.0, current version 7.0.29)
@rpath/libcurand.7.0.dylib (compatibility version 0.0.0, current version 7.0.29)
/usr/local/lib/libglog.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/local/lib/libprotobuf.9.dylib (compatibility version 10.0.0, current version 10.1.0)
/usr/local/lib/libleveldb.1.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/lib/libsnappy.1.dylib (compatibility version 4.0.0, current version 4.0.0)
/usr/local/lib/liblmdb.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/lib/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
libhdf5_hl.9.dylib (compatibility version 10.0.0, current version 10.0.0)
libhdf5.9.dylib (compatibility version 10.0.0, current version 10.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
/usr/local/opt/opencv/lib/libopencv_core.2.4.dylib (compatibility version 2.4.0, current version 2.4.11)
/usr/local/opt/opencv/lib/libopencv_highgui.2.4.dylib (compatibility version 2.4.0, current version 2.4.11)
/usr/local/opt/opencv/lib/libopencv_imgproc.2.4.dylib (compatibility version 2.4.0, current version 2.4.11)
/usr/local/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libcudnn.6.5.dylib (compatibility version 0.0.0, current version 6.5.48)
/usr/local/lib/libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

Also I looked at links of the libcudart library-

$ otool -L /usr/local/cuda/lib/libcudart.7.0.dylib
/usr/local/cuda/lib/libcudart.7.0.dylib:
@rpath/libcudart.7.0.dylib (compatibility version 0.0.0, current version 7.0.29)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

In both the outputs, library links seem to be broken. But make all and test are fine, this means dynamic linking is okay, loading is not?

May be the problem is environment variable related, my variables are-
$ echo $DYLD_FALLBACK_LIBRARY_PATH
/usr/local/cuda/lib:/Developer/NVIDIA/CUDA-7.0/lib:/Users/deepsamal/anaconda/lib:/usr/local/lib:/usr/lib:/opt/intel/composer_xe_2015.2.132/mkl/lib/:/opt/intel/composer_xe_2015.2.132/compiler/lib

$ echo $DYLD_LIBRARY_PATH
/usr/local/cuda/lib

$ echo $LD_LIBRARY_PATH
/usr/local/cuda/lib:/opt/intel/composer_xe_2015.2.132/mkl/lib/

$ ECHO $PATH
/Developer/NVIDIA/CUDA-7.0/bin:/Users/deepsamal/anaconda/bin:/Applications/MATLAB_R2014a.app/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin

All advised paths are included, in fact, I even tried without setting LD_LIBRARY_PATH, same error. Any idea what might be the cause?

[iwen-kang]

I have exactly the same problem on my MacBookPro OS 10.10, CUDA 7.0. Any help will be much appreciated, thanks!
=========== terminal output ===========
sudo make runtest
.build_release/tools/caffe
dyld: Library not loaded: @rpath/libcudart.7.0.dylib
Referenced from: /Developer/caffe/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5

output from "env | grep LIB":
DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:/Applications/anaconda/lib:/usr/local/lib:/usr/lib
LD_LIBRARY_PATH=/usr/local/lib
DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-7.0/cudnn-6.5-osx-v2:/Developer/NVIDIA/CUDA-7.0/lib

[shelhamer]

Follow the installation instructions and ask installation questions on the caffe-users group. You should not set DYLD_LIBRARY_PATH.

[MartinHjelm]

This is bit old I guess. And I shouldn't be spamming with install advice but I just had the same problem. At the caffe user forum https://groups.google.com/forum/#!msg/caffe-users/BkMspfbYLIE/Pi6SUVANAKUJ someone said fix the DYLD_FALLBACK_LIBRARY_PATH variable for me it was just to add the /usr/local/cuda/lib as the first path

export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/lib:/usr/lib:/Developer/NVIDIA/CUDA-7.0/lib:

This worked for me.

[wlike]

set DYLD_FALLBACK_LIBRARY_PATH solved my problem too

[gsabran]

I've set my DYLD_FALLBACK_LIBRARY_PATH as suggested and followed the advices from robertsdionne:
echo $DYLD_FALLBACK_LIBRARY_PATH
/usr/local/cuda/lib:/usr/local/lib:/Users/guigou/caffe/distribute/lib:/Developer/NVIDIA/CUDA-7.5/lib:

and I still get

ImportError                               Traceback (most recent call last)
<ipython-input-1-1cca3aa1f8c5> in <module>()
----> 1 import caffe

/Users/guigou/caffe/python/caffe/__init__.py in <module>()
----> 1 from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
      2 from ._caffe import set_mode_cpu, set_mode_gpu, set_device, Layer, get_solver, layer_type_list
      3 from .proto.caffe_pb2 import TRAIN, TEST
      4 from .classifier import Classifier
      5 from .detector import Detector

/Users/guigou/caffe/python/caffe/pycaffe.py in <module>()
     11 import numpy as np
     12 
---> 13 from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
     14         RMSPropSolver, AdaDeltaSolver, AdamSolver
     15 import caffe.io

ImportError: dlopen(/Users/guigou/caffe/python/caffe/_caffe.so, 2): Library not loaded: @rpath/libcudart.7.5.dylib
  Referenced from: /Users/guigou/caffe/python/caffe/_caffe.so
  Reason: image not found

[qiuwch]

I have the exactly the same issue. I also already set

DYLD_LIBRARY_PATH
DYLD_FALLBACK_LIBRARY_PATH

My version is also CUDA7.5. I think when I used CUDA7.0 before, everything is fine.

[dg2]

Caffe seems to be linking in a pretty funny way ... If I run

otool -L .build_release/tools/caffe

I see something like this

.build_release/tools/caffe:
    @rpath/libcaffe.so (compatibility version 0.0.0, current version 0.0.0)
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
    @rpath/libcudart.7.5.dylib (compatibility version 0.0.0, current version 7.5.20)
    @rpath/libcublas.7.5.dylib (compatibility version 0.0.0, current version 7.5.20)
    @rpath/libcurand.7.5.dylib (compatibility version 0.0.0, current version 7.5.20)
    /usr/local/opt/glog/lib/libglog.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/opt/gflags/lib/libgflags.2.dylib (compatibility version 2.0.0, current version 2.1.2)
    /usr/local/lib/libprotobuf.9.dylib (compatibility version 10.0.0, current version 10.1.0)
    /usr/local/lib/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.1.1)
    libhdf5_hl.10.dylib (compatibility version 11.0.0, current version 11.1.0)
    libhdf5.10.dylib (compatibility version 11.0.0, current version 11.1.0)
        ...

So some library paths are static, others are run-time paths and others are just assumed to be in the executable path (which they aren't). Unfortunately, in my case at least there are not rpath's set for either the executables or the libcaffe :( Just check `otool -l .build_release/tools/caffe | grep RPATH'

If you need a quick and dirty fix, you can add your CUDA path as an rpath

install_name_tool -add_rpath /usr/local/cuda/lib .build_release/tools/caffe
install_name_tool -add_rpath /usr/local/cuda/lib .build_release/test/test_all.testbin
install_name_tool -add_rpath /usr/local/cuda/lib .build_release/lib/libcaffe.so

With that I can run make runtest, but you should do the same for a few more files (mostly every caffe binary and also libcaffe.so) if you want to use them. And I also expect trouble with the HDF5 libraries at some point, but you can also use install_name_tool to fix those issues. Anyway, the most reasonable option is to fix the Makefile to do a proper job in OS X.

[cecilpang]

If you are installing on OS X El Capitan, DYLD_FALLBACK_LIBRARY_PATH is cleared by the new System Integrity Protection feature of El Capitan:

https://developer.apple.com/library/watchos/documentation/Security/Conceptual/System_Integrity_Protection_Guide/System_Integrity_Protection_Guide.pdf

make runtest completed successfully after I disabled System Integrity Protection.

[a7744hsc]

@cecilpang Thanks for your tips and it really helps. here is the steps to disable Integrity Protection:

1. Boot to Recovery OS by restarting your machine and holding down the Command and R keys at startup.
2. Launch Terminal from the Utilities menu.
3. Enter the following command:
   $ csrutil disable
   4.restart done

[jiajunshen]

@a7744hsc @cecilpang Your solutions resolved my problem. Thanks!

[a7744hsc]

@jiajunshen Very glad to hear that

[cchriste]

Worked... but so annoying! And now what about my system's integrity? Will I ever be able to trust it again?

[shal233]

Before you run 'make runtest', do 'make clean' once.

[fanshiqing]

@a7744hsc @cecilpang Your solutions resolved my problem too. Many thanks!
BTW, Is there any other possible walk-around without disabling the Integrity Protection of Mac OS?

[kmhofmann]

Shouldn't this maybe be reopened, since there is no good fix for this?
(I don't consider disabling the Mac OS X Integrity Protection a good fix...)

[usernameandme]

Please refer https://github.com/BVLC/caffe/issues/3227 for a workaround without disabling Integrity protection on OSX 10.11.

[apuljain]

What should be the fix if there is no cuda gpu?