Incubator-mxnet: fail to install mxnet using GPU

[Yunrui]

hi all,

I cannot install mxnet after enabling CUDA and upgrading to OS X EI Capitan. I read previous post, one possibility is about LD_LIBRARY_PATH, but I get it set.

Actually, at least I get mxnet installed (with GPU) on OS X yosimite, but it complains something like "cuda runtime incompatible with library". Then I decided to upgrade to EI Capitan and installed new version of CUDA for 10.11. Then I encountered a build problem of blas, later solved by modification of my settings. Then comes to this issue.

=============== Env =====================
export LD_LIBRARY_PATH=/usr/local/cuda:/usr/local/cuda/lib:/usr/local:/usr/local/lib:$LD_LIBRARY_PATH

=============== ERROR ==================
installing to /Library/Frameworks/R.framework/Versions/3.2/Resources/library/mxnet/libs
* R
** demo
** inst
** preparing package for lazy loading
** help
** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
Error : .onLoad failed in loadNamespace() for 'mxnet', details:
call: dyn.load(file, DLLpath = DLLpath, ...)
error: 无法载入共享目标对象‘/Library/Frameworks/R.framework/Versions/3.2/Resources/library/mxnet/libs/libmxnet.so’：:
dlopen(/Library/Frameworks/R.framework/Versions/3.2/Resources/library/mxnet/libs/libmxnet.so, 10): Library not loaded: @rpath/libcudart.7.5.dylib
Referenced from: /Library/Frameworks/R.framework/Versions/3.2/Resources/library/mxnet/libs/libmxnet.so
Reason: image not found
错误: 载入失败
停止执行
ERROR: loading failed
removing ‘/Library/Frameworks/R.framework/Versions/3.2/Resources/library/mxnet’

=============== My settings ===============
ADD_LDFLAGS = -I/usr/local/opt/openblas/lib -I/usr/local/cuda/lib

ADD_CFLAGS = -I/usr/local/opt/openblas/include

USE_CUDA = 1

USE_CUDA_PATH = /usr/local/cuda

[pluskid]

Maybe instead of LD_LIBRARY_PATH, try using DYLD_LIBRARY_PATH ?

[pluskid]

Also as a workaround, you can fix the @rpath problem by fixing the libmxnet.so directly. Firstly, use  otool -L to list referenced libraries of libmxnet.so. For those references with @rpath/…, use install_name_tool -change to change them to absolute path. Something like

Install_name_tool -change @rpath/libcudart.7.0.0.dyld /usr/local/cuda/lib/libcudart.7.0.0.dyld

[Yunrui]

thanks @pluskid, replacing @rpath indeed fixed this problem.

But, I still get an error when training with GPU. I am already using latest package of CUDA thru https://developer.nvidia.com/cuda-downloads. how did you resolve this issue?

model <- mx.model.FeedForward.create(lenet, X=train.array, y=train.y,

• ctx=device.gpu, num.round=5, array.batch.size=100,
• learning.rate=0.05, momentum=0.9, wd=0.00001,
• eval.metric=mx.metric.accuracy,
• epoch.end.callback=mx.callback.log.train.metric(100))
  Start training with 1 devices
  [12:13:00] ./dmlc-core/include/dmlc/logging.h:208: [12:13:00] src/storage/storage.cc:41: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: CUDA driver version is insufficient for CUDA runtime version
  错误: [12:13:00] src/storage/storage.cc:41: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: CUDA driver version is insufficient for CUDA runtime version

[Yunrui]

My CUDA driver version is: 7.5.21. The library version I used is also 7.5.

[tqchen]

Maybe you want to first check if other cuda based library runs, as installed cuda driver does not necessarily mean you are using it. By default ubuntu have another driver that was quite hard to get rid of.

[Yunrui]

Thanks, the issue is my macbook pro (13') is using integrated graphic card, Intel iris, which I think not supported by CUDA. :-(.....

[changkun]

Well, Integrity Protection seems like a really big problem when frameworks do not adapt for OSX 10.11. On the Runtime Protections section we know that any dynamic linker (dyld) environment variables,such as DYLD_LIBRARY_PATH, are purged when launching protected processes.
As result, I got the same problem with @Yunrui which shows

Library not loaded: @rpath/libcudart.7.5.dylib

The solution provided by Apple lead us to shut down System Integrity Protection which means we need to go to OS Recovery Mode(Command + R + Power), use Terminal from Utilities Menu and input:

$ csrutil disable

Reboot the system then everything goes fine.

[gouchangjiang]

Also I have across the same question at ubuntu OS. I have solved it by changing LD_LIBRARY_PATH environment variable. I set LD_LIBRARY_PATH="where the libcudart.7.0.5 is". If you don't know how to change environment variable, the links following may help: https://help.ubuntu.com/community/EnvironmentVariables
http://serverfault.com/questions/201709/how-to-set-ld-library-path-in-ubuntu
may it help somebody.

[leocnj]

I am also trying installing the R package (w/ GPU support) on OS X EI Capitan. After doing

• disabling Integrity Protection as suggested by @changkun
• set LD_LIBRARY_PATH to include "where the libcudart.7.0.5 is"
• set DYLD_FALLBACK_LIBRARY_PATH to include "where the libcudart.7.0.5 is"; note that this step may not be mandatory

I can compile the R package and a testing running on both CPU and GPU works inside the R sessions initialized in a terminal.

However, when loading mxnet inside RStudio (the latest version), the following error pops up

> require(mxnet)
Loading required package: mxnet
Error : .onLoad failed in loadNamespace() for 'mxnet', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/Library/Frameworks/R.framework/Versions/3.2/Resources/library/mxnet/libs/libmxnet.so':
  dlopen(/Library/Frameworks/R.framework/Versions/3.2/Resources/library/mxnet/libs/libmxnet.so, 10): Library not loaded: @rpath/libcudart.7.5.dylib
  Referenced from: /Library/Frameworks/R.framework/Versions/3.2/Resources/library/mxnet/libs/libmxnet.so
  Reason: image not found

Any idea on enabling mxnet in RStudio? Thanks.

[tqchen]

@leocnj I am not export in RStuidio, but I guess this have something to do with the env library path of RStudio, @thirdwing do you have any suggestions?

[thirdwing]

@leocnj Three possible solutions you can try:

(1) put the export LD_LIBRARY_PATH = ... into .bashrc, then logout;

(2) Startup RStudio from terminal by using /Applications/RStudio.app/Contents/MacOS/RStudio;

(3) use Sys.setenv in RStudio before you load mxnet.

[leocnj]

@thirdwing , (2) works for me. Thanks.

Before seeing your comment, I already tried (3) this morning and got no luck. For (1), I always had LD_LIBARY_PATH (containing /usr/local/cuda/lib) in my .bashrc file.

[voreiche]

Thanks @pluskid for pointing in the right direction.
I found the following command helpful on OSX 10.11.4:
sudo install_name_tool -add_rpath /usr/local/cuda/lib lib/libmxnet.so
This will add the path to your local cuda library installation (/usr/local/cuda/lib) to the list of linker commands listed in the libmxnet.so library. It assumes that you are at MXNET_HOME.