Caffe: CUDA driver version is insufficient for CUDA runtime version

[antimani]

Hi all,

I was trying to install caffe. I followed steps given here:
http://caffe.berkeleyvision.org/installation.html
After copying Makefile.config.example from Makefile.config (uncommented the required part)
Then make all and make test, runs with out any error but after doing "make runtest", It get this error.

Note: Randomizing tests' orders with a seed of 28931 .
[==========] Running 401 tests from 74 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from BlobSimpleTest/0, where TypeParam = float
[ RUN ] BlobSimpleTest/0.TestInitialization
[ OK ] BlobSimpleTest/0.TestInitialization (0 ms)
[ RUN ] BlobSimpleTest/0.TestPointers
F0718 18:53:42.368805 2078274320 syncedmem.cpp:47] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version
* Check failure stack trace:
@ 0x1152a4a8a google::LogMessage::Fail()
@ 0x1152a3ce8 google::LogMessage::SendToLog()
@ 0x1152a473a google::LogMessage::Flush()
@ 0x1152a80f8 google::LogMessageFatal::~LogMessageFatal()
@ 0x1152a4f25 google::LogMessageFatal::~LogMessageFatal()
@ 0x10e4875fe caffe::SyncedMemory::to_gpu()
@ 0x10e4870ee caffe::SyncedMemory::gpu_data()
@ 0x10e444976 caffe::Blob<>::gpu_data()
@ 0x10e3063dc caffe::BlobSimpleTest_TestPointers_Test<>::TestBody()
@ 0x10e41aa4c testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x10e40ac5a testing::Test::Run()
@ 0x10e40bba2 testing::TestInfo::Run()
@ 0x10e40c270 testing::TestCase::Run()
@ 0x10e4117b7 testing::internal::UnitTestImpl::RunAllTests()
@ 0x10e41b344 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x10e4114c9 testing::UnitTest::Run()
@ 0x10e2f7c49 main
@ 0x7fff95b1f5fd start
make: ** [runtest] Abort trap: 6

I'm running this on a MacBook Pro with OS X Mavericks.
Any idea how I could solve this issue? Software details are:

ProductName: Mac OS X
ProductVersion: 10.9.4
BuildVersion: 13E28
Xcode 5.1.1
Build version 5B1008
Cuda tookit: 6.0

Thank you very much in advance.

Manisha

[askender]

Same situation here
Thanks

[Yangqing]

You need to update your cuda driver - usually when you install the toolkut there is an option to install the latest driver with that toolkit, but I am sure you can install it separately too.

[shelhamer]

You should always install the driver separately to the latest version. The
bundled one is usually out it date and in particular the 331.* series is
broken.

On Friday, July 25, 2014, Yangqing Jia [email protected] wrote:

You need to update your cuda driver - usually when you install the toolkut
there is an option to install the latest driver with that toolkit, but I am
sure you can install it separately too.

—
Reply to this email directly or view it on GitHub
https://github.com/BVLC/caffe/issues/736#issuecomment-50160917.

Evan Shelhamer

[asampat3090]

I read we can run CUDA without an nVidia GPU, but I get the same error when I try this. Seems like I need the driver but I'm not sure it is possible without the physical hardware. Is there a workaround?

[Yangqing]

If you would like to run CUDA without the GPU, compile with CPU_ONLY by
uncommenting here:

https://github.com/BVLC/caffe/blob/master/Makefile.config.example#L8

Yangqing

On Tue, Feb 24, 2015 at 9:25 PM, Anand Sampat [email protected]
wrote:

I read we can run CUDA without an nVidia GPU, but I get the same error
when I try this. Seems like I need the driver but I'm not sure it is
possible without the physical hardware. Is there a workaround?

Reply to this email directly or view it on GitHub
https://github.com/BVLC/caffe/issues/736#issuecomment-75907620.

[asampat3090]

Yea I tried that but then I got a completely different error (which seems to be unsolved based on what I see here - https://groups.google.com/forum/#!msg/caffe-users/kAYc1REXJT4/Lcwcshm0bcMJ). But in any case I was just wondering if we could emulate the GPU using the CUDA libraries without the hardware.

[jianfengluo1]

I got same problem. I uncommented out CPU_ONLY : =1 in my Makefile.config, then use make runtest.
I still got error message like below:
[----------] 6 tests from SGDSolverTest/3, where TypeParam = N5caffe9DoubleGPUE
[ RUN ] SGDSolverTest/3.TestLeastSquaresUpdateLROneTenth
F0408 23:04:03.994313 2135417616 syncedmem.cpp:57] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version
* Check failure stack trace: *
@ 0x10eb531da google::LogMessage::Fail()
@ 0x10eb5223d google::LogMessage::SendToLog()
@ 0x10eb52e60 google::LogMessage::Flush()
@ 0x10eb598f8 google::LogMessageFatal::~LogMessageFatal()
@ 0x10eb53675 google::LogMessageFatal::~LogMessageFatal()
@ 0x10e74cfe1 caffe::SyncedMemory::to_gpu()
@ 0x10e74cb5e caffe::SyncedMemory::gpu_data()
@ 0x10e6b2456 caffe::Blob<>::gpu_data()
@ 0x10e7783c2 caffe::InnerProductLayer<>::Forward_gpu()
@ 0x10e6ea7e3 caffe::Layer<>::Forward()
@ 0x10e738b6e caffe::Net<>::ForwardFromTo()
@ 0x10e739238 caffe::Net<>::Forward()
@ 0x10dfb3719 caffe::GradientBasedSolverTest<>::ComputeLeastSquaresUpdate()
@ 0x10dfb2ff0 caffe::GradientBasedSolverTest<>::TestLeastSquaresUpdate()
@ 0x10e223afc testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x10e21169a testing::Test::Run()
@ 0x10e212492 testing::TestInfo::Run()
@ 0x10e212ba0 testing::TestCase::Run()
@ 0x10e2194c5 testing::internal::UnitTestImpl::RunAllTests()
@ 0x10e224524 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x10e2190a9 testing::UnitTest::Run()
@ 0x10deed41d main
@ 0x7fff97ab75fd start
@ 0x2 (unknown)
/bin/sh: line 1: 10346 Abort trap: 6

[ysolovyov]

You have to set CPU_ONLY := 1 in Makefile.config as well, then all tests are passed.

[pythonanonuser]

I'm having the same problem as the initial post, but I've updated the CUDA driver to the latest on the website. More specifically, when I try make runtest, the error I get is

Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version

I've tried uninstalling CUDA and the drivers and reinstalling, but the error still seems to persist. Any advice?

[alfredox10]

I am having the exact same issue and I am on the latest stable CUDA driver which is 352.41. I also tried on the newer 355.11.

on the newer driver the error is a bit different:
syncedmem.cpp:65] Check failed: error == cudaSuccess (38 vs. 0) no CUDA-capable device is detected

[riccitensor]

The same problem with Mac OS X 10.9.5 and CUDA 10.10 for x86_64 architecture (using Torch7).

The error is:

$ luajit -lcutorch
luajit: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at /tmp/luarocks_cutorch-scm-1-3638/cutorch/lib/THC/THCGeneral.c:16
stack traceback:
[C]: at 0x085b05b0
[C]: in function 'require'
/usr/local/share/lua/5.1/cutorch/init.lua:2: in main chunk
[C]: at 0x01081a1980
[C]: at 0x0108174e90

Looks like it's CUDA-related.

[hycis]

I have the same problem, running mac OS X 10.11 with coda 7.5, i run deviceQuery and everything looks normal, and when i run theano on gpu, also runs smoothly, the only issues arise with torch. So I have sufficient confident to believe the "CUDA driver version is insufficient for CUDA runtime version" error is related to cutorch and not cuda. Probably some bug or checksum error in cutorch.

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GT 750M"
CUDA Driver Version / Runtime Version 7.5 / 7.5
CUDA Capability Major/Minor version number: 3.0
Total amount of global memory: 2048 MBytes (2147024896 bytes)
( 2) Multiprocessors, (192) CUDA Cores/MP: 384 CUDA Cores
GPU Max Clock rate: 926 MHz (0.93 GHz)
Memory Clock rate: 2508 Mhz
Memory Bus Width: 128-bit
L2 Cache Size: 262144 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.5, NumDevs = 1, Device0 = GeForce GT 750M
Result = PASS

[chqsark]

Same issue here. I followed the installation steps for Mac and my cuda is 7.5. No matter I installed the cuda driver together with the toolkit installer or separately, I got the same "CUDA driver version is insufficient" error.

In order to run it, I have to set CPU_ONLY := 1 in Makefile.config and redo the make, and use the CPU training mode for my models, which makes it run smoothly.

Have anyone resolved the cuda driver problem?

Thanks!

[kli-casia]

Hey guys, I have the same error too.
I compile the latest caffe master.

Note: Randomizing tests' orders with a seed of 19616 .
[==========] Running 1868 tests from 255 test cases.
[----------] Global test environment set-up.
[----------] 6 tests from FlattenLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN      ] FlattenLayerTest/3.TestForward
E0217 07:42:05.704968   758 common.cpp:104] Cannot create Cublas handle. Cublas won't be available.
E0217 07:42:05.705651   758 common.cpp:111] Cannot create Curand generator. Curand won't be available.
F0217 07:42:05.705719   758 syncedmem.hpp:18] Check failed: error == cudaSuccess (35 vs. 0)  CUDA driver version is insufficient for CUDA runtime version
*** Check failure stack trace: ***
    @     0x2b633d6ccd0d  google::LogMessage::Fail()
    @     0x2b633d6d20ca  google::LogMessage::SendToLog()
    @     0x2b633d6cedf0  google::LogMessage::Flush()
    @     0x2b633d6cf0dd  google::LogMessageFatal::~LogMessageFatal()
    @     0x2b6340dfd628  caffe::SyncedMemory::mutable_cpu_data()
    @     0x2b6340ed003c  caffe::Blob<>::Reshape()
    @     0x2b6340ed04d9  caffe::Blob<>::Reshape()
    @     0x2b6340ed056c  caffe::Blob<>::Blob()
    @           0x5bd99b  caffe::FlattenLayerTest<>::FlattenLayerTest()
    @           0x5bdc6b  testing::internal::TestFactoryImpl<>::CreateTest()
    @           0x90fbc3  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0x9078e5  testing::TestInfo::Run()
    @           0x907a05  testing::TestCase::Run()
    @           0x909c98  testing::internal::UnitTestImpl::RunAllTests()
    @           0x909f27  testing::UnitTest::Run()
    @           0x50e1ff  main
    @     0x2b6341cd6fa5  __libc_start_main
    @           0x50dda5  (unknown)
make: *** [runtest] Aborted (core dumped)

The information of nvidia-smi is

| NVIDIA-SMI 352.39     Driver Version: 352.39         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K20m          Off  | 0000:08:00.0     Off |                    0 |
| N/A   29C    P0    50W / 225W |     11MiB /  4799MiB |     53%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Thank you very much!

[jainanshul]

I have the same problem, has anyone able to solve it without using CPU only variant.

[alfredox10]

Try to install nvidia-modprobe with synaptic package manager. That fixed it for me, for some reason my issue was the drivers were installed but not getting initialized.

[jainanshul]

I am using OSX and not Ubuntu.

[chuckseberino]

I found this thread searching for a similar problem I have on a different piece of code. My configuration is on OS X 10.11.3, CUDA 7.5 on a MBP with a GeForce GT 650M. I resolved the issue by compiling with shared CUDA libraries instead of static (*.dylib instead of *_static.a). Might try looking to see how you are linking.

[robot010]

@chuckseberino Hi, can you explain how to resolved the issue by compiling with shared CUDA libraries instead of static (*.dylib instead of *_static.a). I've checked the Makefile.config and didn't find any '*.dylib' in it. I am still new to caffe, and I went through the installation steps smoothly except the 'make runtest' part, thank you! :)

[chuckseberino]

@Bato803 I am not a caffe user, so I can't really comment on exactly how caffe builds. I ran into the exact same problem with my own code and found this thread when googling for a solution. In my case I got it to work after tweaking my own configuration to use shared libraries instead of static ones. Maybe checking the output of your build to see exactly what libraries are being linked might help to see if the problem is the same or not.

[AhmedElsiddieg]

Any help? :( when i run make runtest -j $(($(nproc) + 1))
I get:

.build_release/tools/caffe
caffe: command line brew
usage: caffe <command> <args>

commands:
  train           train or finetune a model
  test            score a model
  device_query    show GPU diagnostic information
  time            benchmark model execution time

  Flags from tools/caffe.cpp:
    -gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
      '-gpu all' to run on all available GPUs. The effective training batch
      size is multiplied by the number of devices.) type: string default: ""
    -iterations (The number of iterations to run.) type: int32 default: 50
    -model (The model definition protocol buffer text file.) type: string
      default: ""
    -sighup_effect (Optional; action to take when a SIGHUP signal is received:
      snapshot, stop or none.) type: string default: "snapshot"
    -sigint_effect (Optional; action to take when a SIGINT signal is received:
      snapshot, stop or none.) type: string default: "stop"
    -snapshot (Optional; the snapshot solver state to resume training.)
      type: string default: ""
    -solver (The solver definition protocol buffer text file.) type: string
      default: ""
    -weights (Optional; the pretrained weights to initialize finetuning,
      separated by ','. Cannot be set simultaneously with snapshot.)
      type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle 
Cuda number of devices: 0
Setting to use device 0
Current device id: 0
Current device name: 
Note: Randomizing tests' orders with a seed of 49981 .
[==========] Running 1949 tests from 259 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from FilterLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN      ] FilterLayerTest/2.TestReshape
E0528 02:23:09.611954  7231 common.cpp:113] Cannot create Cublas handle. Cublas won't be available.
E0528 02:23:09.628177  7231 common.cpp:120] Cannot create Curand generator. Curand won't be available.
F0528 02:23:09.628284  7231 syncedmem.hpp:18] Check failed: error == cudaSuccess (30 vs. 0)  unknown error
*** Check failure stack trace: ***
    @     0x2ada35497daa  (unknown)
    @     0x2ada35497ce4  (unknown)
    @     0x2ada354976e6  (unknown)
    @     0x2ada3549a687  (unknown)
    @     0x2ada37576528  caffe::SyncedMemory::mutable_cpu_data()
    @     0x2ada375a37cc  caffe::Blob<>::Reshape()
    @     0x2ada375a3c79  caffe::Blob<>::Reshape()
    @     0x2ada375a3d0c  caffe::Blob<>::Blob()
    @           0x46ae52  testing::internal::TestFactoryImpl<>::CreateTest()
    @           0x891cb3  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0x8888b5  testing::TestInfo::Run()
    @           0x8889d5  testing::TestCase::Run()
    @           0x88bd18  testing::internal::UnitTestImpl::RunAllTests()
    @           0x88bfa7  testing::UnitTest::Run()
    @           0x46269f  main
    @     0x2ada38428ec5  (unknown)
    @           0x469b69  (unknown)
    @              (nil)  (unknown)
make: *** [runtest] Aborted`

I'm on ubuntu 14.04
The nvidia-smi result is

Sat May 28 02:44:04 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 355.11     Driver Version: 355.11         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 630M     Off  | 0000:01:00.0     N/A |                  N/A |
| N/A   56C    P0    N/A /  N/A |    249MiB /  2047MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+

[codecolony]

Same problem on Mac OSX. I even updated the driver but no change in error. Anyone got it working?

[codecolony]

OK, it was a dumb mistake from my side. The code I was running was setting gpu mode internally. I changed it to cpu mode and all is well.

[yosnapin]

Same issue here...
Urgently looking for a fix. Any ideas?
(I need the GPU mode)

[shaoxuan92]

Hi, China here. Same issue .. stucked in here when doing my final term project

[AhmedElsiddieg]

CUDA runtime error (30) might show if your program is unable to create or open the /dev/nvidia-uvm device file. This is usually fixed by installing package nvidia-modprobe:

sudo apt-get install nvidia-modprobe

and Note that you can't use CuDNN if you have a GPU compute capability of < 3.0
Hope this helps :)

[shaoxuan92]

thank you ,I have already installed nvidia-modprobe before,
It still doesn't work when I run make runtest command.
it goes like
caffe: command line brew
....
....
modprobe: error couldont insert 'nvidia_340_uvm': invalid argument
'''
'''

modprobe
cant creat cublas handle ..
cant creat curand gerator..

2016-06-19

Shao Xuan

发件人："Ahmed Elsiddieg A. Abdalla" [email protected]
发送时间：2016-06-19 22:07
主题：Re: [BVLC/caffe] CUDA driver version is insufficient for CUDA runtime version (#736)
收件人："BVLC/caffe"[email protected]
抄送："shaoxuan92"[email protected],"Comment"[email protected]

CUDA runtime error (30) might show if your program is unable to create or open the /dev/nvidia-uvm device file. This is usually fixed by installing package nvidia-modprobe:
sudo apt-get install nvidia-modprobe
and Note that if you can't use CuDNN if you have a GPU compute capability of < 3.0
Hope this helps :)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[AhmedElsiddieg]

@shaoxuan92 I'll post what worked for me in the order that I followed. (from my earlier comment you can see my laptop specs.)
1-Install nvidia driver: https://www.youtube.com/watch?v=A33xwPMdZzk
2-install CUDA: https://www.youtube.com/watch?v=QbvcYnc6o2U and https://www.youtube.com/watch?v=XLEm3jmRHMk
3-Now you can skip the nvidia driver and CUDA and proceed afterwards from here: https://github.com/tiangolo/caffe/blob/5c5c46ec21d41b573d8d23c77ce09c044fa4000c/docs/install_apt2.md and when you get to the make runtest stage use sudo apt-get install nvidia-modprobe beforehand.
This was painful indeed!
Hope it helps and good luck on your project :)

[shaoxuan92]

@kli-nlpr same problem with you , maybe you could follow what i have done with reference to the above comment by @AhmedElsiddieg
@AhmedElsiddieg Thank you for your video ! I followed the video but with slight changes.
http://blog.csdn.net/u013634684/article/details/50813527 hopefully you guys could read Chinese. briefly I just sudo apt-get --purge remove nvidia-* ,and follow the video.
2 when installing cuda. be aware there is a tip :sudo apt-get autoremove when I install!!! I think this is the key.( cause i think the autoversion of cublas is an old version5.5 instead of 7.5)

[danieleghisi]

Same problem here, on Mac OSX 10. The solutions I've read so far in this thread only seem to be tailored on Linux...
Has anyone managed to solve this on Mac? I have Cuda 7.5...

[HuaizhengZhang]

Same problem here! Mac 10.11

*** Aborted at 1468578165 (unix time) try "date -d @1468578165" if you are using GNU date ***
PC: @     0x7fff8b0cbf06 __pthread_kill
*** SIGABRT (@0x7fff8b0cbf06) received by PID 42025 (TID 0x7fff7b391000) stack trace: ***
    @     0x7fff928a652a _sigtramp
    @     0x7fff5c75ee60 (unknown)
    @     0x7fff900e76e7 abort
    @        0x104033193 H5check_version
    @        0x1048af125 caffe::HDF5DataLayer<>::LoadHDF5FileData()
    @        0x1048ae94c caffe::HDF5DataLayer<>::LayerSetUp()
    @        0x104921049 caffe::Net<>::Init()
    @        0x10491fbc5 caffe::Net<>::Net()
    @        0x1049354e1 caffe::Solver<>::InitTrainNet()
    @        0x1049347e7 caffe::Solver<>::Init()
    @        0x10493455b caffe::Solver<>::Solver()
    @        0x1035bf824 caffe::SGDSolver<>::SGDSolver()
    @        0x1035d08ff caffe::AdamSolver<>::AdamSolver()
    @        0x1035d083c caffe::AdamSolverTest<>::InitSolver()
    @        0x1035bf738 caffe::GradientBasedSolverTest<>::InitSolverFromProtoString()
    @        0x1035c18b4 caffe::GradientBasedSolverTest<>::RunLeastSquaresSolver()
    @        0x1035c8112 caffe::GradientBasedSolverTest<>::TestLeastSquaresUpdate()
    @        0x1037a7dac testing::internal::HandleExceptionsInMethodIfSupported<>()
    @        0x10379734a testing::Test::Run()
    @        0x103798142 testing::TestInfo::Run()
    @        0x103798850 testing::TestCase::Run()
    @        0x10379e3c7 testing::internal::UnitTestImpl::RunAllTests()
    @        0x1037a8604 testing::internal::HandleExceptionsInMethodIfSupported<>()
    @        0x10379e0d9 testing::UnitTest::Run()
    @        0x1034a01c5 main
    @     0x7fff9dbf95ad start

[RachelSun]

Same problem and trying to find the solution...

[TigerDeng]

@ALL Note that Macbook pro does not have NVIDIA graphic card.

[RachelSun]

Uncomment CPU_ONLY := 1

[cemsaz]

I get this message on Mac when I run pyfaster rcnn demo.py. I already set CPU_ONLY:=1

[RachelSun]

I comment everything CUDA related, such as: #USE_CUDNN := 1, # CUDA_DIR :=/usr/local/cuda, etc.

[erohan]

I also got the same error like "No CUDA driver detected 35 vs. 0" but i solved through tracing the installation mechanism @nvidia web site. (http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#axzz4FiEeAwS6)

I am running on Ubuntu 14.04 and through the link i wrote above, if you setup the driver and other required packages (approxiamtely 2 GB disk needed) through remote *.deb package installation everything goes well. So the steps like below:

0-) Perform through here for Ubuntu. (http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#ubuntu-installation)
1-) Perform post installation for Cuda also. (http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions)
2-) make the Nvidia Cuda samples to check installation.
2.1 - Go to directory /usr/local/cuda/samples
2.2 - Run sudo make -j8
2.3 - Go to directory after building of samples: /usr/local/cuda/samples/bin/x86_64/linux/release
2.4- Run deviceQuery execuatable to check.
3-) Rebuild the caffe with make all -j8
4-) Run tools/demo.py to check again.

[NickShargan]

@alfredox10 Thanks a lot. It works!

[DhyegoNieto]

I have the same problem using mac 10.11 + CUDA 7.5.30 So I tried just the CPU only ( CPU_ONLY := 1), then:
make clean
make all
make test
make runtest

The error isn't happening anymore.

[jacobunderlinebenseal]

same mistake, use cuda 8.0 with GTX 1060

[svanschalkwyk]

Same issue: https://github.com/BVLC/caffe/issues/5229
Mmmm, just checked my CUDA Samples. This seems to be a CUDA issue only....
[SOLVED]
sudo apt-get purge nvidia*
sudo apt-get purge nvidia*
Download latest NVIDIA driver from NVIDIA website. Follow install.
You may have errors with it finding libGL.so.1:
ln -sf /usr/lib/x86_64-linux-gnu/libGL.so.1 /usr/lib/libGL.so.1

[RuaYahya]

I received the same error as in the attached photo. The nvidia and cudnn are installed. My laptop is MacBook pro but the operating system is Ubuntu 16.04 . The GPU is 512 Mb

. Can I still run it or the specification of my laptop is not sufficient. Please Advice. Thanks in advance.

[svanschalkwyk]

Hi.
It isn't seeing your CUDA device.
Run nvidia-smi:
[image: Inline image 1]
Then add to your ~/.bashrc (or whatever shell you're using) the following
line
"export CUDA_VISIBLE_DEVICES=0,1"
Using the EXACT number(s) that nvidia-smi gave you. In my case, 0 and 1.
Yours may start at 1.

>
>

[MaffooBristol]

I can assure you that my Macbook Pro does have an NVDIA graphics card...

[RuaYahya]

Thanks Mr.svanschalkwyk and Mr.MaffooBristol for your comments. You help to understand the problem and I have passed the runtest command.
The problem is that my MacbookPro does not have the NVIDIA graphics card and therefore I try it on another laptop and it works fine.
Thanks again

[mrgloom]

Get same error:
~
[2017-05-03 13:16:29] Caffe::set_mode: CPU
F0503 13:16:29.411273 4273 cudnn_conv_layer.cpp:52] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version
~

What if I want to build project which is linked with caffe library which was build with Cuda and CuDNN and I'm doing it under VirtualBox Ubuntu 14.04 and my Makefile contain -DCPU_ONLY flag, so I want to run only CPU code under VM is this possible?

[lc8631058]

I have the same problem, I just wonder if it is caused by lack of Navidia GPU in my MacBook Pro ?

[svanschalkwyk]

There is no emulator for Nvidia GPUs that I know of. You absolutely require an NVidia GPU.

[o0oke]

All can be solved by money, most of the DL framworks required NV GPU compute capability bigger than 3.0, refer to: https://developer.nvidia.com/cuda-gpus

I meet the same issue today. and I must ask boss to upgrade my computer.. :)

[alexeystrakh]

I have the same error on my Macbook Pro and I was able to rebuild everything with the CPU-only mode. Unfortunately, the solver is really slow using CPU only and it takes hours to complete 1000 train iteration while I'm expecting millions of iterations to be done.

Is it possible to run caffe with GPU enabled mode on my Macbook Pro?

[AminBW]

Hi
I had the same problem in Ubuntu:
cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version

I solved it easily by installing nvidia-modprobe and cuda-driver using apt-get:
sudo apt-get install nvidia-modprobe
sudo apt-get install cuda-driver

hope it will work for you ;)

[HTLife]

@AminBW Do you mean install in the container or the host?

[xiaoyanzhuo]

I got similar problem since I started running a GPU version docker images just using 'docker run...', (sometimes really hard to discover), which should use 'nvidia-docker run...' Hope can help someone who may have the similar problem.

[mahaling]

@xiaoyanzhuo your solution worked for me!.

[PouriaCh]

I had the same problem with "make runtest". I put 'sudo' in the beginning and it's gone!
sudo make runtest

[lleiiell]

@xiaoyanzhuo same to me, try docker run --gpus .. for latest version,

[martinmCGG]

I got similar problem since I started running a GPU version docker images just using 'docker run...', (sometimes really hard to discover), which should use 'nvidia-docker run...' Hope can help someone who may have the similar problem.

This also happens when one tries to run the unit tests in a Dockerfile, executed by docker build, which does not have a GPU...

[zhangxiaoya]

I got similar problem since I started running a GPU version docker images just using 'docker run...', (sometimes really hard to discover), which should use 'nvidia-docker run...' Hope can help someone who may have the similar problem.

This works for me, i use cuda in docker.