Caffe: Set mode cpu fails?

[shayo]

Hi,

Installed Caffe Yesterday. Everything works fine if I run things on a machine with gpu.
However, when I try to run things on a machine without gpu, I get the following error:

import caffe
caffe.set_mode_cpu()

WARNING: Logging before InitGoogleLogging() is written to STDERR
E1111 11:05:00.244045 15828 common.cpp:104] Cannot create Cublas handle. Cublas won't be available.
E1111 11:05:00.250761 15828 common.cpp:111] Cannot create Curand generator. Curand won't be available.

Then, when trying to load the reference model, python crashes:
net = caffe.Net(proto_file, model_file, caffe.TEST)
I1111 11:04:24.878739 10921 net.cpp:49] Initializing net from parameters:
....
I1111 11:04:24.879976 10921 net.cpp:106] Creating Layer conv1
I1111 11:04:24.879987 10921 net.cpp:454] conv1 <- data
I1111 11:04:24.880014 10921 net.cpp:411] conv1 -> conv1
F1111 11:04:28.563159 10921 cudnn_conv_layer.cpp:52] Check failed: error == cudaSuccess (38 vs. 0) no CUDA-capable device is detected
* Check failure stack trace: *
Aborted

[shayo]

OK. This is related to the CUDNN flag (which I set to 1)
I thought that CUDNN = 1 will just add GPU support if I switch to gpu mode, but it looks like adding this flag prevents from running in CPU mode altogether.

[SilverWaveGL]

You can specify explicitly in the conv layer param definitions to use the non-cudnn layer version:
under convolution_param add "engine: CAFFE"

Alternatively, you can change the caffe code: GetConvolutionLayer() to select ConvolutionParameter_Engine_CUDNN only if "Caffe::mode() == Caffe::GPU"
though you need to apply the same fix to similar functions in layer_factory.cpp

[hberntsen]

I encountered a similar issue, I was using Caffe in CPU mode with a GPU available that did not have sufficient memory available. This resulted in errors like
F0325 10:33:59.589546 13150 cudnn_relu_layer.cpp:13] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR

The layer factory should not attempt to use the cuDNN engine in CPU only situations. So: If the engine is set to default, only consider the cuDNN engine in GPU mode.

[rddesmond]

The problem here is that the LayerSetup for CuDNN does GPU things and does not respect the mode flag. Some fixes:

1. The quick fix is to make it so that the CuDNN layers are not instantiated if the mode is CPU. This can be done either in the layer_factory or by modifing the model (see https://github.com/BVLC/caffe/issues/3317#issuecomment-157423852 above).
2. The changes to the layer_factory in 1, but also with a specific runtime flag which disables/enables CuDNN support. This would allow for fallback from a CuDNN-enabled GPU to a CUDA-only one (which can't be done at runtime right now.)
3. Another quick fix is to make it so that the CuDNN layers have guards around the initialization to make it avoid the GPU parts. This doesn't seem any better than 1, though (and is more work). It makes the layer work in CPU mode, but it does the exact same thing as the CAFFE layer. (see https://github.com/BVLC/caffe/issues/3953#issuecomment-206625346)
4. The crux of the issue to me is that the CuDNN engine, while using the normal GPU/CPU split in Layer, has GPU work in the setup and thus can't be used in a CPU-only environment. (see my proposal in https://github.com/BVLC/caffe/issues/4187)

Considering my proposal in 4, I'm inclinded to create a PR for option 1 as a quick fix now and ignore that the CuDNN layers don't support switching to CPU mode.

[jeremy-rutman]

hear hear, i've hit this error as well and it is rather unexpected and confusing to get gpu errors seeing as how I set mode cpu . the issue is closed but the (imho bad) behavior persists

[dashesy]

Why is the PR closed? this looks like a legit bug. CPU mode should not try to use CUDNN if it caffe is compiled USE_CUDNN

[rddesmond]

@dashesy The PR that was closed was a fix that was applied to a fork of BVLC's code. Nothing has been updated on the BVLC master.

[yezhengli-Mr9]

Any tutorial to switch to CPU-only caffe?caffe.set_mode_cpu() seems not enough. I am studying another problem anyway.