Dlib: The "simple CUDA program test" fails on Visual Studio when run from another CMake script

[reunanen]

Just learned that dlib exists. It looks amazing!

I'm trying to build today's version on Visual Studio 2015 Update 3 on Windows 10 Pro.

The basic build process fails to compile the simple CUDA program:

D:\dlib\build>cmake -G "Visual Studio 14 2015 Win64" ..
-- The C compiler identification is MSVC 19.0.24215.1
-- The CXX compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler: D:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
-- Check for working C compiler: D:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: D:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
-- Check for working CXX compiler: D:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- C++11 activated.
-- Searching for BLAS and LAPACK
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v8.0 (found suitable version "8.0", minimum required is "7.5")
-- Looking for cuDNN install...
-- Building a CUDA test project to see if your compiler is compatible with CUDA...
-- *** CUDA was found but your compiler failed to compile a simple CUDA program so dlib isn't going to use CUDA. ***
-- *** cuDNN V5.0 OR GREATER NOT FOUND.  DLIB WILL NOT USE CUDA. ***
-- *** If you have cuDNN then set CMAKE_PREFIX_PATH to include cuDNN's folder.
-- Configuring done
-- Generating done
-- Build files have been written to: D:/dlib/build

Building the CUDA test manually seems to work, however there are warnings. Is it because of these warnings that the result is interpreted as a failure?

D:\dlib\build>cmake -G "Visual Studio 14 2015 Win64" ..\dlib\cmake_utils\test_for_cuda
-- The C compiler identification is MSVC 19.0.24215.1
-- The CXX compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler: D:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
-- Check for working C compiler: D:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: D:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
-- Check for working CXX compiler: D:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v8.0 (found suitable version "8.0", minimum required is "7.5")
-- Configuring done
-- Generating done
-- Build files have been written to: D:/dlib/build

D:\dlib\build>cmake --build .
Microsoft (R) Build Engine version 14.0.25420.1
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 19.12.2016 11:59:08.
Project "D:\dlib\build\ALL_BUILD.vcxproj" on node 1 (default targets).
Project "D:\dlib\build\ALL_BUILD.vcxproj" (1) is building "D:\dlib\build\ZERO_CHECK.vcxproj" (2) on node 1 (default target s).
PrepareForBuild:
  Creating directory "x64\Debug\ZERO_CHECK\".
  Creating directory "x64\Debug\ZERO_CHECK\ZERO_CHECK.tlog\".
InitializeBuildStatus:
  Creating "x64\Debug\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  Checking Build System
  CMake does not need to re-run because D:/dlib/build/CMakeFiles/generate.stamp is up-to-date.
FinalizeBuildStatus:
  Deleting file "x64\Debug\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild".
  Touching "x64\Debug\ZERO_CHECK\ZERO_CHECK.tlog\ZERO_CHECK.lastbuildstate".
Done Building Project "D:\dlib\build\ZERO_CHECK.vcxproj" (default targets).

Project "D:\dlib\build\ALL_BUILD.vcxproj" (1) is building "D:\dlib\build\cuda_test.vcxproj" (3) on node 1 (default targets).
PrepareForBuild:
  Creating directory "cuda_test.dir\Debug\".
  Creating directory "D:\dlib\build\Debug\".
  Creating directory "cuda_test.dir\Debug\cuda_test.tlog\".
InitializeBuildStatus:
  Creating "cuda_test.dir\Debug\cuda_test.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
ComputeCustomBuildOutput:
  Creating directory "D:\dlib\build\CMakeFiles\cuda_test.dir\Debug\".
CustomBuild:
  Building NVCC (Device) object CMakeFiles/cuda_test.dir/Debug/cuda_test_generated_cuda_test.cu.obj
  cuda_test.cu

CUSTOMBUILD : nvcc warning : The -std=c++11 flag is not supported with the configured host compiler. Flag will be ignored.
 [D:\dlib\build\cuda_test.vcxproj]

CUSTOMBUILD : nvcc warning : The -std=c++11 flag is not supported with the configured host compiler. Flag will be ignored.
 [D:\dlib\build\cuda_test.vcxproj]

  cuda_test.cu

  Building Custom Rule D:/dlib/dlib/cmake_utils/test_for_cuda/CMakeLists.txt
  CMake is re-running because D:\dlib\build\CMakeFiles\generate.stamp is out-of-date.
    the file 'D:/dlib/build/CMakeFiles/cuda_test.dir/cuda_test_generated_cuda_test.cu.obj.depend'
    is newer than 'D:\dlib\build\CMakeFiles\generate.stamp.depend'
    result='-1'
  -- Configuring done
  -- Generating done
  -- Build files have been written to: D:/dlib/build
Lib:
  D:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_amd64\Lib.exe /OUT:"D:\dlib\build\Debug\cuda_test.lib" /N
  OLOGO /MACHINE:X64  /machine:x64 D:\dlib\build\CMakeFiles\cuda_test.dir\Debug\cuda_test_generated_cuda_test.cu.obj
  cuda_test.vcxproj -> D:\dlib\build\Debug\cuda_test.lib
FinalizeBuildStatus:
  Deleting file "cuda_test.dir\Debug\cuda_test.tlog\unsuccessfulbuild".
  Touching "cuda_test.dir\Debug\cuda_test.tlog\cuda_test.lastbuildstate".
Done Building Project "D:\dlib\build\cuda_test.vcxproj" (default targets).

PrepareForBuild:
  Creating directory "x64\Debug\ALL_BUILD\".
  Creating directory "x64\Debug\ALL_BUILD\ALL_BUILD.tlog\".
InitializeBuildStatus:
  Creating "x64\Debug\ALL_BUILD\ALL_BUILD.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  Building Custom Rule D:/dlib/dlib/cmake_utils/test_for_cuda/CMakeLists.txt
  CMake does not need to re-run because D:\dlib\build\CMakeFiles\generate.stamp is up-to-date.
FinalizeBuildStatus:
  Deleting file "x64\Debug\ALL_BUILD\ALL_BUILD.tlog\unsuccessfulbuild".
  Touching "x64\Debug\ALL_BUILD\ALL_BUILD.tlog\ALL_BUILD.lastbuildstate".
Done Building Project "D:\dlib\build\ALL_BUILD.vcxproj" (default targets).


Build succeeded.

"D:\dlib\build\ALL_BUILD.vcxproj" (default target) (1) ->
"D:\dlib\build\cuda_test.vcxproj" (default target) (3) ->
(CustomBuild target) ->
  CUSTOMBUILD : nvcc warning : The -std=c++11 flag is not supported with the configured host compiler. Flag will be ignored. [D:\dlib\build\cuda_test.vcxproj]
  CUSTOMBUILD : nvcc warning : The -std=c++11 flag is not supported with the configured host compiler. Flag will be ignored. [D:\dlib\build\cuda_test.vcxproj]

    2 Warning(s)
    0 Error(s)

Time Elapsed 00:00:11.08

Will try to figure this out on my own, but any hints would be greatly appreciated.

[primeMover2011]

2016-12-19 11:06 GMT+01:00 Juha Reunanen [e-mail]:

Build succeeded.

...

0 Error(s)

Visual Studio did actually not fail building the CUDA program.

[reunanen]

@primeMover2011 : Yep, I know. But it looks like the main CMake script still interprets it as a failure (because of the warnings? – just a guess), and therefore disables CUDA; consequently (?), the example programs appear to be running on CPU only.

The title of the issue might be misleading though; will try to improve it.

[reunanen]

Manually bypassing the check for the CUDA test result here appears to help.

[davisking]

I haven't tested it yet but you probably need to remove the -std=c++11 flag from the list of NVCC flags set in dlib/CMakeLists.txt. I recently changed this part of the cmake setup to avoid bugs in cmake's C++11 enabling on mac os x, and prior to that I'm pretty sure the visual studio cuda detection worked correctly.

[reunanen]

In fact it now looks like this has nothing to do with the -std=c++11 warning.

Rather, the problem appears to be this:

  Failed to run D:/dlib/build/dlib/cuda_test_build/$(VCInstallDir)bin (The system cannot find the file specified.


  ).

(For what it's worth, I was able to retrieve the error using the OUTPUT_VARIABLE option of CMake's try_compile.)

So it seems that somehow ${CUDA_HOST_COMPILER} goes wrong. It should be $(VCInstallDir)bin, but for some reason it becomes ${PROJECT_BINARY_DIR}/cuda_test_build/$(VCInstallDir)bin. This could be a CMake issue too (I'm using version 3.7.1).

Indeed the file d:\dlib\build\dlib\cuda_test_build\CMakeCache.txt says:

//Host side compiler used by NVCC
CUDA_HOST_COMPILER:FILEPATH=D:/dlib/build/dlib/cuda_test_build/$(VCInstallDir)bin

When I run the CUDA test separately, this problem is not there.

If I remove "-DCUDA_HOST_COMPILER=${CUDA_HOST_COMPILER}", then for some reason there's no error any longer, and stuff generally seems to work (at least on this level – now getting linker errors, but that's another story I guess).

Why is this line not a problem then? It looks the same, but is related to the cuDNN test.

[davisking]

Hmm. I also tried to compile a bunch of older versions of dlib and all of
them give this error. Those versions did work though. I think this is due
to a cmake update that broke the way cuda or try_compile work.

Also, the cudnn check doesn't compile any .cu files, so that's probably why
it still works.

Also, the only reason the CUDA_HOST_COMPILER option is there is because of
funny behavior on some versions of arch linux (see
https://github.com/davisking/dlib/pull/292). So maybe the solution is to
avoid using that flag when using visual studio.

[reunanen]

Looks like there may now be similar symptoms, when targeting Visual Studio 2017 using CMake 3.8.0.

Workaround: downgrade to Visual Studio 2015.

[davisking]

A lot of users have also reported that Visual Studio 2017 crashes or goes into an infinite loop when compiling some of the DNN files. VC2017's C++11 support seems to have gotten worse since 2015.

[reunanen]

Thank you Davis, good to know. This CUDA/CMake problem doesn't look like a C++ compiler issue, though (but rather like some paths are wrong, or something). At any rate I'm not going to dig any deeper right now, if VC2017 anyway doesn't work very well. The workaround (downgrade to VC2015) is just fine for me, at least for the time being.

[InterruptSpeed]

can confirm this worked:
cmake -G "Visual Studio 14 2015 Win64" -T host=x64 ..

where VS2017 equivalent failed:
cmake -G "Visual Studio 15 2017 Win64" -T host=x64 ..

[davisking]

If you get the very latest VS2017 it should work.

[faboules]

True

[azmathmoosa]

No! It doesn't work!

Latest VS 2017 15.5.3 (Just installed hours ago)
CMAKE 3.9.3
CUDA v9.1
CUDNN 9.1

The C compiler identification is MSVC 19.12.25834.0
The CXX compiler identification is MSVC 19.12.25834.0
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.12.25827/bin/Hostx86/x64/cl.exe
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.12.25827/bin/Hostx86/x64/cl.exe -- works
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.12.25827/bin/Hostx86/x64/cl.exe
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.12.25827/bin/Hostx86/x64/cl.exe -- works
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Detecting CXX compile features
Detecting CXX compile features - done
Looking for sys/types.h
Looking for sys/types.h - found
Looking for stdint.h
Looking for stdint.h - found
Looking for stddef.h
Looking for stddef.h - found
Check size of void*
Check size of void* - done
Enabling SSE2 instructions
Looking for png_create_read_struct
Looking for png_create_read_struct - found
Searching for BLAS and LAPACK
Searching for BLAS and LAPACK
Found Intel MKL BLAS/LAPACK library
Looking for cblas_ddot
Looking for cblas_ddot - found
Looking for sgesv
Looking for sgesv - found
Looking for sgesv_
Looking for sgesv_ - found
Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.1 (found suitable version "9.1", minimum required is "7.5")
Looking for cuDNN install...
Found cuDNN: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.1/lib/x64/cudnn.lib
Building a CUDA test project to see if your compiler is compatible with CUDA...
* CUDA was found but your compiler failed to compile a simple CUDA program so dlib isn't going to use CUDA. *
Disabling CUDA support for dlib. DLIB WILL NOT USE CUDA
C++11 activated.
Configuring done

[davisking]

I mean it compiles now. Older versions of visual studio would crash or hang.

If you want to use CUDA you need to use a version of visual studio supported by NVIDIA, which means visual studio 2015. CUDA doesn't support the very latest version of visual studio 2017.

[azmathmoosa]

Ok. It works if we use toolset v140.
VS15 _2017_ Win 64 -T v140 works correctly.

[ariel415el]

Hi, davisking
you said

...CUDA doesn't support the very latest version of visual studio 2017.

Which version did you mean?
From their site It seems,
that it does.

http://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html

So will Dlib compile with VS2017?

[davisking]

There are multiple versions of visual studio 2017. This issue has nothing to do with dlib. It's a compatibility problem between the visual studio version and cuda. See here: https://github.com/davisking/dlib/issues/1035

[Bienqq]

I have the same problem with Visual Studio 2017 Enterprise. CUDA works with TensorFlow I have CUDA 9.0 and cuDNN installed

CUDA was found but your compiler failed to compile a simple CUDA program so dlib isn't going to use CUDA.

[azmathmoosa]

@Bienqq Make sure your CUDA host compiler flag is pointing to the correct location of cl.exe. The default value is incorrect.
see this

[tigros]

This problem arose for me because of a quote in the path environment variable. If you see "someword was unexpected at this time." when launching vs dev cmd prompt don't take it lightly! Just check the path variable for something odd.

[kraghavk]

@davisking I got it to compile with just VS 2015 Build Tools while having VS 2017 installed too. But I had to hack around a few things.

First, had to make the following change in dlib/CMakeLists.txt

if (NOT MSVC) # see https://github.com/davisking/dlib/issues/363
    list(APPEND CUDA_TEST_CMAKE_FLAGS "-DCUDA_HOST_COMPILER=${CUDA_HOST_COMPILER}")
else()
    list(APPEND CUDA_TEST_CMAKE_FLAGS "-DCUDA_HOST_COMPILER=C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe")
endif()

Note the else part. That took care of the test cuda project the setup tries to build.

But I still had to override the CUDA_HOST_COMPILER during the actual dlib compilation too, which I could pass as --set CUDA_HOST_COMPILER="C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe" to setup.py

It would be great if dlib/CMakeLists.txt could use the option passed to setup.py. If I knew enough about cmake, I would have been really glad to send you a pull request with this fixed :)

Thanks for a really great library! You help us mere mortals achieve great things in computer vision and machine learning.

HTH
Raghu

[davisking]

Thanks for the post, but I don't think this is a great idea. I think the newest version of cuda supports visual studio 2017 now. So if you want to use visual studio 2017 with cuda just get the newest cuda that provides official support for visual studio 2017.

[kraghavk]

@davisking, yeah, the latest CUDA should solve this problem. But I cannot upgrade, because tensorflow :( (sigh)

[davisking]

@kraghavk Ah, dang. Sometimes you just gotta do what you gotta do :)

[kraghavk]

@davisking, once again, just out of curiosity, is it possible to make the dlib/CMakeLists.txt file use the CUDA_HOST_COMPILER variable that we can pass on to setup.py? If you can point me in a direction I could look for, I would be glad to submit a PR with this fix.

Because, I would think that it makes sense to expect that CUDA_HOST_COMPILER variable is respected and used by all cmake files. If passing that variable fixes the issue in one place and does not for the CUDA test project compilation, then it is a bug. No?

[davisking]

Doesn't --set pass the option to cmake? Also, look for appearances of CUDA_HOST_COMPILER in dlib/CMakeLists.txt. You can see what is going on.

[kraghavk]

No! When I tried with --set I still kept getting the compilation error with the test CUDA project. That is the reason why I had to hard code the path to cl.exe in dlib/CMakeLists.txt

[davisking]

Look at the CMakeLists.txt file. I just did and it turns out there is an if statement in there that explicitly forbids passing CUDA_HOST_COMPILER when using visual studio. There is also a link to a github issue that talks about why in detail. Maybe there is a better solution.