Caffe: find_package(Caffe REQUIRED) fails because of 3rdparty libraries for Windows build

[jasjuang]

Your system configuration

Operating system: Windows 10
Compiler: Visual Studio 2015
CUDA version (if applicable): 8.0
CUDNN version (if applicable): Not applicable
BLAS: built in version enabled by -DBLAS=open in cmake
Python or MATLAB version (for pycaffe and matcaffe respectively): 2.7 for Python and no Matlab

Issue summary

find_package(Caffe REQUIRED) fails because of the 3rdparty libraries cannot be located by CMake.

Steps to reproduce

Build and Install Caffe from scratch,

git clone https://github.com/BVLC/caffe cd caffe && mkdir build && cd build git checkout windows cmake -A x64 -DBLAS=open -DCMAKE_INSTALL_PREFIX="C:/Program Files/Caffe" .. msbuild Caffe.sln /p:Configuration=Release msbuild INSTALL.vcxproj /p:Configuration=Release

Now try to use Caffe with your own CMakeLists that looks like

``````
cmake_minimum_required(VERSION 3.7)
project(CaffeExample)

set(PROJECT_SRCS ${PROJECT_SOURCE_DIR}/CaffeExample.cpp)

find_package( Caffe REQUIRED )

add_executable(${PROJECT_NAME} ${PROJECT_SRCS})

target_link_libraries(${PROJECT_NAME} ${Caffe_LIBRARIES})
``````

CaffeExample.cpp can be any minimal code using Caffe. Now if I try to run cmake for my own code,

``````
cd CaffeExample && mkdir build && cd build
cmake -A x64 ..

CMake Error at C:/externallibs/caffe/build/CaffeConfig.cmake:42 (find_package):
Could not find a package configuration file provided by "gflags" with any
of the following names:

gflagsConfig.cmake
gflags-config.cmake

Add the installation prefix of "gflags" to CMAKE_PREFIX_PATH or set
"gflags_DIR" to a directory containing one of the above files. If "gflags"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
CaffeExample/CMakeLists.txt:6 (find_package)

-- Configuring incomplete, errors occurred!
See also "C:/Users/jasju/Desktop/CaffeExample/build/CMakeFiles/CMakeOutput.log".
See also "C:/Users/jasju/Desktop/CaffeExample/build/CMakeFiles/CMakeError.log".
``````

At the line 42 of CaffeConfig.cmake there is a find_package(gflags REQUIRED NO_MODULE), since only config mode is allowed, according to https://cmake.org/cmake/help/v3.0/command/find_package.html, it will only work if gflags happen to at one of the default locations it search for in config mode. But Caffe put the 3rd party libraries into C:\Users\jasju\.caffe\dependencies\libraries_v140_x64_py27_1.1.0\libraries\lib which is impossible to be discovered by CMake. It would be great if this problem can be fixed.

The exact same CMakeLists.txt and CaffeExample.cpp works without a problem on Ubuntu 16.04.

[willyd]

Thanks for the detailed report. Does it work if you add C:\Users\jasju\.caffe\dependencies\libraries_v140_x64_py27_1.1.0\libraries to the CMAKE_PREFIX_PATH variable.

[jasjuang]

@willyd Thanks for the quick reply, CMAKE_PREFIX_PATH is actually C:\Users\jasju\.caffe\dependencies\libraries_v140_x64_py27_1.1.0\libraries by default when I build Caffe

[willyd]

@jasjuang I guess the easiest way would be to use the caffe-builder-config.cmake as a cache initialization parameter like so: cmake ... -C C:\Users\jasju\.caffe\dependencies\libraries_v140_x64_py27_1.1.0\libraries\caffe-builder-config.cmake

[jasjuang]

@willyd Adding the cache initialization works, I can now compile my code with Caffe.

But is it possible to automate this somehow so I don't have to type this long argument? Having to input where caffe-builder-config.cmake is a bit cumbersome and even if I write a bat file for this it will only work on my computer.

[willyd]

All right I'll see what I can do.

[jasjuang]

@willyd I just realized another potential problem by this approach when I delete the build folder and attempt to redo it from scratch, by loading the cache, I am forced to use the 3rd party OpenCV and glog that comes with Caffe, which breaks compilation because I am using the latest OpenCV feature. Is there a way to selectively link to the 3rd party library if only I don't have them myself (like hdf5 gflags etc)?

[willyd]

You can definitely do that but it won't super simple. If you look in the caffe-builder-config.cmake you will see that there is a few set(... CACHE) commands for each library. Just override the value of all these variables (using the cmake-gui for example) with your own, and CMake should find your libraries instead of the prebuilt ones.

[jasjuang]

@willyd Yes I realized that I can override them on cmake-gui and it will still work, but in an ideal situation it would be great if this can be automated. Meaning a simple cmake -A x64 .. for my own repository should work without any hassle when using external libraries. This also makes it possible for me to setup continuous integration.

[willyd]

@jasjuang You can override from the command line for CI purpose. This should have the same effect as using the cmake-gui.

I realize that the current setup is suboptimal for your use case but I only have limited time I can spend on dependencies management and I already spent too much.

I am currently investigating using vcpkg to manage the dependencies as most dependencies (only LevelDB is missing) are available in there. If you think this would be a good alternative for you let me know and I'll try to speed up the progress.

[jasjuang]

@willyd vcpkg looks like the equivalent of apt-get on linux (correct me if I am wrong), it will for sure be helpful because I can selectively install things (hdf5 glfags) from vcpkg and still build OpenCV and glog from source. The only thing is the external libraries from vcpkg needs to have a gflagsConfig.cmake, and it has to be installed in a location that is discoverable by CMake when I am using find_package(gflags REQUIRED)

[jasjuang]

@willyd I played around with vcpkg and I must said this is the FUTURE. All the open source libraries in there are very up to date and has config.cmake file along with it. This alleviates so much pain for installing open source library on windows. I am able to replace all my existing external libraries that I built from source by the ones at vcpkg mostly seamlessly except OpenCV and PCL which I need CUDA support.

I attempted to build Caffe from source using the 3rdparty libs from vcpkg but as you mentioned LevelDB is missing and I am unable to point Caffe to search for 3rdparty libraries at C:\dev\vcpkg\packages instead of C:\Users\jasju\.caffe\dependencies\libraries_v140_x64_py27_1.1.0\libraries despite I add in -DCMAKE_PREFIX_PATH="C:\dev\vcpkg\packages" when I am doing CMake for Caffe.

I absolutely look forward to your change to manage Caffe 3rdparty libraries with vcpkg. Thanks a lot for your great work.

[MVoz]

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Kitware\CMake\Packages\Caffe]
"73e5991b457e7f61b2aebd79e199b600"="C:\Users\jasju\.caffe\dependencies\libraries_v140_x64_py27_1.1.0\libraries\"

cmake find *-config.cmake

[willyd]

@jasjuang You can have a look at this branch https://github.com/willyd/caffe/tree/vcpkg

[jasjuang]

@willyd I just tried and it's not able to find snappy, and I already installed snappy in vcpkg.

git clone https://github.com/willyd/caffe/ cd caffe && mkdir build && cd build git checkout vcpkg cmake -A x64 -DCMAKE_PREFIX_PATH="C:\dev\vcpkg\packages" ..

and it shows the below CMake error

CMake Error at C:/Program Files/CMake/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake:137 (message): Could NOT find Snappy (missing: Snappy_INCLUDE_DIR Snappy_LIBRARIES) Call Stack (most recent call first): C:/Program Files/CMake/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake:377 (_FPHSA_FAILURE_MESSAGE) cmake/Modules/FindSnappy.cmake:22 (find_package_handle_standard_args) cmake/Dependencies.cmake:86 (find_package) CMakeLists.txt:80 (include)

[willyd]

If you read the vcpkg docs or this script: https://github.com/willyd/caffe/blob/vcpkg/scripts/build_vcpkg.cmd

You will realize you should use the provided toolchain file. Please note however, that my script disables LevelDB so CMake should not even search for it.

[jasjuang]

@willyd I replaced -DCMAKE_PREFIX_PATH="C:\dev\vcpkg\packages" with -DCMAKE_TOOLCHAIN_FILE:FILEPATH=C:\dev\vcpkg\scripts\buildsystems\vcpkg.cmake and the same error still shows up. What exactly does the CMAKE_TOOLCHAIN_FILE do?

[MVoz]

cmake -DCMAKE_PREFIX_PATH= ...\vcpkg\packages\glog_x86-windows\ .. \share\glog // glog-config.cmake
https://cmake.org/cmake/help/v3.3/command/find_package.html#command:find_package
find_package()

[willyd]

@jasjuang A quick update: I have a caffe vcpkg port working at https://github.com/willyd/vcpkg/tree/caffe and I updated my caffe vcpkg branch. Let me know if you can successfully install the caffe port using vcpkg.

[jasjuang]

@willyd There seems to be a hash issue

``````
C:dev\test\vcpkg>vcpkg install caffe
The following packages will be built and installed:
caffe:x64-windows
Building package caffe:x64-windows...
-- CURRENT_INSTALLED_DIR=C:/dev/test/vcpkg/installed/x64-windows
-- DOWNLOADS=C:/dev/test/vcpkg/downloads
-- CURRENT_PACKAGES_DIR=C:/dev/test/vcpkg/packages/caffe_x64-windows
-- CURRENT_BUILDTREES_DIR=C:/dev/test/vcpkg/buildtrees/caffe
-- CURRENT_PORT_DIR=C:/dev/test/vcpkg/ports/caffe/.
-- Downloading https://github.com/willyd/caffe/archive/vcpkg.zip...
-- Downloading https://github.com/willyd/caffe/archive/vcpkg.zip... OK
-- Testing integrity of downloaded file...
CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:47 (message):

File does not have expected hash:

      File path: [ C:/dev/test/vcpkg/downloads/caffe-vcpkg.zip ]
  Expected hash: [ 89b8ecb4dce2488c326342d4a35d9bed6e8a10e582294c269da6f3d363086f1d5c986a21a9b38dd63ca0d0ab325be6a8361df8740e850ad05ff6dab7706d729e ]
    Actual hash: [ f55e9921c90e183a8ad65f2afc32af5bef9be5422c20f261223a2e26203d8a045fb537b993e45e566d5b7f1eacd97d01cf48e643753130529d465716d21e378b ]

The file may be corrupted.

Call Stack (most recent call first):
scripts/cmake/vcpkg_download_distfile.cmake:87 (test_hash)
ports/caffe/portfile.cmake:26 (vcpkg_download_distfile)
scripts/ports.cmake:73 (include)

Error: Building package caffe:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: caffe:x64-windows
Vcpkg version: 0.0.81-e486fcea60df19808716e8cfc8b0147989b6aa74

Additionally, attach any relevant sections from the log files above.

C:dev\test\vcpkg>
``````

I manually removed C:/dev/test/vcpkg/downloads/caffe-vcpkg.zip and re-install it the same error message appears.

[willyd]

@jasjuang Ooops. I forgot to update the hash after I pushed some more changes. Sorry about that.

When the hash is good the port builds fine but the library does not work. I think I fixed the problem but I need to do more tests. I'll keep you posted.

[ras0219-msft]

@jasjuang You might need to clear out the build tree and re-create the whole thing using the CMAKE_TOOLCHAIN_FILE -- a lot of cached variables need to change if you originally created it without the toolchain.

@willyd You might have a more comfortable dev workflow using the recently added vcpkg_from_github()[1]:

vcpkg_from_github(
    OUT_SOURCE_PATH SOURCE_PATH
    REPO willyd/caffe
    HEAD_REF vcpkg
)

This will pull a completely fresh archive from the vcpkg branch whenever you do vcpkg install caffe --head. It also automatically performs the unpack and exports the appropriate folder to the SOURCE_PATH variable.

[1] https://vcpkg.readthedocs.io/en/latest/maintainers/vcpkg_from_github/

[willyd]

@ras0219-msft Thanks this will surely be helpful!

[jasjuang]

@willyd Recently someone did a port for leveldb in vcpkg and it got merged recently. Do you think it's a good time to revisit this?

[willyd]

@jasjuang Thanks for letting me know. I will try to free up some time to get this done soon.

[willyd]

@jasjuang

Can I kindly request your help to test my vcpkg port:

With VS 2017 and CUDA 9 installed open a cmd prompt and type:

git clone https://github.com/willyd/vcpkg.git
cd vcpkg
git checkout caffe
bootstrap-vcpkg.bat
# for a CPU only build
vcpkg install caffe --triplet x64-windows
# for a GPU build
vcpkg install caffe[cuda] --triplet x64-windows --featurepackages

Please let me know if you can sucessfully build caffe using this.

[jasjuang]

@willyd I copied your caffe port folder onto my vcpkg and did a few tests.

vcpkg install caffe --> successful
vcpkg install caffe[cuda] --featurepackages --> successful (I am on latest CUDA 9.1)
vcpkg install caffe[*] --featurepackages --> failed because mkl port does not exist

If I manually enable OpenCV, change WITH_OPENCV and USE_OPENCV to both on,

vcpkg install caffe[cuda] --featurepackages --> successful

If I manually enable leveldb, change USE_LEVELDB to on,

vcpkg install caffe[cuda] --featurepackages --> failed

Thanks for the great work. I think you can add opencv as another feature package, and if mkl and leveldb can be a quick fix that will be great too, if not I say still submit a PR because this port would save a lot of people's time who is trying to compile caffe on windows already!

[willyd]

Thanks @jasjuang for the quick response.

vcpkg install caffe --> successful
vcpkg install caffe[cuda] --featurepackages --> successful (I am on latest CUDA 9.1)

Great!

vcpkg install caffe[*] --featurepackages --> failed because mkl port does not exist

I did not want to create an mkl port, but one like the cuda port should probably be created. In the meantime I just display an error message and fail the build so it works as intended.

If I manually enable OpenCV, change WITH_OPENCV and USE_OPENCV to both on,

vcpkg install caffe[cuda] --featurepackages --> successful

Good to know. Will do.

If I manually enable leveldb, change USE_LEVELDB to both on,

vcpkg install caffe[cuda] --featurepackages --> failed

Most likely fails since leveldb depends on snappy on Linux and does not in vcpkg.

[jasjuang]

@willyd There is actually snappy in vcpkg, but when I take a closer look on the leveldb port in vcpkg, I realized it's not the original one from google, it's actually a fork that is currently being used for bitcoin, so I guess that's probably the reason that it's failing.

[SupunArunoda]

@willyd
The below error occur when executing the command

Could not find a configuration file for package "HDF5" that is compatible
with requested version "".

The following configuration files were considered but not accepted:

C:/Users/supun/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/cmake/hdf5-config.cmake, version: 1.8.16 (64bit)