Caffe: when can caffe support visual studio 2017

Created on 27 May 2017  ยท  31Comments  ยท  Source: BVLC/caffe

Please use the caffe-users list for usage, installation, or modeling questions, or other requests for help.
_Do not post such requests to Issues._ Doing so interferes with the development of Caffe.

Please read the guidelines for contributing before submitting this issue.

Issue summary

Steps to reproduce

If you are having difficulty building Caffe or training a model, please ask the caffe-users mailing list. If you are reporting a build error that seems to be due to a bug in Caffe, please attach your build configuration (either Makefile.config or CMakeCache.txt) and the output of the make (or cmake) command.

Your system configuration

Operating system: windows 10 creaters update
Compiler: visual studio 2017
CUDA version (if applicable): 8.0
CUDNN version (if applicable): 6.0
BLAS:
Python or MATLAB version (for pycaffe and matcaffe respectively): python 3.5

windows
Source
picture WenmuZhou

Most helpful comment

@mjahanifar

If you want to help can you try my vcpkg port:

With VS 2017 and CUDA 9 installed open a cmd prompt and type:

git clone https://github.com/willyd/vcpkg.git
cd vcpkg
git checkout caffe
bootstrap-vcpkg.bat
# for a CPU only build
vcpkg install caffe --triplet x64-windows
# for a GPU build
vcpkg install caffe[cuda] --triplet x64-windows --featurepackages

This won't build the python bindings, I will include them later.

picture willyd on 14 Dec 2017
👍4 ❤1

All 31 comments

As soon as CUDA 9 is out we will start to support VS 2017.

willyd picture willyd on 28 May 2017
👍3

@willyd
CUDA 9 is out! Can we expect a windows Caffe compatible with Visual Studio 2107 and CUDA 9 anytime soon?

mjahanifar picture mjahanifar on 2 Dec 2017

@mjahanifar It is on my TODO list. I will try to allocate some time for this in the next weeks. Watch this issue I will post again here when done.

willyd picture willyd on 4 Dec 2017
👍3

@mjahanifar

If you want to help can you try my vcpkg port:

With VS 2017 and CUDA 9 installed open a cmd prompt and type:

git clone https://github.com/willyd/vcpkg.git
cd vcpkg
git checkout caffe
bootstrap-vcpkg.bat
# for a CPU only build
vcpkg install caffe --triplet x64-windows
# for a GPU build
vcpkg install caffe[cuda] --triplet x64-windows --featurepackages

This won't build the python bindings, I will include them later.

willyd picture willyd on 14 Dec 2017
👍4 ❤1

@willyd I try your way on windows 10 64bit and VS2017 15.5.2, with CUDA 9.x and cuDNN 7.0.5.

I think you should show your environment for CUDA and VS2017.
Under my environment, there have some errors and I take a long time to deal with them.

  1. "sm 20 and 21 not support" error.
    A: This is easy, goto <your path>\vcpkg\buildtrees\caffe\src\caffe-cdf0f3185d076a894410720a2d806c250daca7a6\cmake, and open "Cuda.cmake". Just Modiy the list like this is all right, "set(Caffe_known_gpu_archs "30 35 50 60 61")". If not when do 'vcpkg install caffe[cuda] --triplet x64-windows --featurepackages' will get nvcc error and not build caffe project. The error for which you can see the log and make sure what happen.

  2. "error -- unsupported Microsoft Visual Studio version! Only the versions 2012, 2013, 2015 and 2017 are supported!" compile-time error.
    A: The CUDA bug here. You can see <your path>/NVIDIA GPU Computing Toolkit\CUDA\v9.<0/1>\include\crt\host_config.h the 133th line:#if _MSC_VER < 1600 || _MSC_VER > 1911, change the 1911 to 1912.

  3. "expression must have a constant value" (for vs2017 toolset.)
    A: This is painful issue and make me take more time to deal with it. The offical blog and this blog save my life. However, this can not enable for cmake, and I'm a beginner for vcpkg. Use a not best way to resolve this. I don't know how to set VS toolset version for vcpkg and there are must have the better way than my way.
    When you update VS toolset to 14.11, like official blog, you will get some file here: <your path>\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\14.11, then backup the dealut files in parent dir "Build". Copy "14.11" dir files to "Build" dir.
    Rename:
    Microsoft.VCToolsVersion.14.11.props -> Microsoft.VCToolsVersion.default.props
    Microsoft.VCToolsVersion.14.11.txt -> Microsoft.VCRedistVersion.default.txt
    Microsoft.VCToolsVersion.14.11.txt -> Microsoft.VCToolsVersion.default.txt
    Then, you can not edit msBuild for VS 2017.

from @FrisaSZ

I installed v140 toolset through vs2017 installer and added set(VCPKG_PLATFORM_TOOLSET v140) to vcpkg/triplets/x64-windows.cmake. This method avoids modifying the default toolset of vs2017.

The better way than my way. ๐Ÿ‘

  1. "CUDA Capability is 9.1" (equal the CUDA version.)
    A. I also do some modifing for Cuda.cmake, and found __nvcc_out have some warning line, the warning include nvcc absolute path, so the capability parse to 9.1.
    change this lineexecute_process(COMMAND "${CUDA_NVCC_EXECUTABLE}" "--run" "${__cufile}" to execute_process(COMMAND "${CUDA_NVCC_EXECUTABLE} ${CUDA_NVCC_FLAGS}" "--run" "${__cufile}"
    The error will disappearance.

And then the vcpkg install caffe[cuda] --triplet x64-windows --featurepackages running well and I get

Installing package caffe[core,cuda]:x64-windows... done
Elapsed time for package caffe:x64-windows: 21.46 min

Total elapsed time: 21.46 min

The package caffe:x64-windows provides CMake targets:

    find_package(Caffe REQUIRED)
    target_link_libraries(main PRIVATE caffe)

Hopeful, the above content can help others.

xiaoweiChen picture xiaoweiChen on 30 Dec 2017
👍2

@xiaoweiChen Thanks for the detailed error report. I will see if I can reproduce 1. and 4. and fix them in the caffe source code. I cannot do much about 2. and 3. but I will document that VS 2017 15.5 does not work in the install procedure. You may want to downgrade to 15.4 until NVIDIA release a CUDA version that supports this version of VS. (see https://www.visualstudio.com/en-us/productinfo/installing-an-earlier-release-of-vs2017)

willyd picture willyd on 10 Jan 2018

I build caffe using my own libs (opencv 3.4.0-dev glog gflags protobuf ) using vs 2017 version 15.5.4 but without CUDA. You can find my caffe branch here

I can run caffe using this sample

I still need to modify VS solution because release and debug lib are not known (opencv_world)

LaurentBerger picture LaurentBerger on 18 Feb 2018

@willyd My new caffe branch could be a potential fix for compiler incompatibility with CUDA, since it has no kernels inlined in C++ code. Something to discuss for the future?

naibaf7 picture naibaf7 on 18 Feb 2018

Problem with cmake... i try the version of your repository with error. After without cmake the same result. The first version had a more recent version.
Regards

JavierRuano picture JavierRuano on 19 Feb 2018

@LaurentBerger @LupinDefense My windows branch now works with both VS 2015 and VS 2017. Of course with CUDA you have to use VS 2017 15.2 or lower as stated in the README. I will land those changes on the BVLC windows branch this week.

willyd picture willyd on 20 Feb 2018

@naibaf7 I am not sure I completely understand what you are proposing here.

The incompatibility between VS 2017 and CUDA comes from the fact that NVIDIA adds support for newer compilers on new releases of the CUDA toolkit and Microsoft broke the compatibility between VS 2017 and CUDA 9.x with their latest update of the compiler.

willyd picture willyd on 20 Feb 2018

@willyd My branch now compiles all CUDA kernels at runtime (NVRTC, which I believe uses LLVM/Clang), and for the currently selected GPU's architecture, instead of statically when building Caffe (with fixed architectures selected at compile-time). The kernels are then cached in an SQLITE3 database for future retrieval.
The implications of this are that it does not matter what host-compiler you use to compile Caffe. Any C++11 compiler will do, regardless of what compatibility nVidia CUDA currently has.

I think that is a good way forward, taking away a huge burden on the user and maintainers when compiling Caffe. It also allows just upgrading the GPU without having to recompile Caffe in order to use the newest architecture.

naibaf7 picture naibaf7 on 20 Feb 2018

@willyd Thanks for your work. I couldn't build your fork of vcpkg (there was a ssl error), so I copied your caffe port to the official fork and it worked.
@xiaoweiChen Thanks for your solutions. I met all the 4 errors you mentioned. For error 3, I installed v140 toolset through vs2017 installer and added set(VCPKG_PLATFORM_TOOLSET v140) to vcpkg/triplets/x64-windows.cmake. This method avoids modifying the default toolset of vs2017.

FrisaSZ picture FrisaSZ on 26 Feb 2018

@willyd Why not remove the vs 2017 support from vcpkg.cmake?
My error comes with CUDA 9.0, because it needs Boost 1.65, if i use vs 2015, which is the problem?
How to configure cmake to do it?
if i discard the integration with vs2017, how to configure anaconda with pycaffe and CUDA 9.0?
if is there a solution with cmake-gui or cmake/vcpkg_copy_tool_dependencies.cmake? or without vcpkg in
Dependencies.cmake ( for example find_package(Boost 1.46 COMPONENTS python) this line could modify to Boost 1.65)?
I ask before here
https://github.com/BVLC/caffe/issues/6272

Thanks in advance.
Javier Ruano

JavierRuano picture JavierRuano on 4 Mar 2018

@JavierRuano - having similar issues (we want to use different compilers and libraries when building Caffe for performance tuning), we provided Collective Knowledge python wrappers to detect installed compilers and dependencies, and then let you choose them interactively. CMake is then automatically invoked with correct -D vars (paths, versions, etc).

If it's of interest, you can try it as following (I just tested the latest Caffe windows branch with Windows 10, VS 2015, CUDA 9.0.176, cuDNN 7.0.4, Boost 1.65.1, Python 3.6.3):

$ pip install ck
$ ck pull repo --url=https://github.com/dividiti/ck-caffe
$ ck install package:lib-caffe-bvlc-master-cudnn-universal --env.CAFFE_BUILD_PYTHON=ON

You can then use CK virtual env to test your installation (in fact, you can install and use multiple versions of Caffe like that):

$ ck show env
$ ck virtual env --tags=lib,caffe

$ python
$ > import caffe

or you can try classification example via CK:

$ ck run program:caffe
(select classification)

Hope it's of any help ...

gfursin picture gfursin on 11 Mar 2018

@willyd
Spent all week trying to install caffe with CUDA 9.1. Finally, figured out that I need boost 1.65+ version. So is it possible to update prebuilt dependencies with corresponding version of boost? I'm using Visual Studio 2015.

@gfursin Greetings from MIPT;). How can I manually set my Python and Visual studio (I have them not within the system disk) installation in CK? I can't find information about it in the official docs.
Also ck seems like something that I've been searching for. If it's also possible to patch caffe (change source code) and rebuild in the same way, then it's perfect.

zzzzzzzzzzzzz picture zzzzzzzzzzzzz on 30 Mar 2018

Hi @zzzzzzzzzzzzz,

How is it going in MIPT ;) ?

You can manually detect Python and Visual Studio in a non-system directory via CK as following:

$ ck detect soft:compiler.python --search_dirs={NON SYSTEM DIR}
$ ck detect soft:compiler.microsoft --search_dirs={NON SYSTEM DIR}

Please tell me if it works for you

You can see extra options as following:

$ ck detect soft --help

Yes, you can patch and install CUDA Caffe with above deps as following (we actually do it from time to time):

$ ck install package:lib-caffe-bvlc-master-cudnn-universal

After CK builds and installs all missing dependencies, downloads Caffe from GitHub and starts building it,
interrupt it, patch Caffe (it will be in %USER%\CK-TOOLS directory) and do the following:

$ ck install package:lib-caffe-bvlc-master-cudnn-universal --rebuild

In such case, CK will skip downloading and will normally rebuild your patched version.

After it's done, you can test it using CK virtual env:

$ ck virtual env:{environment UID reported after successful Caffe build}

If something is not working, please don't hesitate to tell us. We have a growing community using CK (ACM is integrating CK with the ACM Digital Library and we use it in our ReQuEST tournaments to optimize deep learning algorithms in a reproducible way) so any feedback is useful. I also add our CK gurus to CC: @psyhtest, @ens-lg4 and @fvella ...

By the way, I just downloaded CUDA 9.1.85 with cuDNN 7.1.2 and successfully compiled Caffe via CK out of the box with Microsoft compiler v19.00.24215.1 and the following dependencies:

  Resolving software dependencies ...

*** Dependency 1 = python (Python for interfacing):
    Resolved. CK environment UID = a3f5485ab9914428 (detected version 3.6.3)
*** Dependency 2 = cmake (cmake):
    Resolved. CK environment UID = 2f423bbf7481b04e (detected version 3.10.2)
*** Dependency 3 = lib-boost (Boost library):
    Resolved. CK environment UID = 28608bcd44472b1a (version 1.65.1)
*** Dependency 4 = lib-gflags (GFlags library):
    Resolved. CK environment UID = 8ea848959503f379 (version 2.2.0)
*** Dependency 5 = lib-glog (GLog library):
    Resolved. CK environment UID = af3208845d0a9057 (version 0.3.5-a6a166d)
*** Dependency 6 = compiler (C++ compiler):
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.0
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
    Resolved. CK environment UID = 64b81bfa4311512d (detected version 19.00.24215.1)
*** Dependency 7 = compiler-cuda (CUDA compiler >=7.0):
    Resolved. CK environment UID = 86a98122454ab76c (detected version 9.1.85)
*** Dependency 8 = lib-cudnn (CUDA DNN library):
    Resolved. CK environment UID = d4af48fac2179aa0 (version 7.1.2)
*** Dependency 9 = lib-opencv (OpenCV library):
    Resolved. CK environment UID = f223b17af1f4d0bb (version 3.3.0)
*** Dependency 10 = lib-lmdb (LMDB library):
    Resolved. CK environment UID = fb48c9f065f7264f (version 0.9.21-60d5002)
*** Dependency 11 = lib-hdf5 (HDF5 library):
    Resolved. CK environment UID = f3b1979d05a42f29 (version 1.10.1)
*** Dependency 12 = lib-blas (BLAS library):
    Resolved. CK environment UID = 7d3ecc2c956a4119 (version 0.2.20-5dde4e65)
*** Dependency 13 = lib-protobuf-host (ProtoBuf host compiler):
    Resolved. CK environment UID = 990ca7485393ee0b (version 3.5.1)
  -----------------------------------

I indeed used Boost 1.65.1 (you can use our CK package v1.65.1-min-for-caffe which build only minimal functions required by Caffe thus speeding up compilation by about 10x on Windows).

Unfortunately, I still didn't manage to build all above with the latest VS 2017 compiler (19.11.25547), so if you manage to figure out the problem and provide a patch, I can add it to our CK packages!

If you need help, don't hesitate to get in touch here, privately or via our open CK mailing list ...

Have fun and thanks,
Grigori

gfursin picture gfursin on 31 Mar 2018

@gfursin It goes well, we are almost Moscow now=)
Build is corrupting on the BLAS installation. When I tried to build caffe in windows cmd I had an issue like: "." is not internal or external command... So I understood that there are some commands during build like: $./run_something. Then I decided to switch to cygwin console. Building of the library starts and finishes. But make install fails with the following error

    OpenBLAS build complete. (BLAS CBLAS)

    OS               ... WINNT
    Architecture     ... x86_64
    BINARY           ... 64bit
    C compiler       ... GCC  (command line : gcc)
    Library Name     ... libopenblas_piledriverp-r0.2.18.a (Multi threaded; Max num-threads is 4)

    To install the library, you can run "make PREFIX=/path/to/your/installation install".

    โ–’โ–’โ–’โฅฌโ–’ โ–’โ–’ ใค โ–’โ–’โ–’โ–’ โ–’โ–’โ–’โ–’ ใช โ–’โ–’โ–’โ–’โ–’ โ–’โ–’โ–’โ–’.
    /usr/bin/sh: ./getarch: Permission denied
    mingw32-make: *** [config.h] Error 126
    mingw32-make -f Makefile.install install
    mingw32-make[1]: Entering directory 'Q:/Dmitriy/cygwin/home/Admin/CK-TOOLS/lib-openblas- 
    0.2.18- 
    mcl-19.00.24215.1-compiler.gcc-6.3.0-windows-64/src'
    Generating openblas_config.h in Q:\Dmitriyygwin\home\Admin\CK-TOOLS\lib-openblas-0.2.18-mcl- 
    19.00.24215.1-compiler.gcc-6.3.0-windows-64\install/include/usr/bin/sh: 
    Q:DmitriycygwinhomeAdminCK-TOOLSlib-openblas-0.2.18-mcl-19.00.24215.1-compiler.gcc-6.3.0- 
    windows-64install/include/openblas_config.h: No such file or directory
    Makefile.install:23: recipe for target 'install' failed
    mingw32-make[1]: *** [install] Error 1
    mingw32-make[1]: Leaving directory 'Q:/Dmitriy/cygwin/home/Admin/CK-TOOLS/lib-openblas-0.2.18- 
    mcl-19.00.24215.1-compiler.gcc-6.3.0-windows-64/src'
    Makefile:313: recipe for target 'install' failed
    mingw32-make: *** [install] Error 2

    Error: make install failed!
    โ–’โ–’ ใค โ–’โ–’โ–’โ–’ โ–’โ–’โ–’โ–’ ใช โ–’โ–’โ–’โ–’โ–’โ–’ โ–’โ–’โ–’โ–’ โ–’โ–’โ–’โ–’โญฎโ–’โ–’ ไ ฉโ–’โ–’ - err

    Cleaning obj directory ...


    Setting up environment for installed package ...
    (full path = Q:\Dmitriy\cygwin\home\Admin\CK-TOOLS\lib-openblas-0.2.18-mcl-19.00.24215.1- 
    compiler.gcc-6.3.0-windows-64\install/lib/libopenblas.dll.a)

    Software entry found: lib.openblas (252d8364ec25497b)

    -----------------------------------
    Resolving software dependencies ...

    *** Dependency 1 = utils-unix-win (Unix Utils for Windows):

    Resolved. CK environment UID = a266d0a831804f87

    *** Dependency 2 = compiler-perl (Perl):

    Resolved. CK environment UID = 5928136f582be2e0 (detected version 5.24.1)

    *** Dependency 3 = compiler-mingw (C compiler):

    Resolved. CK environment UID = 008bc6b93d10a37b (detected version 6.3.0)

    *** Dependency 4 = compiler (C++ compiler):

    Resolved. CK environment UID = ea5a4477ae52b8ba (detected version 19.00.24215.1)

    *** Dependency 5 = compiler-gfortran (GNU Fortran compiler):

    Resolved. CK environment UID = 008bc6b93d10a37b (detected version 6.3.0)
    -----------------------------------
    CK error: [package] software not found in a specified path (Q:\Dmitriy\cygwin\home\Admin\CK- 
    TOOLS\lib-openblas-0.2.18-mcl-19.00.24215.1-compiler.gcc-6.3.0-windows- 
    64\install/lib/libopenblas.dll.a)!

Also install/lib/ folder conists of some empty folders. As I can see there are also some strange things like Q:\Dmitriycygwin and Q\Dmitriy......../usr/local/bin. Maybe it's better to use another console client? Should I install openblas manually and point ck to it's build?

zzzzzzzzzzzzz picture zzzzzzzzzzzzz on 1 Apr 2018

Hi again @zzzzzzzzzzzzz. Sorry for delay in replying - I was travelling ...

I just checked and it seems that we fixed this problem for CK openblas packages v0.2.19 and 0.2.20 but not in v0.2.18 - sorry about that. May I just ask you to install OpenBLAS 0.20 via CK and tell me if it works on your machine:

$ ck install package --tags=openblas,v0.2.20

I just checked it and it worked fine on my platform:

Cleaning obj directory ...

Setting up environment for installed package ...
  (full path = D:\Work1\CK\ck-repos\local\env\8895285cc7253930\install/lib/libopenblas.dll.a)

  Software entry found: lib.openblas (252d8364ec25497b)
  -----------------------------------
  Resolving software dependencies ...

*** Dependency 1 = utils-unix-win (Unix Utils for Windows):
    Resolved. CK environment UID = 6968c19d614036a1 (detected version 8.26)
*** Dependency 2 = compiler-perl (Perl):
    Resolved. CK environment UID = ad406095c0652db5 (detected version 5.24.1)
*** Dependency 3 = compiler-mingw (C compiler):
    Resolved. CK environment UID = f1bb7044ff80cf50 (detected version 6.3.0)
*** Dependency 4 = compiler-gfortran (GNU Fortran compiler):
    Resolved. CK environment UID = f1bb7044ff80cf50 (detected version 6.3.0)
  -----------------------------------

Detected GIT revision:                 5dde4e65
Detected GIT date time of last commit: Mon Jul 24 12:03:35 2017 +0800

Environment entry updated (8895285cc7253930)!

Recording CK configuration to D:\Work1\CK\ck-repos\local\env\8895285cc7253930\ck-install.json ...

Installation path: D:\Work1\CK\ck-repos\local\env\8895285cc7253930

Installation time: 1737.0867004394531 sec.

C:\Users\fgg>

Also note that make install has some issues at the end of installation, but you can ignore it ... If it works, you can use 0.2.20 with Windows Caffe ... And I will later try to fix CK package for OpenBlas 0.2.18 or will deprecate it (I remember we had various of issues with it on Windows) ...

P.S. Any plans to have metro in MIPT? Then it will be really Moscow ;) ...

gfursin picture gfursin on 5 Apr 2018

Hello, @gfursin!
I tried to use 0.2.20 version of the openblas and had the same issue. But it was in the context of caffe installation, I'll try to install version 0.2.20 manually with the command you specidied above and will write about results here later.

P.S. Actually, the plans are already exist, see https://mipt.ru/news/stantsiya_metro_fiztekh_otkroetsya_v_2021_godu

zzzzzzzzzzzzz picture zzzzzzzzzzzzz on 5 Apr 2018

@zzzzzzzzzzzzz Cool! Thanks! If it still doesn't work, there may be an issue with "Unix Utils for Windows" - maybe wrong version. This package allows you to build openblas from cmd via CK without the need for CygWin or anything else ...

gfursin picture gfursin on 5 Apr 2018

@gfursin It looks like unix utils problem.
C:\Program Files\unixutils\usr\local\wbin\install.exe: invalid option -- p
I found the following inside Makefile.install of the openblas 0.2.20:

ifndef NO_LAPACKE
    @echo Copying LAPACKE header files to $(DESTDIR)$(OPENBLAS_INCLUDE_DIR)
    @-install -pm644 $(NETLIB_LAPACK_DIR)/LAPACKE/include/lapacke.h "$(DESTDIR)$(OPENBLAS_INCLUDE_DIR)/lapacke.h"
    @-install -pm644 $(NETLIB_LAPACK_DIR)/LAPACKE/include/lapacke_config.h "$(DESTDIR)$(OPENBLAS_INCLUDE_DIR)/lapacke_config.h"
    @-install -pm644 $(NETLIB_LAPACK_DIR)/LAPACKE/include/lapacke_mangling_with_flags.h.in "$(DESTDIR)$(OPENBLAS_INCLUDE_DIR)/lapacke_mangling.h"
    @-install -pm644 $(NETLIB_LAPACK_DIR)/LAPACKE/include/lapacke_utils.h "$(DESTDIR)$(OPENBLAS_INCLUDE_DIR)/lapacke_utils.h"
endif

#for install static library
ifndef NO_STATIC
    @echo Copying the static library to $(DESTDIR)$(OPENBLAS_LIBRARY_DIR)
    @install -pm644 $(LIBNAME) "$(DESTDIR)$(OPENBLAS_LIBRARY_DIR)"
    @cd "$(DESTDIR)$(OPENBLAS_LIBRARY_DIR)" ; \
    ln -fs $(LIBNAME) $(LIBPREFIX).$(LIBSUFFIX)
endif

Also I've read --help of unixutils

Usage: C:\Program Files\unixutils\usr\local\wbin\install.exe [OPTION]... SOURCE DEST           (1st format)
  or:  C:\Program Files\unixutils\usr\local\wbin\install.exe [OPTION]... SOURCE... DIRECTORY   (2nd format)
  or:  C:\Program Files\unixutils\usr\local\wbin\install.exe -d [OPTION]... DIRECTORY...       (3rd format)
In first two formats, copy SOURCE to DEST or multiple SOURCE(s) to
DIRECTORY, while setting permission modes and owner/group.  In third
format, make all components of the given DIRECTORY(ies).

  -b, --backup        make backup before removal
  -c                  (ignored)
  -d, --directory     create [leading] directories, mandatory for 3rd format
  -g, --group=GROUP   set group ownership, instead of process' current group
  -m, --mode=MODE     set permission mode (as in chmod), instead of rw-r--r--
  -o, --owner=OWNER   set ownership (super-user only)
  -s, --strip         strip symbol tables, only for 1st and 2nd formats
  -S, --suffix=SUFFIX override the usual backup suffix
  -V, --version-control=WORD   override the usual version control
      --help          display this help and exit
      --version       output version information and exit

The backup suffix is ~, unless set with SIMPLE_BACKUP_SUFFIX.  The
version control may be set with VERSION_CONTROL, values are:

  t, numbered     make numbered backups
  nil, existing   numbered if numbered backups exist, simple otherwise
  never, simple   always make simple backups

Report bugs to [email protected]

As I can see it really doesn't have -p option. So I tried to just substitute corresponding places (-pm to just -m) and run make install. But of course this doesn't work as I am not inside the correct environment (It can't find gcc, make, etc). When I try to rerun ck install package --tags=openblas,v0.2.20, openblas downloads and builds again from the very beginning, so my changes to Makefile.install just don't make sence.
So,
Is there any way to enable multiple environments in ck (to be able to run make install inside openblas installation path)? I have lots of them, but it would be greate if I could run them together:

$ ck show env
Env UID:         Target OS: Bits: Name:                  Version:             Tags:

cf8a202618d116eb windows-64    64 python                 3.6.4                64bits,compiler,host-os-windows-64,lang-python,python,target-os-windows-64,v3,v3.6,v3.6.4
5aba7ea4cb1b600b windows-64    64 cuDNN library          7.1.2                64bits,cuda,cudnn,dnn,host-os-windows-64,lib,target-os-windows-64,v7,v7.1,v7.1.2
43866e91b28dae63 windows-64    64 cmake                  3.10.2               64bits,channel-stable,cmake,host-os-windows-64,target-os-windows-64,tool,v3,v3.10,v3.10.2
a266d0a831804f87 windows-64    64 Unix Utils for Windows                      64bits,host-os-windows-64,target-os-windows-64,unix-win,utils
5928136f582be2e0 windows-64    64 Perl                   5.24.1               64bits,channel-stable,compiler,host-os-windows-64,lang-perl,perl,target-os-windows-64,v5,v5.24,v5.24.1,v5.24.1.1
3c1db4cb7bea758c windows-64    64 OpenCV library         3.3.0                64bits,channel-stable,compiled-by-mcl,compiled-by-mcl-19.00.24215.1,host-os-windows-64,lib,opencv,target-os-windows-64,v3,v3.3,v3.3.0
73a93822244c5d89 windows-64    64 Nvidia CUDA Compiler   9.1.85               64bits,compiler,cuda,host-os-windows-64,lang-c-cuda,lang-cpp-cuda,target-os-windows-64,v9,v9.1,v9.1.85
f88562607fb0c2b6 windows-64    64 Microsoft SDK          7.0A                 64bits,host-os-windows-64,microsoft,sdk,target-os-windows-64,v7,v7.0,windows
ea5a4477ae52b8ba windows-64    64 Microsoft C compiler   19.00.24215.1        64bits,compiler,host-os-windows-64,lang-c,lang-cpp,mcl,microsoft,target-os-windows-64,v19,v19.0,v19.0.24215,v19.0.24215.1
7d9e14b838817c7a windows-64    64 LMDB library           0.9.21-60d5002       64bits,channel-stable,compiled-by-mcl,compiled-by-mcl-19.00.24215.1,host-os-windows-64,lib,lmdb,target-os-windows-64,v0,v0.9,v0.9.21,v0.9.21.0
266412ba0f1efb09 windows-64    64 HDF5 library           1.10.1               64bits,channel-stable,compiled-by-mcl,compiled-by-mcl-19.00.24215.1,hdf5,host-os-windows-64,lib,target-os-windows-64,v1,v1.10,v1.10.1
008bc6b93d10a37b windows-64    64 GNU C compiler         6.3.0                64bits,channel-stable,compiler,fortran,gcc,gfortran,host-os-windows-64,lang-c,lang-cpp,lang-f77,lang-f90,lang-f95,target-os-windows-64,v6,v6.3,v6.3.0
8279172f430aa524 windows-64    64 GLog library           0.3.5-a6a166d        64bits,channel-stable,compiled-by-mcl,compiled-by-mcl-19.00.24215.1,glog,host-os-windows-64,lib,target-os-windows-64,v0,v0.3,v0.3.5,v0.3.5.0
6c813460d8c077e6 windows-64    64 GFlags library         2.2.0                64bits,channel-stable,compiled-by-mcl,compiled-by-mcl-19.00.24215.1,gflags,host-os-windows-64,lib,target-os-windows-64,v2,v2.2,v2.2.0
e262ac4056f24e28 windows-64    64 Boost library          1.65.1-min-for-caffe 64bits,boost,channel-stable,compiled-by-mcl,compiled-by-mcl-19.00.24215.1,compiled-by-unknown_host-compiler,host-os-windows-64,lib,min-for-caffe,target-os-windows-64,v1,v1.65,v1.65.1,v1.65.1-min-for-caffe,v1.65.1.0,v1.65.1.0.0,v1.65.1.0.0.0

As I remember I installed unixutils manually from sourceforge website, maybe I have a wrong version (but I can't find where to look for it's version). Maybe I've chosen the wrong place to download it O_o

P.S. Is there any way to set the specific path to ck installation? If I run cygwin, then ck show env shows packages above, but if I run windows cmd, then ck show env shows CK error: [env] can't find to CK entry "module:env". I think it's because by default it looks for installation inside C:\Users\Admin folder, but I have it inside Q:\Dmitriy\cygwin\home\Admin folder.

zzzzzzzzzzzzz picture zzzzzzzzzzzzz on 8 Apr 2018

Hi again @zzzzzzzzzzzzz .

1) Yes, it's possible to customize CK directories via following environment variables:

  • CK_REPOS - specifies directory where all CK repositories are created (note that you should change it before first CK usage, otherwise you local settings from the local repo will be lost)
  • CK_TOOLS - specifies directory where all packages will be installed
  • CK_ROOT - CK root directory

See https://github.com/ctuning/ck/wiki/Customization for further details.

2) Yes, you can preset multiple environments to test your further installation as following:
```
$ ck virtual env:a266d0a831804f87,008bc6b93d10a37b

gfursin picture gfursin on 10 Apr 2018

Basically, list your above environments separated by comma (above examples presets environment for GCC 6.3.0 and for unix utils.

3) Looks like version of your unix utils is old. I use unix utils from Windows Git client. When I install it, I select an option to install unix utils. Then, if you detect unix utils via CK, you should get an option to select one from Git:

$ ck detect soft --tags=utils,unix-win

On my machine I get the following suggestion: "C:Program Files\Git\usr\bin\expr.exe"

I checked the latest Git from here: https://gitforwindows.org/ and "install.exe" has flag -p ...

May I ask you to check it please. If it will work with OpenBLAS, I think I will just add a CK package with unix utils from this Git release to automate installation and avoid wrong versions ...

4) We actually had a similar need to debug CK packages with CK environment not only on Windows, but also on Linux and MacOS (I add to CC @fvella @psyhtest and @ens-lg4), so I am actually thinking to add a flag --debug to "ck install package" which will open a shell just before building a package but with all CK environment pre-set. In such case you will not need to manually set up all CK environments via "ck virtual env" ... I will tell you when done ...

Good luck ;) !

gfursin picture gfursin on 10 Apr 2018

@zzzzzzzzzzzzz - I just added CK package "utils-unix-win-from-git-2.7.10" which automatically installs Git for Windows and register Unix tools from it - they should normally be compatible to build OpenBLAS. You should first update ck-env repository (with packages):

$ ck pull repo:ck-env

and then you can try to install the new package:

$ ck install package:utils-unix-win-from-git-2.7.10

Then you can try to install openblas:

$ ck install package:lib-openblas-0.2.20-universal

Just select unix utils 2.7.10 if CK gives you a choice ...
Please tell me if it works.

I also added "--debug" flag to "ck install package". For example, if above openBLAS installation fails again, try the following:

$ ck install package:lib-openblas-0.2.20-universal --rebuild --debug

Then just before calling cmake/make, CK will open "cmd" with the preset environment (unix utils, GCC, etc). When you exit it, CK will continue building a package ...

Hope it will be of any use and thanks a lot for providing feedback ...

gfursin picture gfursin on 10 Apr 2018

@zzzzzzzzzzzzz - a small update: I just noticed that my new package "utils-unix-win-from-git-2.7.10" messes up original Git installation on Windows. So I changed it. Now it downloads Git-2.17.0-32-bit.tar.bz2 and simply untars it instead of downloading Git-2.17.0-32-bit.exe and running full Windows installation. You can install an updated version as following:

$ ck pull repo:ck-env
$ ck install package:utils-unix-win-from-git-2.7.10

Sorry for this mix up ...

gfursin picture gfursin on 11 Apr 2018

@willyd I followed your vcpkg method on Win 7. I tried both vcpkg install caffe --triplet x64-windows and vcpkg install caffe[cuda] --triplet x64-windows --featurepackages, but I got the same error consequently :

Error: Unable to install default features because can't find CONTROL for caffe:x64-windows

Do you know what has been wrong?

SeanSyue picture SeanSyue on 17 Jul 2018

Successfully manage to built Caffe for AMD GPU's OpenCL for Windows 10 (Creators Update). Anyone interested, here is my forked repo. Goodluck :) https://github.com/abhiTronix/caffe-opencl-windows-amd

abhiTronix picture abhiTronix on 18 Jul 2018

@willyd I try your way on windows 10 64bit and VS2017 15.5.2, with CUDA 9.x and cuDNN 7.0.5.

I think you should show your environment for CUDA and VS2017.
Under my environment, there have some errors and I take a long time to deal with them.

  1. "sm 20 and 21 not support" error.
    A: This is easy, goto <your path>\vcpkg\buildtrees\caffe\src\caffe-cdf0f3185d076a894410720a2d806c250daca7a6\cmake, and open "Cuda.cmake". Just Modiy the list like this is all right, "set(Caffe_known_gpu_archs "30 35 50 60 61")". If not when do 'vcpkg install caffe[cuda] --triplet x64-windows --featurepackages' will get nvcc error and not build caffe project. The error for which you can see the log and make sure what happen.
  2. "error -- unsupported Microsoft Visual Studio version! Only the versions 2012, 2013, 2015 and 2017 are supported!" compile-time error.
    A: The CUDA bug here. You can see <your path>/NVIDIA GPU Computing Toolkit\CUDA\v9.<0/1>\include\crt\host_config.h the 133th line:#if _MSC_VER < 1600 || _MSC_VER > 1911, change the 1911 to 1912.
  3. "expression must have a constant value" (for vs2017 toolset.)
    A: This is painful issue and make me take more time to deal with it. The offical blog and this blog save my life. However, this can not enable for cmake, and I'm a beginner for vcpkg. Use a not best way to resolve this. I don't know how to set VS toolset version for vcpkg and there are must have the better way than my way.
    When you update VS toolset to 14.11, like official blog, you will get some file here: <your path>\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\14.11, then backup the dealut files in parent dir "Build". Copy "14.11" dir files to "Build" dir.
    Rename:
    Microsoft.VCToolsVersion.14.11.props -> Microsoft.VCToolsVersion.default.props
    Microsoft.VCToolsVersion.14.11.txt -> Microsoft.VCRedistVersion.default.txt
    Microsoft.VCToolsVersion.14.11.txt -> Microsoft.VCToolsVersion.default.txt
    Then, you can not edit msBuild for VS 2017.

from @FrisaSZ

I installed v140 toolset through vs2017 installer and added set(VCPKG_PLATFORM_TOOLSET v140) to vcpkg/triplets/x64-windows.cmake. This method avoids modifying the default toolset of vs2017.

The better way than my way. ๐Ÿ‘

  1. "CUDA Capability is 9.1" (equal the CUDA version.)
    A. I also do some modifing for Cuda.cmake, and found __nvcc_out have some warning line, the warning include nvcc absolute path, so the capability parse to 9.1.
    change this lineexecute_process(COMMAND "${CUDA_NVCC_EXECUTABLE}" "--run" "${__cufile}" to execute_process(COMMAND "${CUDA_NVCC_EXECUTABLE} ${CUDA_NVCC_FLAGS}" "--run" "${__cufile}"
    The error will disappearance.

And then the vcpkg install caffe[cuda] --triplet x64-windows --featurepackages running well and I get

Installing package caffe[core,cuda]:x64-windows... done
Elapsed time for package caffe:x64-windows: 21.46 min

Total elapsed time: 21.46 min

The package caffe:x64-windows provides CMake targets:

    find_package(Caffe REQUIRED)
    target_link_libraries(main PRIVATE caffe)

Hopeful, the above content can help others.

Much thanks for your detailed error reports. Based on your suggestion, I have successfully compiled it using vs2017 with CUDA9.0 and cudnn7.2. Sincerely.

lostsea picture lostsea on 28 Sep 2018

@WenmuZhou I asked it to them, you could see https://developer.nvidia.com/nvidia_bug/2068106

JavierRuano picture JavierRuano on 4 Nov 2018

Great topic.

taosiqin1991 picture taosiqin1991 on 15 Apr 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Ruhjkg picture Ruhjkg  ยท  3Comments
iamhankai picture iamhankai  ยท  3Comments
OpenHero picture OpenHero  ยท  3Comments
erogol picture erogol  ยท  3Comments
malreddysid picture malreddysid  ยท  3Comments