Caffe: Optimize build process on debian jessie

[cdluminate]

I'd suggest that adding this script for debian users.

#!/bin/bash
# manipulate header path, before building caffe on debian jessie
# usage:
#1. cd root of caffe
#2. bash <this_script>
#3. build

# transformations :
#  #include "hdf5/serial/hdf5.h" -> #include "hdf5/serial/hdf5.h"
#  #include "hdf5_hl.h" -> #include "hdf5/serial/hdf5_hl.h"

find . -type f -exec sed -i -e 's^"hdf5.h"^"hdf5/serial/hdf5.h"^g' -e 's^"hdf5_hl.h"^"hdf5/serial/hdf5_hl.h"^g' '{}' \;

[shelhamer]

Thanks for posting your one-liner but this seems like it would be better fixed by the right include dirs instead of transforming the source.

[inexxt]

Had the same problem on Ubuntu 15.04, solution worked, thanks.

[oseiskar]

To get the build to pass on Debian Jessie, I had to (in addition to the above)

1. modify INCLUDE_DIRS in Makefile.config

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/

1. create symlinks as instructed here

cd /usr/lib/x86_64-linux-gnu sudo ln -s libhdf5_serial.so.8.0.2 libhdf5.so sudo ln -s libhdf5_serial_hl.so.8.0.2 libhdf5_hl.so

1. Work around #337 by installing g++-4.6 from the Wheezy repo and adding to Makefile.config:

CUSTOM_CXX := g++-4.6

[snj33v]

3.Work around #337 by installing g++-4.6 from the Wheezy repo and adding to Makefile.config:
another way is seperating host compiler using flags(-ccbin) in nvcc
line:341
from
NVCCFLAGS += -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)
to
NVCCFLAGS += -ccbin=g++-4.6 -Xcompiler -fPIC $(COMMON_FLAGS)

[OverStruck]

Thanks for this. It worked in Debian Jessie

[chenyang-charles]

Works on Ubuntu 15.04

[acgtyrant]

@shelhamer But fixing the right include dirs does not solve another problem:

/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5

We have to create two symbolic links manually.

See can't find hdf5.h when build caffe.

[dgeo]

Here: instead of using gcc-4.6, using nvidia-cuda-toolkit-6.5.14 from stretch seems to work also to workaround #337

[javadba]

Is there any more complete summary on how to do this build? After attempting to merge various bits and pieces from this as well as StackOverflow questions, I end up with the following all the same .

LD -o .build_debug/lib/libcaffe.so.1.0.0-rc3
/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5
/usr/bin/ld: cannot find -lcudnn
/usr/bin/ld: cannot find -lcblas
/usr/bin/ld: cannot find -latlas
collect2: error: ld returned 1 exit status
Makefile:554: recipe for target '.build_debug/lib/libcaffe.so.1.0.0-rc3' failed
make: *** [.build_debug/lib/libcaffe.so.1.0.0-rc3] Error 1

I had used apt-get install on the relevant libraries, updated paths and the Makefile.config . Quite a number of steps. There does not seem to be a firm consensus on the correct aproach so there was a certain degree of mix and match.

I am on ubuntu 15.10 with nVidia cuda .

[snj33v]

@javadba, + where ever you have installed cudnn headers and library file

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include /usr/include/hdf5/serial/ /usr/local/cuda/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu 

as for gcc workaroud try to use clang for everthing

Update: Dont follow this anymore.

[PaulBGD]

This worked for me, thanks.

[binary1too]

Don't mess with symlinks! Just edit Makefile:

--- LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5
+++ LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial

[Key-Liu]

This worked for me on Ubuntu 16.04, thanks.

[raditha]

Worked for me on linux mint.Thank you.

[TTy32]

Works on Debian 8.7. Thanks!

[tim-win]

Can we reopen this? Still failing on Ubuntu 16 without manual workarounds.

And update the title- this is not a request for 'optimization', currently the build process as provided is broken on debian based OSes.

[iamsaurabhc]

Update : Works on Ubuntu 16.04 too! Thanks

[jeremy-rutman]

step #1 from @oseiskar works for linux ubuntu 16.04 (actually mint based on that).

[tkondrashov]

your comment appears to be wrong:
# #include "hdf5/serial/hdf5.h" -> #include "hdf5/serial/hdf5.h"

but the solution worked! thank you!

[StrugglingXu]

Hello,when i Build all the targets!
cd gmake
make config=release64
cd ../
error:
==== Building terrainrlUtil (release64) ====
Linking terrainrlUtil
/usr/bin/ld: ../external/jsoncpp/build/debug/src/lib_json/libjsoncpp.a(json_value.cpp.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
../external/jsoncpp/build/debug/src/lib_json/libjsoncpp.a: 无法添加符号: 错误的值
collect2: error: ld returned 1 exit status
terrainrlUtil.make:139: recipe for target '../lib/libterrainrlUtil.so' failed
make[1]: * [../lib/libterrainrlUtil.so] Error 1
Makefile:20: recipe for target 'terrainrlUtil' failed
make: * [terrainrlUtil] Error 2
please help me to solve this question.Thanks

[g4nym3de]

Works on Ubuntu 16.04, thanks!

[Time0o]

These workarounds don't seem to fix the CMake build. Has anyone gotten that to work (in particular on Debian)? I'm stuck because CMake is unable to locate HDF5 even after creating symlinks for header files and shared objects in the "right" locations. I'm starting to think the whole build process has been broken for years on Debian.

[WatsonZhouAnda]

@StrugglingXu
Hi!
I met the same problem as you when I try to "make config=release64" in /gmake file. Have you solved it yet?