Caffe: Pull request #6346 does not compile - even if verified

[fpvision]

Issue summary

I know this is a build issue that is not supposed to be here, but it relates to the merge verification procedures that did not catch this build error

Merge pull request #6346 from jerryz123/HDF5_config SHA: e15898a

Can not compile in without HDF5 anyway. I suspect you only tried to build it on a target that has HFD5 installed, because in my environment the build fails in 2 places.

I use buildroot to cross compile, Neither my buildhost nor my target has any HDF5 stuff installed.

Steps to reproduce

Build on a host with NO HDF5 references at all !!!!
cmake -DUSE_HDF5=OFF

build will fail

Suggested Solution

in cmake/Dependencies.cmake
lines 45-48 should have been guarded by the USE_HDF5 sentinel
Cmake fails since no HDF5 stuff is available.

In src /caffe/net.cpp
line 774 the function H5Fis_hdf5 is referenced
Should have been guarded by the USE_HDF5 sentinel

[jsigee87]

Hi thanks for filing this issue, as it looks like I am having this same problem. I do actually have hdf5 installed and my INCLUDE_LIB points to it. Could you elaborate on the exact fix, I am not familiar with the concept of a sentinel.

My error-

$ <caffe-dir> make all
CXX src/caffe/net.cpp
src/caffe/net.cpp: In member function ‘void caffe::Net<Dtype>::CopyTrainedLayersFrom(const string&)’:
src/caffe/net.cpp:774:42: error: there are no arguments to ‘H5Fis_hdf5’ that depend on a template parameter, so a declaration of ‘H5Fis_hdf5’ must be available [-fpermissive]
   if (H5Fis_hdf5(trained_filename.c_str())) {
                                          ^
src/caffe/net.cpp:774:42: note: (if you use ‘-fpermissive’, G++ will accept your code, but allowing the use of an undeclared name is deprecated)
src/caffe/net.cpp: In instantiation of ‘void caffe::Net<Dtype>::CopyTrainedLayersFrom(const string&) [with Dtype = float; std::__cxx11::string = std::__cxx11::basic_string<char>]’:
src/caffe/net.cpp:992:1:   required from here
src/caffe/net.cpp:774:17: error: ‘H5Fis_hdf5’ was not declared in this scope
   if (H5Fis_hdf5(trained_filename.c_str())) {
                 ^
src/caffe/net.cpp: In instantiation of ‘void caffe::Net<Dtype>::CopyTrainedLayersFrom(const string&) [with Dtype = double; std::__cxx11::string = std::__cxx11::basic_string<char>]’:
src/caffe/net.cpp:992:1:   required from here
src/caffe/net.cpp:774:17: error: ‘H5Fis_hdf5’ was not declared in this scope
Makefile:591: recipe for target '.build_release/src/caffe/net.o' failed
make: *** [.build_release/src/caffe/net.o] Error 1

[fpvision]

Hi John
One of these days I'll have to figure out how to create a proper pull request for a fix to get it through the right test channels, but I haven't got the time right now, so this is a brief of what I did:

OBS: My great concern is - this patch will make it compile successfully. But I don't know it there are any functional impacts.

what I did was make patch on
_src/caffe/net.cpp Line 772-785_

template <typename Dtype>
void Net<Dtype>::CopyTrainedLayersFrom(const string& trained_filename) {
#ifdef USE_HDF5

  if (H5Fis_hdf5(trained_filename.c_str())) {
    CopyTrainedLayersFromHDF5(trained_filename);
  } else {
    CopyTrainedLayersFromBinaryProto(trained_filename);
  }
#else
  CopyTrainedLayersFromBinaryProto(trained_filename);
#endif

the actual patch that I use:

--- net.cpp
+++ net.cpp
@@ -771,11 +771,17 @@

 template <typename Dtype>
 void Net<Dtype>::CopyTrainedLayersFrom(const string& trained_filename) {
+#ifdef USE_HDF5
+
   if (H5Fis_hdf5(trained_filename.c_str())) {
     CopyTrainedLayersFromHDF5(trained_filename);
   } else {
     CopyTrainedLayersFromBinaryProto(trained_filename);
   }
+#else
+  CopyTrainedLayersFromBinaryProto(trained_filename);
+#endif
+
 }`

If you are using CMAKE
Also in cmake/Dependencies.cmake

 --- cmake/Dependencies.cmake
+++ cmake/Dependencies.cmake
@@ -43,12 +43,13 @@
 include(cmake/ProtoBuf.cmake)

 # ---[ HDF5
-find_package(HDF5 COMPONENTS HL REQUIRED)
-list(APPEND Caffe_INCLUDE_DIRS PUBLIC ${HDF5_INCLUDE_DIRS})
-list(APPEND Caffe_LINKER_LIBS PUBLIC ${HDF5_LIBRARIES} ${HDF5_HL_LIBRARIES})
-
 # This code is taken from https://github.com/sh1r0/caffe-android-lib
 if(USE_HDF5)
+
+  find_package(HDF5 COMPONENTS HL REQUIRED)
+  list(APPEND Caffe_INCLUDE_DIRS PUBLIC ${HDF5_INCLUDE_DIRS})
+  list(APPEND Caffe_LINKER_LIBS PUBLIC ${HDF5_LIBRARIES} ${HDF5_HL_LIBRARIES})
+
   find_package(HDF5 COMPONENTS HL REQUIRED)
   include_directories(SYSTEM ${HDF5_INCLUDE_DIRS} ${HDF5_HL_INCLUDE_DIR})
   list(APPEND Caffe_LINKER_LIBS ${HDF5_LIBRARIES} ${HDF5_HL_LIBRARIES})

[jsigee87]

Awesome thanks! I will dig into this soon and report back if I am still having issues.

You may already know this, but you can make a PR by:
1) Navigating to the repo you want to make a PR for
2) Forking it into your personal GitHub by clicking the fork button
3) Committing the changes to your personal copy
4) On the original repo open a pull request by clicking the new pull request button
5) Click the look across forks option

Thanks again!

[toby5box]

Same problem, so +1, would like this fixed upstream.

[mablem8]

This problem should be filed as an issue if it hasn't already. The fix outlined above allows compilation, but many tests will fail due to "Unknown layer type: HDF5Data" in test_gradient_based_solver.cpp and one in test_upgrade_proto.cpp.

[chumingqian]

Issue summary

I know this is a build issue that is not supposed to be here, but it relates to the merge verification procedures that did not catch this build error

Merge pull request #6346 from jerryz123/HDF5_config SHA: e15898a

Can not compile in without HDF5 anyway. I suspect you only tried to build it on a target that has HFD5 installed, because in my environment the build fails in 2 places.

I use buildroot to cross compile, Neither my buildhost nor my target has any HDF5 stuff installed.

Steps to reproduce

Build on a host with NO HDF5 references at all !!!!
cmake -DUSE_HDF5=OFF

build will fail

Suggested Solution

in cmake/Dependencies.cmake
lines 45-48 should have been guarded by the USE_HDF5 sentinel
Cmake fails since no HDF5 stuff is available.

In src /caffe/net.cpp
line 774 the function H5Fis_hdf5 is referenced
Should have been guarded by the USE_HDF5 sentinel

It seems to be not work. Althrough it can pass "make all ". but when "make runtest " it appears the following wrongs:

[----------] 8 tests from RMSPropSolverTest/1, where TypeParam = caffe::CPUDevice
[ RUN ] RMSPropSolverTest/1.TestRMSPropLeastSquaresUpdateWithRmsDecay
F0618 11:32:47.019492 30089 layer_factory.hpp:81] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: HDF5Data

[gentlebreeze1]

@jsigee87 i meet the same question,how do you solve it?

[galkinvv]

The ugly but simplest way I found to build caffe without hdf5 is adding hack to cmake oprions: -DCMAKE_CXX_FLAGS=-DH5Fis_hdf5=false*bool