Incubator-mxnet: Build error w/ TVM in NNVM.

[KellenSunderland]

Description

MXNet fails during compilation for users without NVM installed on their local machine.

Environment info (Required)

----------Python Info----------
Version      : 3.6.3
Compiler     : GCC 7.2.0
Build        : ('default', 'Nov 20 2017 20:41:42')
Arch         : ('64bit', '')
----------System Info----------
Platform     : Linux-4.13.0-36-generic-x86_64-with-debian-stretch-sid
system       : Linux
node         : kellen-MS-7B09
release      : 4.13.0-36-generic
version      : #40~16.04.1-Ubuntu SMP Fri Feb 16 23:25:58 UTC 2018
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             1
NUMA node(s):          1
Vendor ID:             AuthenticAMD
CPU family:            23
Model:                 1
Model name:            AMD Ryzen Threadripper 1950X 16-Core Processor
Stepping:              1
CPU MHz:               2200.000
CPU max MHz:           3400,0000
CPU min MHz:           2200,0000
BogoMIPS:              6799.74
Virtualization:        AMD-V
L1d cache:             32K
L1i cache:             64K
L2 cache:              512K
L3 cache:              8192K
NUMA node0 CPU(s):     0-31
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 mwaitx hw_pstate retpoline retpoline_amd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0100 sec, LOAD: 0.7794 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0335 sec, LOAD: 0.1580 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.2177 sec, LOAD: 0.8037 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0414 sec, LOAD: 1.2321 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0222 sec, LOAD: 0.1792 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0292 sec, LOAD: 0.1181 sec.

Package used (Python/R/Scala/Julia):
Cmake

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio):
gcc

MXNet commit hash:
becec31cbba1458429461630a14c47f80d2dfa18

Build config:
mkdir build && cd build && cmake .. && make -j $(nproc)

Error Message:

$ make
-- CMake version '3.5.1' using generator 'Unix Makefiles'
-- CMAKE_BUILD_TYPE is unset, defaulting to Release
-- Detecting Intel(R) MKL: trying mklml_intel
-- Detecting Intel(R) MKL: trying mklml
-- Detecting Intel(R) MKL: trying mkl_rt
CMake Warning at 3rdparty/mkldnn/cmake/MKL.cmake:177 (message):
  Intel(R) MKL not found.  Some performance features may not be available.
  Please run scripts/prepare_mkl.sh to download a minimal set of libraries or
  get a full version from https://software.intel.com/en-us/intel-mkl
Call Stack (most recent call first):
  3rdparty/mkldnn/cmake/OpenMP.cmake:24 (include)
  3rdparty/mkldnn/CMakeLists.txt:57 (include)


-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE) 
-- VTune profiling environment is unset
-- Could NOT find MKL (missing:  MKL_INCLUDE_DIR MKLML_GNU_LIBRARY MKLDNN_LIBRARY) 
--  MKL not found
-- Could NOT find MKL (missing:  MKL_INCLUDE_DIR MKLML_GNU_LIBRARY MKLDNN_LIBRARY) 
-- Found OpenBLAS libraries: /usr/lib/libopenblas.so
-- Found OpenBLAS include: /usr/include
-- CUDA detected: 9.0
-- Found cuDNN (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- Running GPU architecture autodetection
-- Found CUDA arch 7.0
-- Added CUDA NVCC flags for: sm_70
-- Could NOT find Gperftools (missing:  GPERFTOOLS_LIBRARIES GPERFTOOLS_INCLUDE_DIR) 
-- Could NOT find Jemalloc (missing:  JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR) 
--  OpenCV_LIBS=opencv_core;opencv_highgui;opencv_imgproc
-- OpenCV found (/usr/share/OpenCV)
-- LIBOMP: Operating System     -- Linux
-- LIBOMP: Target Architecture  -- x86_64
-- LIBOMP: Build Type           -- Release
-- LIBOMP: OpenMP Version       -- 50
-- LIBOMP: Library Kind         -- SHARED
-- LIBOMP: Library Type         -- normal
-- LIBOMP: Fortran Modules      -- FALSE
-- LIBOMP: Build                -- 20140926
-- LIBOMP: Use Stats-gathering  -- FALSE
-- LIBOMP: Use Debugger-support -- FALSE
-- LIBOMP: Use ITT notify       -- TRUE
-- LIBOMP: Use OMPT-support     -- FALSE
-- LIBOMP: Use Adaptive locks   -- TRUE
-- LIBOMP: Use quad precision   -- TRUE
-- LIBOMP: Use TSAN-support     -- FALSE
-- LIBOMP: Use Hwloc library    -- FALSE
-- LIBOMP: Cannot find llvm-lit.
-- LIBOMP: Please put llvm-lit in your PATH, set LIBOMP_LLVM_LIT_EXECUTABLE to its full path or point OPENMP_LLVM_TOOLS_DIR to its directory
CMake Warning at 3rdparty/openmp/runtime/cmake/LibompUtils.cmake:21 (message):
  LIBOMP: The check-libomp target will not be available!
Call Stack (most recent call first):
  3rdparty/openmp/runtime/test/CMakeLists.txt:62 (libomp_warning_say)


-- Could NOT find Jemalloc (missing:  JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR) 
-- Found cuDNN (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
You have called ADD_LIBRARY for library mxnet without any source files. This typically indicates a problem with your CMakeLists.txt file
-- Configuring done
-- Generating done
-- Build files have been written to: /home/kellen/Development/incubator-mxnet/build
[  2%] Built target dmlc
[  2%] Building CXX object CMakeFiles/mxnet_static.dir/src/nnvm/tvm_bridge.cc.o
/home/kellen/Development/incubator-mxnet/src/nnvm/tvm_bridge.cc:37:37: fatal error: tvm/runtime/packed_func.h: No such file or directory
compilation terminated.
CMakeFiles/mxnet_static.dir/build.make:91354: recipe for target 'CMakeFiles/mxnet_static.dir/src/nnvm/tvm_bridge.cc.o' failed
make[2]: *** [CMakeFiles/mxnet_static.dir/src/nnvm/tvm_bridge.cc.o] Error 1
CMakeFiles/Makefile2:139: recipe for target 'CMakeFiles/mxnet_static.dir/all' failed
make[1]: *** [CMakeFiles/mxnet_static.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2

Possible Work Around

Very exciting that we're integrating TVM, but is it something we should make a configurable feature?
Maybe we can define a USE_TVM and put guards around this include in NNVM.
https://github.com/apache/incubator-mxnet/blob/master/src/nnvm/tvm_bridge.cc#L37

[tqchen]

gits submodule update --init --recursive

[KellenSunderland]

Thanks! @tqchen. Totally missed those.

[tqchen]

This is something a user might miss as well and maybe we should post it on some trouble shooting page.

[aaronmarkham]

Is this something we can catch in the make step (check for the submodules)... early on so you don't get halfway through the build and get this mysterious error?

[KellenSunderland]

My hope is that googling errors with tvm_bridge in them will lead users here.

Another option for managing large source dependencies via cmake would be to use ExternalProjects. These could be configured to fetch source dependencies automatically for users. Something to consider as our dep chain grows.

[ThomasDelteil]

@KellenSunderland compile error brought me here indeed 😄

[mirekphd]

Nope, this did not help:
git submodule update --init --recursive
still getting this error:
include/mxnet/operator.h:33:23: fatal error: nnvm/node.h: No such file or directory

Can anyone produce a working Dockerfile?
@tqchen : "gits" certainly doesn't look OK;)

[tqchen]

@mirekphd try to see if you have folder 3rdparty/tvm/nnvm. do a make clean(remove all cmake related caches) and redo the build

[nchafni]

hitting this error as well, git submodule update --init --recursive doesn't work for me, has this been solved?

[renjiahao0928]

hitting this error as well, git submodule update --init --recursive doesn't work for me, has this been solved?

same error,it doesn't work, Can you tell me how to solve it?

[szha]

@renjiahao0928 could you file a new issue for how you came across the issue? thank you