Incubator-mxnet: mxnet build error.: g++: error: /home/ubuntu/mxnet/3rdparty/dmlc-core/libdmlc.a: No such file or directory

[kaisark]

Description

mxnet build from source error.
Performed pip install as workaround for now (pip install mxnet-cu80 --pre)

Environment info (Required)

What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.
(dl4cv-mxnetpy27) ubuntu@ip-172-31-0-76:~/mxnet$ python diagnose.py 
----------Python Info----------
('Version      :', '2.7.12')
('Compiler     :', 'GCC 5.4.0 20160609')
('Build        :', ('default', 'Nov 20 2017 18:23:56'))
('Arch         :', ('64bit', 'ELF'))
------------Pip Info-----------
('Version      :', '10.0.1')
('Directory    :', '/home/ubuntu/.virtualenvs/dl4cv-mxnetpy27/local/lib/python2.7/site-packages/pip')
----------MXNet Info-----------
('Version      :', '1.2.0')
('Directory    :', '/home/ubuntu/.virtualenvs/dl4cv-mxnetpy27/local/lib/python2.7/site-packages/mxnet')
('Commit Hash   :', '97511ba943c436492ae044ae0de2046cd89621bf')
----------System Info----------
('Platform     :', 'Linux-4.4.0-1055-aws-x86_64-with-Ubuntu-16.04-xenial')
('system       :', 'Linux')
('node         :', 'ip-172-31-0-76')
('release      :', '4.4.0-1055-aws')
('version      :', '#64-Ubuntu SMP Thu Apr 5 17:06:36 UTC 2018')
----------Hardware Info----------
('machine      :', 'x86_64')
('processor    :', 'x86_64')
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping:              1
CPU MHz:               2699.984
CPU max MHz:           3000.0000
CPU min MHz:           1200.0000
BogoMIPS:              4600.10
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              46080K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single retpoline kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0096 sec, LOAD: 0.3116 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0030 sec, LOAD: 0.0883 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0030 sec, LOAD: 0.3319 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0025 sec, LOAD: 0.0952 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0368 sec, LOAD: 0.1834 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.1932 sec, LOAD: 0.3368 s

Package used (Python/R/Scala/Julia):
Python

For Scala user, please provide:

1. Java version: (java -version)
2. Maven version: (mvn -version)
3. Scala runtime if applicable: (scala -version)

For R user, please provide R sessionInfo():

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio):
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)

MXNet commit hash:
(Paste the output of git rev-parse HEAD here.)
602523e643d315f2293442689000dc41e20ba3d9

Build config:

Error Message:

make[1]: Leaving directory '/home/ubuntu/mxnet/3rdparty/ps-lite'
cd /home/ubuntu/mxnet/3rdparty/dmlc-core; make libdmlc.a USE_SSE=1 config=/home/ubuntu/mxnet/config.mk; cd /home/ubuntu/mxnet
make[1]: Entering directory '/home/ubuntu/mxnet/3rdparty/dmlc-core'
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fopenmp -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=1 -DDMLC_USE_AZURE=0 -msse2 -o s3_filesys.o src/io/s3_filesys.cc
src/io/s3_filesys.cc:4:23: fatal error: curl/curl.h: No such file or directory
compilation terminated.
Makefile:87: recipe for target 's3_filesys.o' failed
make[1]: * [s3_filesys.o] Error 1
make[1]: Leaving directory '/home/ubuntu/mxnet/3rdparty/dmlc-core'
ar crv lib/libmxnet.a
g++ -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/home/ubuntu/mxnet/3rdparty/mshadow/ -I/home/ubuntu/mxnet/3rdparty/dmlc-core/include -fPIC -I/home/ubuntu/mxnet/3rdparty/nnvm/include -I/home/ubuntu/mxnet/3rdparty/dlpack/include -I/home/ubuntu/mxnet/3rdparty/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -mf16c -I/usr/local/cuda/include -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/local/include/opencv -I/usr/local/include -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK -DMSHADOW_USE_CUDNN=1 -DMXNET_USE_DIST_KVSTORE -I/home/ubuntu/mxnet/3rdparty/ps-lite/include -I/home/ubuntu/mxnet/deps/include -I/home/ubuntu/mxnet/3rdparty/cub -DMXNET_ENABLE_CUDA_RTC=1 -I/usr/local/cuda/bin/nvcc/include -DMXNET_USE_NCCL=1 -DMXNET_USE_LIBJPEG_TURBO=0 -shared -o lib/libmxnet.so build/src/operator/nn/mkldnn/mkldnn_convolution.o build/src/operator/nn/mkldnn/mkldnn_concat.o build/src/operator/nn/mkldnn/mkldnn_base.o build/src/operator/nn/mkldnn/mkldnn_act.o build/src/operator/nn/mkldnn/mkldnn_softmax.o build/src/operator/nn/mkldnn/mkldnn_deconvolution.o build/src/operator/nn/mkldnn/mkldnn_copy.o build/src/operator/nn/mkldnn/mkldnn_fully_connected.o build/src/operator/nn/mkldnn/mkldnn_pooling.o build/src/operator/nn/mkldnn/mkldnn_sum.o build/src/operator/nn/cudnn/cudnn_algoreg.o build/src/operator/nn/cudnn/cudnn_batch_norm.o build/src/operator/image/image_random.o build/src/operator/contrib/multibox_target.o build/src/operator/contrib/count_sketch.o build/src/operator/contrib/deformable_psroi_pooling.o build/src/operator/contrib/fft.o build/src/operator/contrib/multibox_prior.o build/src/operator/contrib/ctc_loss.o build/src/operator/contrib/quadratic_op.o build/src/operator/contrib/krprod.o build/src/operator/contrib/multi_proposal.o build/src/operator/contrib/adaptive_avg_pooling.o build/src/operator/contrib/bounding_box.o build/src/operator/contrib/bilinear_resize.o build/src/operator/contrib/proposal.o build/src/operator/contrib/psroi_pooling.o build/src/operator/contrib/deformable_convolution.o build/src/operator/contrib/ifft.o build/src/operator/contrib/multibox_detection.o build/src/operator/random/sample_multinomial_op.o build/src/operator/random/multisample_op.o build/src/operator/random/sample_op.o build/src/operator/random/shuffle_op.o build/src/operator/custom/native_op.o build/src/operator/custom/ndarray_op.o build/src/operator/custom/custom.o build/src/operator/tensor/elemwise_binary_broadcast_op_extended.o build/src/operator/tensor/square_sum.o build/src/operator/tensor/elemwise_binary_op_extended.o build/src/operator/tensor/dot.o build/src/operator/tensor/init_op.o build/src/operator/tensor/elemwise_binary_broadcast_op_basic.o build/src/operator/tensor/broadcast_reduce_op_index.o build/src/operator/tensor/elemwise_binary_op.o build/src/operator/tensor/elemwise_unary_op_basic.o build/src/operator/tensor/sparse_retain.o build/src/operator/tensor/broadcast_reduce_op_value.o build/src/operator/tensor/control_flow_op.o build/src/operator/tensor/elemwise_binary_op_basic.o build/src/operator/tensor/elemwise_sum.o build/src/operator/tensor/elemwise_scatter_op.o build/src/operator/tensor/elemwise_binary_scalar_op_extended.o build/src/operator/tensor/indexing_op.o build/src/operator/tensor/elemwise_binary_broadcast_op_logic.o build/src/operator/tensor/la_op.o build/src/operator/tensor/elemwise_binary_op_logic.o build/src/operator/tensor/elemwise_binary_scalar_op_basic.o build/src/operator/tensor/ordering_op.o build/src/operator/tensor/matrix_op.o build/src/operator/tensor/cast_storage.o build/src/operator/tensor/elemwise_binary_scalar_op_logic.o build/src/operator/tensor/elemwise_unary_op_trig.o build/src/operator/nn/softmax.o build/src/operator/nn/layer_norm.o build/src/operator/nn/deconvolution.o build/src/operator/nn/activation.o build/src/operator/nn/pooling.o build/src/operator/nn/fully_connected.o build/src/operator/nn/convolution.o build/src/operator/nn/lrn.o build/src/operator/nn/upsampling.o build/src/operator/nn/concat.o build/src/operator/nn/batch_norm.o build/src/operator/nn/dropout.o build/src/operator/nn/softmax_activation.o build/src/operator/quantization/requantize.o build/src/operator/quantization/dequantize.o build/src/operator/quantization/quantized_conv.o build/src/operator/quantization/quantize_graph_pass.o build/src/operator/quantization/quantized_flatten.o build/src/operator/quantization/quantized_fully_connected.o build/src/operator/quantization/quantized_pooling.o build/src/operator/quantization/quantize.o build/src/operator/nnpack/nnpack_util.o build/src/executor/inplace_addto_detect_pass.o build/src/executor/graph_executor.o build/src/executor/infer_graph_attr_pass.o build/src/executor/attach_op_execs_pass.o build/src/executor/attach_op_resource_pass.o build/src/io/io.o build/src/io/image_aug_default.o build/src/io/iter_image_det_recordio.o build/src/io/image_io.o build/src/io/image_det_aug_default.o build/src/io/iter_csv.o build/src/io/iter_image_recordio.o build/src/io/iter_mnist.o build/src/io/iter_image_recordio_2.o build/src/io/iter_libsvm.o build/src/common/utils.o build/src/common/rtc.o build/src/nnvm/legacy_op_util.o build/src/nnvm/tvm_bridge.o build/src/nnvm/legacy_json_util.o build/src/imperative/cached_op.o build/src/imperative/imperative.o build/src/ndarray/ndarray_function.o build/src/ndarray/ndarray.o build/src/operator/instance_norm.o build/src/operator/svm_output.o build/src/operator/grid_generator.o build/src/operator/leaky_relu.o build/src/operator/bilinear_sampler.o build/src/operator/rnn.o build/src/operator/identity_attach_KL_sparse_reg.o build/src/operator/convolution_v1.o build/src/operator/swapaxis.o build/src/operator/pad.o build/src/operator/softmax_output.o build/src/operator/make_loss.o build/src/operator/operator.o build/src/operator/operator_tune.o build/src/operator/roi_pooling.o build/src/operator/sequence_last.o build/src/operator/batch_norm_v1.o build/src/operator/loss_binary_op.o build/src/operator/spatial_transformer.o build/src/operator/regression_output.o build/src/operator/l2_normalization.o build/src/operator/crop.o build/src/operator/sequence_reverse.o build/src/operator/correlation.o build/src/operator/optimizer_op.o build/src/operator/slice_channel.o build/src/operator/cross_device_copy.o build/src/operator/pooling_v1.o build/src/operator/operator_util.o build/src/operator/sequence_mask.o build/src/engine/naive_engine.o build/src/engine/openmp.o build/src/engine/threaded_engine_pooled.o build/src/engine/threaded_engine.o build/src/engine/engine.o build/src/engine/threaded_engine_perdevice.o build/src/storage/storage.o build/src/c_api/c_api_symbolic.o build/src/c_api/c_api_profile.o build/src/c_api/c_api_ndarray.o build/src/c_api/c_api_executor.o build/src/c_api/c_predict_api.o build/src/c_api/c_api_function.o build/src/c_api/c_api.o build/src/c_api/c_api_error.o build/src/profiler/profiler.o build/src/profiler/aggregate_stats.o build/src/profiler/vtune.o build/src/kvstore/gradient_compression.o build/src/kvstore/kvstore_utils.o build/src/kvstore/kvstore.o build/src/resource.o build/src/initialize.o /home/ubuntu/mxnet/3rdparty/ps-lite/build/libps.a /home/ubuntu/mxnet/3rdparty/dmlc-core/libdmlc.a build/src/operator/nn/cudnn/cudnn_batch_norm_gpu.o build/src/operator/contrib/ctc_loss_gpu.o build/src/operator/contrib/bilinear_resize_gpu.o build/src/operator/contrib/psroi_pooling_gpu.o build/src/operator/contrib/deformable_convolution_gpu.o build/src/operator/contrib/ifft_gpu.o build/src/operator/contrib/multibox_detection_gpu.o build/src/operator/contrib/adaptive_avg_pooling_gpu.o build/src/operator/contrib/multibox_target_gpu.o build/src/operator/contrib/proposal_gpu.o build/src/operator/contrib/count_sketch_gpu.o build/src/operator/contrib/deformable_psroi_pooling_gpu.o build/src/operator/contrib/fft_gpu.o build/src/operator/contrib/multibox_prior_gpu.o build/src/operator/contrib/quadratic_op_gpu.o build/src/operator/contrib/multi_proposal_gpu.o build/src/operator/contrib/bounding_box_gpu.o build/src/operator/random/shuffle_op_gpu.o build/src/operator/random/sample_multinomial_op_gpu.o build/src/operator/random/multisample_op_gpu.o build/src/operator/random/sample_op_gpu.o build/src/operator/custom/native_op_gpu.o build/src/operator/tensor/indexing_op_gpu.o build/src/operator/tensor/elemwise_binary_op_extended_gpu.o build/src/operator/tensor/elemwise_binary_scalar_op_extended_gpu.o build/src/operator/tensor/ordering_op_gpu.o build/src/operator/tensor/cast_storage_gpu.o build/src/operator/tensor/elemwise_binary_scalar_op_basic_gpu.o build/src/operator/tensor/elemwise_unary_op_trig_gpu.o build/src/operator/tensor/elemwise_binary_broadcast_op_basic_gpu.o build/src/operator/tensor/square_sum_gpu.o build/src/operator/tensor/elemwise_binary_op_basic_gpu.o build/src/operator/tensor/elemwise_sum_gpu.o build/src/operator/tensor/init_op_gpu.o build/src/operator/tensor/elemwise_binary_scalar_op_logic_gpu.o build/src/operator/tensor/dot_gpu.o build/src/operator/tensor/elemwise_unary_op_basic_gpu.o build/src/operator/tensor/matrix_op_gpu.o build/src/operator/tensor/control_flow_op_gpu.o build/src/operator/tensor/broadcast_reduce_op_value_gpu.o build/src/operator/tensor/elemwise_binary_broadcast_op_extended_gpu.o build/src/operator/tensor/broadcast_reduce_op_index_gpu.o build/src/operator/tensor/elemwise_scatter_op_gpu.o build/src/operator/tensor/elemwise_binary_broadcast_op_logic_gpu.o build/src/operator/tensor/la_op_gpu.o build/src/operator/tensor/sparse_retain_gpu.o build/src/operator/tensor/elemwise_binary_op_logic_gpu.o build/src/operator/nn/batch_norm_gpu.o build/src/operator/nn/dropout_gpu.o build/src/operator/nn/softmax_activation_gpu.o build/src/operator/nn/lrn_gpu.o build/src/operator/nn/fully_connected_gpu.o build/src/operator/nn/deconvolution_gpu.o build/src/operator/nn/pooling_gpu.o build/src/operator/nn/softmax_gpu.o build/src/operator/nn/activation_gpu.o build/src/operator/nn/layer_norm_gpu.o build/src/operator/nn/convolution_gpu.o build/src/operator/nn/upsampling_gpu.o build/src/operator/nn/concat_gpu.o build/src/operator/quantization/requantize_gpu.o build/src/operator/quantization/quantize_gpu.o build/src/operator/quantization/dequantize_gpu.o build/src/operator/quantization/quantized_conv_gpu.o build/src/operator/quantization/quantized_flatten_gpu.o build/src/operator/quantization/quantized_fully_connected_gpu.o build/src/operator/quantization/quantized_pooling_gpu.o build/src/common/utils_gpu.o build/src/common/random_generator_gpu.o build/src/ndarray/ndarray_function_gpu.o build/src/operator/svm_output_gpu.o build/src/operator/spatial_transformer_gpu.o build/src/operator/optimizer_op_gpu.o build/src/operator/convolution_v1_gpu.o build/src/operator/instance_norm_gpu.o build/src/operator/pad_gpu.o build/src/operator/slice_channel_gpu.o build/src/operator/correlation_gpu.o build/src/operator/l2_normalization_gpu.o build/src/operator/leaky_relu_gpu.o build/src/operator/rnn_gpu.o build/src/operator/crop_gpu.o build/src/operator/sequence_reverse_gpu.o build/src/operator/identity_attach_KL_sparse_reg_gpu.o build/src/operator/swapaxis_gpu.o build/src/operator/sequence_mask_gpu.o build/src/operator/grid_generator_gpu.o build/src/operator/softmax_output_gpu.o build/src/operator/bilinear_sampler_gpu.o build/src/operator/pooling_v1_gpu.o build/src/operator/roi_pooling_gpu.o build/src/operator/batch_norm_v1_gpu.o build/src/operator/loss_binary_op_gpu.o build/src/operator/make_loss_gpu.o build/src/operator/sequence_last_gpu.o build/src/operator/regression_output_gpu.o build/src/kvstore/gradient_compression_gpu.o build/src/kvstore/kvstore_utils_gpu.o -pthread -lm -lcudart -lcublas -lcurand -lcusolver -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib -lcblas -fopenmp -lrt -lcurl -lssl -lcrypto -L/usr/local/lib -lopencv_stitching -lopencv_superres -lopencv_videostab -lopencv_photo -lopencv_aruco -lopencv_bgsegm -lopencv_bioinspired -lopencv_ccalib -lopencv_dpm -lopencv_face -lopencv_freetype -lopencv_fuzzy -lopencv_hdf -lopencv_img_hash -lopencv_line_descriptor -lopencv_optflow -lopencv_reg -lopencv_rgbd -lopencv_saliency -lopencv_stereo -lopencv_structured_light -lopencv_phase_unwrapping -lopencv_surface_matching -lopencv_tracking -lopencv_datasets -lopencv_text -lopencv_dnn -lopencv_plot -lopencv_ml -lopencv_xfeatures2d -lopencv_shape -lopencv_video -lopencv_ximgproc -lopencv_calib3d -lopencv_features2d -lopencv_highgui -lopencv_videoio -lopencv_flann -lopencv_xobjdetect -lopencv_imgcodecs -lopencv_objdetect -lopencv_xphoto -lopencv_imgproc -lopencv_core -llapack -lcudnn /home/ubuntu/mxnet/deps/lib/libprotobuf-lite.a /home/ubuntu/mxnet/deps/lib/libzmq.a -lcufft -lcuda -lnvrtc -L/usr/local/cuda/lib64/stubs -L/usr/local/cuda/bin/nvcc/lib -lnccl \
-Wl,--whole-archive /home/ubuntu/mxnet/3rdparty/nnvm/lib/libnnvm.a -Wl,--no-whole-archive
g++: error: /home/ubuntu/mxnet/3rdparty/dmlc-core/libdmlc.a: No such file or directory
Makefile:466: recipe for target 'lib/libmxnet.so' failed
make: * [lib/libmxnet.so] Error 1

Minimum reproducible example

(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)

Steps to reproduce

(Paste the commands you ran that produced the error.)

1.git clone --recursive https://github.com/apache/incubator-mxnet mxnet

1. cd mxnet
2. git submodule update --init --recursive
3. Please find config.mk from mxnet/make/, copy to mxnet/, and edit/add these lines:

add CUDA options

echo "USE_CUDA=1" >>config.mk
echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
echo "USE_CUDNN=1" >>config.mk
echo "USE_BLAS=atlas" >> config.mk
echo "USE_DIST_KVSTORE = 1" >>config.mk
echo "USE_S3=1" >>config.mk

1. make -j4

What have you tried to solve it?

1. git submodule update --init --recursive
   2.

[roywei]

@kaisark could you try USE_BLAS=openblas ?
@nswamy Could you help add label Installation? Thanks!

[ctcyang]

I ran into this issue too. When specifying USE_S3=1, an additional dependency of libcurl-dev is required by dmlc-core. Doing sudo apt-get install -y libcurl-dev will install required libraries and the needed header /usr/local/include/curl/curl.h.

[piyushghai]

@kaisark Did the above suggestion help in resolving the issue you are seeing ?
If yes, then feel free to close the issue :)

[tribbloid]

@roywei I enabled:

USE_BLAS=openblas

and encountered the same error. My full config.mk is:

ADD_CFLAGS += -I/usr/local/include/opencv
ADD_LDFLAGS += -L/usr/local/lib
USE_OPENCV=1
USE_OPENMP=1
USE_BLAS=openblas
USE_LAPACK=1
USE_DIST_KVSTORE=1
USE_S3=1

[aaronmarkham]

I was able to get around this error:

First, delete the 3rdparty folder. Then check it out again to get a fresh copy. Then submodule update.

rm -rf 3rdparty/
git checkout 3rdparty
git submodule update --init --recursive
make -j$(nproc)

You might want to lead with a make clean for good measure.

[Rivendile]

I encountered the same error. My config.mk is https://github.com/apache/incubator-mxnet/blob/1.5.0/make/pip/pip_linux_cu100.mk, while ADD_CFLAGS and ADD_LDFLAGS are not set.
I can build MxNet when USE_S3=0. I have tried the above methods, but failed. Any suggestions?