Dali: cmake failed - nvJPEG

[shl666]

Here is the thing. After I do the cmake step:

$cmake ..

It gives me the error:

-- The CXX compiler identification is GNU 5.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda-8.0 (found version "8.0")
-- root:
CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find NVJPEG (missing: NVJPEG_INCLUDE_DIR NVJPEG_LIBRARY)
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/FindNVJPEG.cmake:32 (find_package_handle_standard_args)
  cmake/Dependencies.cmake:13 (find_package)
  CMakeLists.txt:47 (include)


-- Configuring incomplete, errors occurred!
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeOutput.log".
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeError.log".

I guess the error is about nvJPEG

I downloaded the pre-release version v0.1 and decompressed it. There is no more other instructions on the official website of NVIDIA (https://developer.nvidia.com/nvjpeg)

THX in advance.

[JanuszL]

Hi,
If you extract it anywhere you need to run cmake with NVJPEG_ROOT_DIR argument which value points to dir where you have your nvJPEG.
Other way is to copy nvJPEG libs to /usr/local/lib/ and headers to /usr/local/include/.

[cliffwoolley]

I'm working on a PR that would make NVJPEG_ROOT_DIR (and similar vars for other dependencies) a bit more usable via ccmake. But yes you can already specify that with something like cmake ... -DNVJPEG_ROOT_DIR=/path/to/nvjpeg/ .

[shl666]

THXs for all replies.

I am working on a public server and do not have access to usr/local.

So I tried the first method:

$cmake -DBUILD_LMDB=ON -NVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg ..
-- root:
CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find NVJPEG (missing: NVJPEG_INCLUDE_DIR NVJPEG_LIBRARY)
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/FindNVJPEG.cmake:32 (find_package_handle_standard_args)
  cmake/Dependencies.cmake:13 (find_package)
  CMakeLists.txt:47 (include)


-- Configuring incomplete, errors occurred!
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeOutput.log".
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeError.log".

Unfortunately, it does not work. It looks not reasonable. Or due to I am not familiar with cmake, wrote the command in a wrong way.

[JanuszL]

To pass variable to cmake you need to use -D_VARIABLE_NAME_. So try:
cmake -DBUILD_LMDB=ON -DNVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg ..

[shl666]

@JanuszL Brilliant!!!!

However, some other errors comes out. It seems lack of CMakelist.txt files in _DALI-master/third_party/googletest_, _DALI-master/third_party/benchmark_ and _DALI-master/third_party/pybind11_. And another Unkown error.

Following are details:

$cmake -DBUILD_LMDB=ON -DNVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg ..
-- root: /home/smluo/GithubProject/cuda-linux64-nvjpeg
-- Found NVJPEG: /home/smluo/GithubProject/cuda-linux64-nvjpeg/include
-- Found NVJPEG    (include: /home/smluo/GithubProject/cuda-linux64-nvjpeg/include, library: /home/smluo/GithubProject/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a)
CMake Error at cmake/Dependencies.cmake:59 (add_subdirectory):
  The source directory

    /home/smluo/GithubProject/DALI-master/third_party/googletest

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:47 (include)


CMake Error at cmake/Dependencies.cmake:66 (add_subdirectory):
  The source directory

    /home/smluo/GithubProject/DALI-master/third_party/benchmark

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:47 (include)


-- Found JPEG_TURBO: /home/smluo/anaconda3/include
-- Found JPEG_TURBO    (include: /home/smluo/anaconda3/include, library: /home/smluo/anaconda3/lib/libturbojpeg.so)
-- Found OpenCV 2.4.9.1 (libs: opencv_core;opencv_imgproc;opencv_core;opencv_imgproc;opencv_highgui)
CMake Error at cmake/Dependencies.cmake:103 (add_subdirectory):
  The source directory

    /home/smluo/GithubProject/DALI-master/third_party/pybind11

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:47 (include)


-- Found LMDB: /usr/include
-- Found LMDB /usr/include : /usr/lib/x86_64-linux-gnu/liblmdb.so
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Found Protobuf version 2 : /usr/include : /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Exclude libs 'libcudart_static.a:libnvjpeg_static.a:libnppc_static.a:libnppi_static.a:libculibos.a:libturbojpeg.a:libopencv_core.a:libopencv_imgproc.a:libopencv_imgcodecs.a:liblmdb.a'
-- Adding dependencies to dali: '/usr/local/cuda-8.0/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so;/home/smluo/GithubProject/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a;/usr/local/cuda-8.0/lib64/libnppc_static.a;/usr/local/cuda-8.0/lib64/libnppi_static.a;/usr/local/cuda-8.0/lib64/libculibos.a;/home/smluo/anaconda3/lib/libturbojpeg.so;opencv_core;opencv_imgproc;opencv_core;opencv_imgproc;opencv_highgui;/usr/lib/x86_64-linux-gnu/liblmdb.so;/usr/lib/x86_64-linux-gnu/libprotobuf.so'
-- Adding dependencies to dali_test.bin: 'dali'
-- Adding dependencies to dali_benchmark.bin: 'dali'
CMake Error at dali/python/CMakeLists.txt:18 (pybind11_add_module):
  Unknown CMake command "pybind11_add_module".


-- Configuring incomplete, errors occurred!
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeOutput.log".
See also "/home/smluo/GithubProject/DALI-master/build/CMakeFiles/CMakeError.log".

[cliffwoolley]

It sounds like you didn't do a recursive clone of the git repo. Try this:

git submodule sync --recursive
git submodule update --init --recursive

[shl666]

@cliffwoolley Great it works!!!

$ cmake -DBUILD_LMDB=ON -DNVJPEG_ROOT_DIR=/home/smluo/GithubProject/cuda-linux64-nvjpeg -DJPEG_TURBO_ROOT_DIR=/home/smluo/anaconda3/envs/python35 ..
-- The CXX compiler identification is GNU 5.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The C compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PythonInterp: /home/smluo/anaconda3/envs/python35/bin/python (found version "3.5.5")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found Git: /usr/bin/git (found version "2.7.4")
-- git Version: v0.0.0-dirty
-- Version: 0.0.0
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Found CUDA: /usr/local/cuda-8.0 (found suitable version "8.0", minimum required is "8.0")
-- Found NVJPEG: /home/smluo/GithubProject/cuda-linux64-nvjpeg/include (found suitable version "9.0", minimum required is "9.0")
CMake Warning at cmake/Dependencies.cmake:37 (message):
  Using nvJPEG 9.0 together with CUDA 8.0 requires NVIDIA drivers compatible
  with CUDA 9.0 or later
Call Stack (most recent call first):
  CMakeLists.txt:47 (include)


-- Found OpenCV: /home/smluo/anaconda3/envs/python35/include;/home/smluo/anaconda3/envs/python35/include/opencv (found suitable version "3.4.1", minimum required is "2.0")
-- Found PythonLibs: /home/smluo/anaconda3/envs/python35/lib/libpython3.5m.so
-- pybind11 v2.3.dev0
-- Found LMDB: /usr/include (found suitable version "0.9.17", minimum required is "0.9")
-- Found Protobuf: /home/smluo/anaconda3/envs/python35/lib/libprotobuf.so;-pthread (found suitable version "3.5.1", minimum required is "2.0")
-- Enabling TensorFlow TFRecord file format support
-- Exclude libs 'libcudart_static.a:libnvjpeg_static.a:libnppc_static.a:libnppi_static.a:libculibos.a:libturbojpeg.a:libopencv_core.a:libopencv_imgproc.a:libopencv_highgui.a:libopencv_imgcodecs.a:liblmdb.a'
-- Adding dependencies to dali: '/usr/local/cuda-8.0/lib64/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so;/home/smluo/GithubProject/cuda-linux64-nvjpeg/lib64/libnvjpeg_static.a;/usr/local/cuda-8.0/lib64/libnppc_static.a;/usr/local/cuda-8.0/lib64/libnppi_static.a;/usr/local/cuda-8.0/lib64/libculibos.a;opencv_core;opencv_imgproc;opencv_imgcodecs;/usr/lib/x86_64-linux-gnu/liblmdb.so;/home/smluo/anaconda3/envs/python35/lib/libprotobuf.so'
-- Adding dependencies to dali_test.bin: 'dali'
-- Adding dependencies to dali_benchmark.bin: 'dali'
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Adding dependencies to backend_impl: 'dali'
-- Configuring done
-- Generating done

But I still gets errors after I run bash $make -j"$(nproc)" install
There are too many warnings and two main errors.

[ 93%] Linking CXX executable python/nvidia/dali/test/dali_benchmark.bin
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
libdali.so: undefined reference to `tjDecompress2'
libdali.so: undefined reference to `nppiJpegDecodeGetScanDeadzoneSize'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_benchmark.bin.dir/build.make:266: recipe for target 'dali/python/nvidia/dali/test/dali_benchmark.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_benchmark.bin] Error 1
CMakeFiles/Makefile2:445: recipe for target 'dali/CMakeFiles/dali_benchmark.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_benchmark.bin.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 93%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/operators/op_schema_test.cc.o
[ 94%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/operators/operator_factory_test.cc.o
[ 95%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/executor/executor_test.cc.o
[ 95%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/pipeline_test.cc.o
[ 96%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/pipeline/op_graph_test.cc.o
[ 97%] Building CXX object dali/CMakeFiles/dali_test.bin.dir/test/dali_test.cc.o
[ 98%] Linking CXX shared module nvidia/dali/backend_impl.cpython-35m-x86_64-linux-gnu.so
[100%] Linking CXX executable python/nvidia/dali/test/dali_test.bin
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
libdali.so: undefined reference to `tjDecompress2'
libdali.so: undefined reference to `nppiJpegDecodeGetScanDeadzoneSize'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_test.bin.dir/build.make:551: recipe for target 'dali/python/nvidia/dali/test/dali_test.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_test.bin] Error 1
CMakeFiles/Makefile2:407: recipe for target 'dali/CMakeFiles/dali_test.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_test.bin.dir/all] Error 2
[100%] Built target backend_impl
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

I only paste the errors.
I have totally no idea about them.

[cliffwoolley]

That one is a bit more of a mystery. From the CMake output I see you are using CUDA 8, support for which was a recent addition that I haven't personally tested at all; that could be the source of the NPPI symbol error. For TurboJpeg I see that CMake found an install in your Conda environment but wasn't able to tell what version of TurboJpeg (libjpeg-turbo) it was; that likely means it was libjpeg-turbo < 1.5 (whereas 1.5 is the minimum requirement and also the version that starts reporting its own version number in the way we presently expect; I'm working on detecting the older versions better).

It seems you could either try passing these two additional flags to CMake: -DBUILD_BENCHMARK=OFF -DBUILD_TEST=OFF or upgrade your libjpeg-turbo package and your CUDA.

But, backing up... is there a reason you can't use the pre-built wheels?

[JanuszL]

Hi,
CUDA 8.0 is not officially supported so you need to know what you are doing when you use it.
nppiJpegDecodeGetScanDeadzoneSize is used by nvJpeg, and the symbol itself is available in CUDA 9.x. If you want to proceed with CUDA 8.0 you need to disable nvJpeg.
Regarding libjpeg-turbo I see:

Call Stack (most recent call first):
  CMakeLists.txt:47 (include)

As compilation itself succeed it means that you have libjpeg-turbo > 1.5 (or hacked header which may be one of the causes why it doesn't work).
Please rerun linking command with VERBOSE option

make VERBOSE=1

And check if libjpeg-turbo is properly linked there.

[JanuszL]

It seems that we can do nvJpeg to be independent form CUDA runtime. We will try to make that happen in next nvJpeg release.

[shl666]

@JanuszL @cliffwoolley THX for your replies.

For libjpeg-turbo, I installed it by using conda (this is the only way cuz I don't have root privileges on the remote server). I found its version (>1.5) could not identified for some reasons so I comment the 88th line of dali/cmake/CMakelist.txt. Its version could be identified later(I dont know why), so I undo this operation.

For CUDA, I updated it to 9.0 and tried to compile DALI again.
It gives the same feedback, except `nppiJpegDecodeGetScanDeadzoneSize' is defined this time.

(python35) [smluo@Node4 ~/GithubProject/dali/build]
$nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
(python35) [smluo@Node4 ~/GithubProject/dali/build]
$make -j"$(nproc)" install
[  0%] Built target libdali.map
[  2%] Built target gtest
[  3%] Built target DALI_PROTO
[  4%] Built target CAFFE2_PROTO
[ 10%] Built target CAFFE_PROTO
[ 12%] Built target TF_PROTO
[ 21%] Built target benchmark
[ 79%] Built target dali
[ 80%] Linking CXX executable python/nvidia/dali/test/dali_benchmark.bin
[ 81%] Linking CXX shared module nvidia/dali/backend_impl.cpython-35m-x86_64-linux-gnu.so
[ 81%] Linking CXX executable python/nvidia/dali/test/dali_test.bin
libdali.so: undefined reference to `tjDecompress2'
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_benchmark.bin.dir/build.make:215: recipe for target 'dali/python/nvidia/dali/test/dali_benchmark.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_benchmark.bin] Error 1
CMakeFiles/Makefile2:443: recipe for target 'dali/CMakeFiles/dali_benchmark.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_benchmark.bin.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
libdali.so: undefined reference to `tjDestroy'
libdali.so: undefined reference to `tjInitDecompress'
libdali.so: undefined reference to `tjDecompress2'
collect2: error: ld returned 1 exit status
dali/CMakeFiles/dali_test.bin.dir/build.make:526: recipe for target 'dali/python/nvidia/dali/test/dali_test.bin' failed
make[2]: *** [dali/python/nvidia/dali/test/dali_test.bin] Error 1
CMakeFiles/Makefile2:405: recipe for target 'dali/CMakeFiles/dali_test.bin.dir/all' failed
make[1]: *** [dali/CMakeFiles/dali_test.bin.dir/all] Error 2
[ 82%] Built target backend_impl
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

Then I followed @cliffwoolley 's suggestion, added bash -DBUILD_BENCHMARK=OFF -DBUILD_TEST=OFF and DALI was installed successfully.

BTW, I still didn't get what's wrong with benchmark and test part.

[JanuszL]

Looking into your CMakeCache.txt file from https://github.com/NVIDIA/DALI/issues/48 I don't see JPEG_TURBO_ROOT_DIR or JpegTurbo_* variables.
Can you try with latest version of code, we did few improvements to libjpeg-turbo finding process.

[shl666]

I commented find_package(JpegTurbo 1.5 REQUIRED) in cmake/Dependencies.cmake before, for avoid error about cannot identify version of JpegTurbo.

I have pulled the latest source code. Unfortunately, it does not work for my situation.

Following information may help:
I installed JpegTurbo by annaconda
```bash
conda install -c conda-forge libjpeg-turbo

And now when I type ```conda list``` I can see it: 
```bash
libjpeg-turbo             1.5.90                        0    conda-forge

[JanuszL]

Try to install pkg-config as well:

conda install -c conda-forge pkg-config

If you comment out find_package(JpegTurbo 1.5 REQUIRED), then cmake is not able to properly find headers and so library and populate appropriate variables.

[shl666]

Nice! It works. It finds my JpegTurbo.

[JanuszL]

Good. Closing.