Tensorrt: Some errors when generating the TensorRT build container

[HMJiangAI]

I have followed your tutor of TensorRT Open Source Software. However, I met these errors:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CUBLASLT_LIB
    linked by target "nvinfer_plugin" in directory /home/jhm/TensorRT/plugin

-- Configuring incomplete, errors occurred!

I don't know why it occurred because I have downloaded the TensorRT binary release and met the requirements of System Packages. I checked /TensorRT/plugin and there is not any file named nvinfer_plugin. My system configuration is:

ubuntu16.04LTS
cuda 10.0
cudnn 7.6.3
tensorrt 6.0.1.5

And the tar file that I chose is TensorRT-6.0.1.5.Ubuntu-16.04.x86_64-gnu.cuda-10.0.cudnn7.6.tar.gz.

[rmccorm4]

Hi @Programmakememad,

Are the proper environment variables for CUDA/CUDNN/etc. set correctly?

You could debug a bit better by trying to find libcublas.so on your system, and then trying to add it's directory (on my system it looks like its /usr/lib/x86_64-linux-gnu as seen below):

$ sudo find / -name libcublas.so
/usr/lib/x86_64-linux-gnu/stubs/libcublas.so
/usr/lib/x86_64-linux-gnu/libcublas.so

And then add that directory to LD_LIBRARY_PATH, something like:

LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH"

Try experimenting with some things like this

[HMJiangAI]

I have checked the environment variables for CUDA and CuDNN, everything is suitable in ~/.bashrc. Besides, occording to your suggestion, I add path of libcublas.so to ~/.bashrc and source ~/.bashrc, then retry building, but the initial errors still exist. I think it means that CUBLASLT_LIB is linked by nvinfer_plugin, so there maybe some files named nvinfer_plugin in TensorRT/plugin?

[rmccorm4]

Can you share the contents of your ~/.bashrc where you set up all of your environment variables?

[HMJiangAI]

I have run
sudo find / -name libcublas.so
and the output is

/usr/local/cuda-10.0/lib64/stubs/libcublas.so
/usr/local/cuda-10.0/lib64/libcublas.so

Besides, my environment variables in ~/.bashrc is

export CUDA_HOME=/usr/local/cuda-10.0
export PATH="/usr/local/cuda-10.0/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64:$LD_LIBRARY_PATH"

[HMJiangAI]

I find that cublasLt is a new API in cuda 10.1 and included in Cmakelist, therefore, the system cannot find it because my system environment is cuda 10.0. I need to use tensorflow-gpu-1.14 and it cannot support cuda 10.1 according to official document.

tensorflow.google.cn/install/gpu

[rmccorm4]

Hi @Programmakememad,

It seems this was a relatively new change from moving some things around from the BERT demo: https://github.com/NVIDIA/TensorRT/commit/77af534c21b7e1bfa322ccbab028477be4c56eb3

You may be able to solve this by just using the release/6.0 branch instead of master since it doesn't contain that change.

If you need to use master for some other reason, as a workaround you may be able to remove/comment out the references to CUBLASLT_LIB because I don't think they're necessary to run the current samples, but I haven't tried this myself: https://github.com/NVIDIA/TensorRT/search?q=cublaslt_lib&unscoped_q=cublaslt_lib

If you want to try the CUDA 10.1 route instead, you can look at this post: https://github.com/NVIDIA/TensorRT/issues/123#issuecomment-551269792

[HMJiangAI]

Thank you for your suggestions and I have already built tensorrt oss and run the demo of mask r-cnn. However, although the demo passed, I still met a warning:

[10/25/2019-16:47:22] [I] [TRT] Some tactics do not have sufficient workspace memory to run.
Increasing workspace size may increase performance, please check verbose output.

Could I fix this problem by changing some settings of software or I must expand my hardware memory?

[HMJiangAI]

Besides, how to install tensorrt oss? I try to install under $TRT_SOURCE and it seems to be incorrect. Then I refer your guidance

TensorRT installation directory is determined as $TRT_LIB_DIR/..
This path is equal to $TRT_RELEASE and I copy all files under $TRT_SOURCE/build to $TRT_RELEASE, updating some files from the original release files. After that, I run sudo make install and it fail, reporting the following:

[  2%] Built target third_party.protobuf
make[2]: *** No rule to make target '../plugin/nmsPlugin/nmsPlugin.cpp', needed by 'plugin/CMakeFiles/nvinfer_plugin.dir/nmsPlugin/nmsPlugin.cpp.o'.  Stop.
CMakeFiles/Makefile2:283: recipe for target 'plugin/CMakeFiles/nvinfer_plugin.dir/all' failed
make[1]: *** [plugin/CMakeFiles/nvinfer_plugin.dir/all] Error 2
Makefile:151: recipe for target 'all' failed
make: *** [all] Error 2

[leimao]

Besides, how to install tensorrt oss? I try to install under $TRT_SOURCE and it seems to be incorrect. Then I refer your guidance

TensorRT installation directory is determined as $TRT_LIB_DIR/..
This path is equal to $TRT_RELEASE and I copy all files under $TRT_SOURCE/build to $TRT_RELEASE, updating some files from the original release files. After that, I run sudo make install and it fail, reporting the following:

[  2%] Built target third_party.protobuf
make[2]: *** No rule to make target '../plugin/nmsPlugin/nmsPlugin.cpp', needed by 'plugin/CMakeFiles/nvinfer_plugin.dir/nmsPlugin/nmsPlugin.cpp.o'.  Stop.
CMakeFiles/Makefile2:283: recipe for target 'plugin/CMakeFiles/nvinfer_plugin.dir/all' failed
make[1]: *** [plugin/CMakeFiles/nvinfer_plugin.dir/all] Error 2
Makefile:151: recipe for target 'all' failed
make: *** [all] Error 2

Hi,
You don't have to manually copy the built OSS files, the installation is a process of copying files.
After building, in the directory $TRT_SOURCE/build/, you run the command make install (use sudo if necessary).

root@acd028959ccf:/tensorrt/bin# cd $TRT_SOURCE/build/
root@acd028959ccf:/workspace/TensorRT/build# make install
[  3%] Built target third_party.protobuf
[ 20%] Built target nvinfer_plugin
[ 38%] Built target nvinfer_plugin_static
[ 38%] Built target caffe_proto
[ 48%] Built target nvcaffeparser
[ 58%] Built target nvcaffeparser_static
[ 59%] Built target gen_onnx_proto
[ 61%] Built target onnx_proto
[ 63%] Built target nvonnxparser_static
[ 66%] Built target nvonnxparser
[ 67%] Built target onnx2trt
[ 68%] Built target getSupportedAPITest
[ 68%] Built target trt_onnxify
[ 69%] Built target sample_char_rnn
[ 69%] Built target sample_dynamic_reshape
[ 70%] Built target sample_fasterRCNN
[ 72%] Built target sample_googlenet
[ 73%] Built target sample_int8
[ 74%] Built target sample_int8_api
[ 75%] Built target sample_mlp
[ 76%] Built target sample_mnist
[ 77%] Built target sample_mnist_api
[ 79%] Built target sample_movielens
[ 80%] Built target sample_movielens_mps
[ 87%] Built target sample_nmt
[ 88%] Built target sample_onnx_mnist
[ 89%] Built target sample_plugin
[ 90%] Built target sample_reformat_free_io
[ 91%] Built target sample_ssd
[ 93%] Built target sample_uff_fasterRCNN
[ 94%] Built target sample_uff_maskRCNN
[ 95%] Built target sample_uff_mnist
[ 95%] Built target sample_uff_plugin_v2_ext
[ 96%] Built target sample_uff_ssd
[100%] Built target trtexec
Install the project...
-- Install configuration: "Release"
-- Up-to-date: /tensorrt/lib/../lib/libnvinfer_plugin.so.6.0.1.0
-- Up-to-date: /tensorrt/lib/../lib/libnvinfer_plugin.so.6.0.1
-- Up-to-date: /tensorrt/lib/../lib/libnvinfer_plugin.so
-- Up-to-date: /tensorrt/lib/../lib/libnvcaffeparser.so.6.0.1.0
-- Up-to-date: /tensorrt/lib/../lib/libnvcaffeparser.so.6.0.1
-- Up-to-date: /tensorrt/lib/../lib/libnvcaffeparser.so
-- Up-to-date: /tensorrt/lib/../bin/onnx2trt
-- Up-to-date: /tensorrt/lib/../lib/libnvonnxparser.so.6.0.1
-- Up-to-date: /tensorrt/lib/../lib/libnvonnxparser.so.6
-- Up-to-date: /tensorrt/lib/../lib/libnvonnxparser.so
-- Up-to-date: /tensorrt/lib/../lib/libnvonnxparser_static.a
-- Up-to-date: /tensorrt/lib/../include/NvOnnxParser.h
-- Up-to-date: /tensorrt/lib/../bin/sample_char_rnn
-- Up-to-date: /tensorrt/lib/../bin/sample_dynamic_reshape
-- Up-to-date: /tensorrt/lib/../bin/sample_fasterRCNN
-- Up-to-date: /tensorrt/lib/../bin/sample_googlenet
-- Up-to-date: /tensorrt/lib/../bin/sample_int8
-- Up-to-date: /tensorrt/lib/../bin/sample_int8_api
-- Up-to-date: /tensorrt/lib/../bin/sample_mlp
-- Up-to-date: /tensorrt/lib/../bin/sample_mnist
-- Up-to-date: /tensorrt/lib/../bin/sample_mnist_api
-- Up-to-date: /tensorrt/lib/../bin/sample_movielens
-- Up-to-date: /tensorrt/lib/../bin/sample_movielens_mps
-- Up-to-date: /tensorrt/lib/../bin/sample_nmt
-- Up-to-date: /tensorrt/lib/../bin/sample_onnx_mnist
-- Up-to-date: /tensorrt/lib/../bin/sample_plugin
-- Up-to-date: /tensorrt/lib/../bin/sample_reformat_free_io
-- Up-to-date: /tensorrt/lib/../bin/sample_ssd
-- Up-to-date: /tensorrt/lib/../bin/sample_uff_fasterRCNN
-- Up-to-date: /tensorrt/lib/../bin/sample_uff_maskRCNN
-- Up-to-date: /tensorrt/lib/../bin/sample_uff_mnist
-- Up-to-date: /tensorrt/lib/../bin/sample_uff_plugin_v2_ext
-- Up-to-date: /tensorrt/lib/../bin/sample_uff_ssd
-- Up-to-date: /tensorrt/lib/../bin/trtexec

[verigle]

[ 31%] Built target nvcaffeparser_static
[ 31%] Running C++ protocol buffer compiler on /home/verigle/Projects/Github/TensorRT/build/parsers/onnx/third_party/onnx/onnx/onnx-operators_onnx2trt_onnx-ml.proto
Scanning dependencies of target onnx_proto
[ 31%] Building CXX object parsers/onnx/third_party/onnx/CMakeFiles/onnx_proto.dir/onnx/onnx_onnx2trt_onnx-ml.pb.cc.o
[ 31%] Building CXX object parsers/onnx/third_party/onnx/CMakeFiles/onnx_proto.dir/onnx/onnx-operators_onnx2trt_onnx-ml.pb.cc.o
[ 32%] Linking CXX static library libonnx_proto.a
[ 33%] Built target onnx_proto
Makefile:151: recipe for target 'all' failed
make: * [all] Error 2

[rmccorm4]

PR #257 fixes this issue by checking for CUDA >= 10.1 to include CUBLASLT

[wangxiang2713]

Thank you for your suggestions and I have already built tensorrt oss and run the demo of mask r-cnn. However, although the demo passed, I still met a warning:

[10/25/2019-16:47:22] [I] [TRT] Some tactics do not have sufficient workspace memory to run.
Increasing workspace size may increase performance, please check verbose output.

Could I fix this problem by changing some settings of software or I must expand my hardware memory?

Hi, have you slove this problem?

[HMJiangAI]

@wangxiang2713 This problem still exists. I try to set maxWorkspace to 6GiB and it still reports this warning. Then I use 7GiB and it reports error:
[E] [TRT] ../rSafe/safeRuntime.cpp (25) - Cuda Error in allocate : 2 (out of memory)
However, my graphics card has 8 GiB memory.

[solarflarefx]

@Programmakememad Were you ever able to solve your problem?