Tensorrt: [TensorRT] WARNING: TensorRT was linked against cuDNN 7.6.3 but loaded cuDNN 7.4.2

[Eric-Zhang1990]

Description

I can load one project model using tensorrt on my docker image, but when I load another project model using ternsorrt, this warning occurs.
"[TensorRT] WARNING: TensorRT was linked against cuDNN 7.6.3 but loaded cuDNN 7.4.2."
After that, the following error occurs.
"rtSafe/safeContext.cpp (133) - Cudnn Error in configure: 7 (CUDNN_STATUS_MAPPING_ERROR)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception"

Environment

TensorRT Version: 6.0.1.5
GPU Type: 2080 ti
Nvidia Driver Version: 430.26
CUDA Version: 10.0
CUDNN Version: 7.6.5
Operating System + Version: ubuntu 16.04
Python Version (if applicable): 2.7
Baremetal or Container (if container which image + tag):

Relevant Files

Steps To Reproduce

[Eric-Zhang1990]

@rmccorm4 Do you know why this happen? I ask the error before, but this time it is the cudnn version, I run the 2 models in the same docker container, one can run correctly, but another does not.

[Eric-Zhang1990]

The problem is I can not find cudnn 7.4.2 in my docker container.
Can anyone help me check this problem? Thanks.

[Eric-Zhang1990]

I change cudnn to 7.4.2 on my own computer, although tensorrt has warning, but it can output correctly. It means cudnn version is not main reason for error ""rtSafe/safeContext.cpp (133) - Cudnn Error in configure: 7 (CUDNN_STATUS_MAPPING_ERROR)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception"".

Can someone give advices for me? Thanks.

[Eric-Zhang1990]

I found the problem, it is caused by the init location of "cuda.Device().make_context()", we should init it in main thread.

[dhkim0225]

I had a same problem and it was because "import torch"

Do not import both torch and tensorrt.

[guanshuicheng]

I found the problem, it is caused by the init location of "cuda.Device().make_context()", we should init it in main thread.

Hello, I encountered the same problem as you. I loaded the TRT engine and the pytorch model at the same time, and the error information is exactly the same as yours. Could you please tell me the location of the initialization thread "cuda.device ().make_context()" that you mentioned? Can you provide more information

[rmccorm4]

I had a same problem and it was because "import torch"
Do not import both torch and tensorrt.

This seems to be the underlying issue @dhkim0225 : https://github.com/pytorch/pytorch/issues/32983

If your pytorch and tensorrt versions are built with the same version of CUDNN, then I don't think you'll see this problem.

[tensorflowt]

I found the problem, it is caused by the init location of "cuda.Device().make_context()", we should init it in main thread.

Hello, I encountered the same problem as you. I loaded the TRT engine and the pytorch model at the same time, and the error information is exactly the same as yours. Could you please tell me the location of the initialization thread "cuda.device ().make_context()" that you mentioned? Can you provide more information

Have you solved the problem yet ？