Real-time-voice-cloning: demo_cli.py works but demo_toolbox.py crashes cuDNN failed to initialize

[MichaelAllenHardeman]

I've gotten demo_cli.py to work correctly, it successfully generates the demo sound file, but demo_toolbox.py crashes. It crashes the same way either if I load the train-clean-100 files or if I load custom files. This is the full log output.
error.txt

[MichaelAllenHardeman]

The main error seems to be this one, in case anyone else gets this crash and searches

2019-08-23 18:43:37.956870: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.2.1 but source was compiled with: 7.4.1.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2019-08-23 18:43:37.978871: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.2.1 but source was compiled with: 7.4.1.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
  File "C:\Users\michael\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
    return fn(*args)
  File "C:\Users\michael\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\michael\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/conv1d}}]]
         [[Tacotron_model/inference/add/_269]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node Tacotron_model/inference/encoder_convolutions/conv_layer_1_encoder_convolutions/conv1d/conv1d}}]]
0 successful operations.
0 derived errors ignored.

[MichaelAllenHardeman]

I'm going to try 1 last thing, which is to downgrade CUDA Toolkit from 10.1 to 10.0 which I see suggested in another git issue. will report back.

Edit: downgrade didn't help.

[Oliver-Piorun]

You are not alone. I get the exact same error.

• Windows 10 (x64)
• Python 3.7.3
• CUDA 10.0.130
• cuDNN 7.6.3
• PyTorch 1.2

Greetings ~

Edit: tensorflow-gpu 1.14.0 is compiled against 7.4.1. I am not sure why the loaded runtime library version is 7.2.1. Is it controlled by the application?

Edit2: I got it working! The cuDNN runtime library is provided by torch. I just replaced the cudnn64_7.dll (which is located in C:\Users\\\AppData\Local\Programs\Python\Python37\Libsite-packages\torch\lib) with the cuDNN 7.6.3 runtime library.

[MichaelAllenHardeman]

nice I'll try that myself and see if I get the same result.

[MichaelAllenHardeman]

Can confirm. replacing C:\Users\<USER>\AppData\Local\Programs\Python\Python37\Lib\site-packages\torch\lib) with the version provided by nvidia works.