Tacotron2: RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

[lysektomas]

Hi!

I have tried to run setup.py and I am getting this error

Dynamic Loss Scaling: True
Distributed Run: False
cuDNN Enabled: True
cuDNN Benchmark: False
Traceback (most recent call last):
  File "train.py", line 284, in <module>
    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
  File "train.py", line 169, in train
    model = load_model(hparams)
  File "train.py", line 81, in load_model
    model = Tacotron2(hparams).cuda()
  File "/home/papoadmin/anaconda3/envs/ai/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/papoadmin/anaconda3/envs/ai/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/home/papoadmin/anaconda3/envs/ai/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/home/papoadmin/anaconda3/envs/ai/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 117, in _apply
    self.flatten_parameters()
  File "/home/papoadmin/anaconda3/envs/ai/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 113, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

Does anyone know, what is causing this?

It is failing on
model = Tacotron2(hparams).cuda()

My environment:
system ubuntu 18.04
gpu: NVIDIA RTX 2080 Ti
python 3.6.0
cuda 10.0
torch 1.0
cudnn 7.4.1

others models which are using torch and tensorflow are working normally.

I have tried to use CUDA_VISIBLE_DEVICES=0 with no success.

[lysektomas]

After few hours of grinding I have removed conda environment and install it all again. Plus I have installed torch using conda. It is working now.

So probably bad torch version.

[lrh000]

hi, I meet a same problem. Did you only change the torch version?
my envirionment:
2080ti
pytorch 0.4.1
python 3.6.7
cuda 10.0
cudnn 7.3.0

[xiaoxiangyeyuwangye]

@lrh000 Have you solved this problem? I met a similar issue.

[lrh000]

@lrh000 Have you solved this problem? I met a similar issue.

@xiaoxiangyeyuwangye Yeah, I have. My OS is Ubuntu16.04. I managed to only install the 2080ti driver through cuda 10.0 runfile and then install cuda9.2&cudnn7.1.4 toolkit. Remember that the most important thing is to download pytorch 0.4.1 adapted to cuda9.2&python version from official url:https://pytorch.org/get-started/previous-versions/
GL!

[xiaoxiangyeyuwangye]

@lrh000 Thanks for your reply. So in your opinion, this issue is caused by the incompatibility of pytorch version, driver version, cuda version and cudnn version, right? In my situation, this bug is really arbitrary. It doesn't happen in every step. However，after some training steps，it happened.

[lrh000]

@lrh000 Thanks for your reply. So in your opinion, this issue is caused by the incompatibility of pytorch version, driver version, cuda version and cudnn version, right? In my situation, this bug is really arbitrary. It doesn't happen in every step. However，after some training steps，it happened.

@xiaoxiangyeyuwangye I understand your problem. When I tested some simple networks, there was no bug appearing. But I met this error every time run the deep BiLSTM. Maybe this incompatibility you mentioned just happens when RNN runs. Check your own networks and the details in this kind of error traceback information, and you may find how to solve your dilemma :)

[xiaoxiangyeyuwangye]

@lrh000 Thanks a lot for your suggestion. In fact, I met this problem when training CycleGAN network. The architecture of it is not so complicated, but the training procedure is.

[Henry-E]

This is funny, I'm running into the same issue RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
It happens when using a 2080ti but not a K40. Both GPUs are using the same environment, code, etc. and pytorch 1.0 . For some reason it gets the error on the 2080ti. Is this some indication of CUDA not being installed correctly?

[lysektomas]

Hi,
I completely removed conda evnironment, cudnn, cuda and graphics drivers.

Then I find everyting related to nvidia trough find /usr | grep nvidia, cuda, ... and removed few old libraries.

After that I installed it again and it was working.

It seems that I had mismatch version between cuda and cudnn, but I don't know.

[playma]

Go to pytorch website, and choose the version which satisfies your cuda version
https://pytorch.org/

cu100 = cuda 10.0

pip3 uninstall torch
pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl