Tacotron2: runtimeerror: dimension out of range (expected to be in range of [-1, 0], but got 1)

[deepconsc]

I've fine-tuned the tacotron2 pretrained model on first different dataset, and it showed promising results.
But, for now I'm training for other type of dataset, and getting the dimension out of range error. Looked into the data in details, but it seems fine, pytorch version is 1.0.1, in conda env.

Here's a screenshot

Any suggestions?

[rafaelvalle]

Check the length of that gate_outputs variable. I suspect it has a single frame.

[deepconsc]

checked the variable length, it wasn't a single frame.

here's the screenshot
@rafaelvalle any thoughts?

[androidof2008]

Did you set batch size to 1?
I met the same issue when I set batch size to 1. And when I set batch size to other value other than 1, didn't meet this issue again.

[deepconsc]

@androidof2008 batch size is 8.

Here's the solution, and really hilarious one - it gives the dimension error on batch size 8 on 8 GPUs, but runs on 1 GPU, without throwing any error.

[antontc]

I have the same error. The problem does not occur immediately (after 600iters fine-tune). Batch size = 34. How I can avoid this error?

[antontc]

The gate_outputs have this structure.

[rafaelvalle]

Check that there exists no culprit data, for example no text, no audio, etc...

[rafaelvalle]

Closing due to inactivity.

[b-schnell]

Hello,

I believe I ran into the same error when using only the model with my training code. The problem is at https://github.com/NVIDIA/tacotron2/blob/131c1465b48be60cb5d3b8ab79cfc663e5c47b6a/model.py#L410
For a batch size B the gate_outputs will be B x 1 where the last dimension is squeezed. However, if your batch size is one both dimensions will be squeezed and you get the error. This is why it fails with a batch size of 8 on 8 GPUs, where the batch size on each GPU is again one. In my case it failed because I do not use drop_last=True in my DataLoader and the last chunk turned out to have a size of one. The solution for me was simply to change the code to gate_output.squeeze(1).

I don't quite understand why it is not failing immediately for @antontc. Maybe the end of the data set is reached after 600 iterations and then the last chunk triggers the error as in my case?

[chazo1994]

Hello,

I believe I ran into the same error when using only the model with my training code. The problem is at

https://github.com/NVIDIA/tacotron2/blob/131c1465b48be60cb5d3b8ab79cfc663e5c47b6a/model.py#L410

For a batch size B the gate_outputs will be B x 1 where the last dimension is squeezed. However, if your batch size is one both dimensions will be squeezed and you get the error. This is why it fails with a batch size of 8 on 8 GPUs, where the batch size on each GPU is again one. In my case it failed because I do not use drop_last=True in my DataLoader and the last chunk turned out to have a size of one. The solution for me was simply to change the code to gate_output.squeeze(1).
I don't quite understand why it is not failing immediately for @antontc. Maybe the end of the data set is reached after 600 iterations and then the last chunk triggers the error as in my case?

It work for me, thanks.