Tacotron2: IEMOCAP training

[xliucs]

I am trying to train the tacotrone 2 on the IEMOCAP dataset. In order to fully take advantage of all the speakers, i added a speaker reference encoder (concatenate with the original text encoder). However, I got this alignment after almost 32k steps (batch size == 24). Is there anyone has ideas regarding my alignment? I also saw this happened on other issues with trainng on LJ dataset. Just wondering what does the alginment even mean.

P.s., my validation loss is already very low, and my mel_predicted looks very good.

[rafaelvalle]

Are you resuming the pre-trained model?

[xliucs]

Are you resuming the pre-trained model?

Thanks for getting back to me. No, I am not resuming the pre-traind model since I added a speaker embeding. I am training it from scratch.

[rafaelvalle]

I assume IEMOCAP is a small dataset. In such cases, Tacotron gets stuck in some local minima and ignores the encoder outputs. One way to avoid this is to decrease the dimensionality of the prenet layer.

Another way to fix this is to train your multi-speaker architecture on LJS first and then fine-tune it to IEMOCAP. This will prevent you from decreasing the dimensionality of the prenet layer.

If you train train your multi-speaker architecture on LJS, use as many weights as you can from the pre-trained LJS model in this repo.

[xliucs]

I assume IEMOCAP is a small dataset. In such cases, Tacotron gets stuck in some local minima and ignores the encoder outputs. One way to avoid this is to decrease the dimensionality of the prenet layer.

Another way to fix this is to train your multi-speaker architecture on LJS first and then fine-tune it to IEMOCAP. This will prevent you from decreasing the dimensionality of the prenet layer.

If you train train your multi-speaker architecture on LJS, use as many weights as you can from the pre-trained LJS model in this repo.

I will look into this. Thanks so much for your help!

[rafaelvalle]

Let us know what works for you such that we propagate the findings.

[patrick-g-zhang]

I assume IEMOCAP is a small dataset. In such cases, Tacotron gets stuck in some local minima and ignores the encoder outputs. One way to avoid this is to decrease the dimensionality of the prenet layer.

Another way to fix this is to train your multi-speaker architecture on LJS first and then fine-tune it to IEMOCAP. This will prevent you from decreasing the dimensionality of the prenet layer.

If you train train your multi-speaker architecture on LJS, use as many weights as you can from the pre-trained LJS model in this repo.

You mean if we train on a small dataset, we need to decrease the dimensionality of prenet layer. Here is my mind, it limits the decoder part getting too more information form ground truth(reference.)
And it can use more information from encoder. Am I right ?

[bfs18]

I assume IEMOCAP is a small dataset. In such cases, Tacotron gets stuck in some local minima and ignores the encoder outputs. One way to avoid this is to decrease the dimensionality of the prenet layer.
Another way to fix this is to train your multi-speaker architecture on LJS first and then fine-tune it to IEMOCAP. This will prevent you from decreasing the dimensionality of the prenet layer.
If you train train your multi-speaker architecture on LJS, use as many weights as you can from the pre-trained LJS model in this repo.

You mean if we train on a small dataset, we need to decrease the dimensionality of prenet layer. Here is my mind, it limits the decoder part getting too more information form ground truth(reference.)
And it can use more information from encoder. Am I right ?

This trick is used in direct speech-to-speech translation paper. https://arxiv.org/abs/1904.06037
Decreasing the prenet bottleneck size would compress the teacher forcing input and increase the information gap between the teacher forcing input and target at training, so the model has to exploit text more to reconstruct the target. However, I'm not sure whether this relates to dataset scale or not.

[patrick-g-zhang]

I assume IEMOCAP is a small dataset. In such cases, Tacotron gets stuck in some local minima and ignores the encoder outputs. One way to avoid this is to decrease the dimensionality of the prenet layer.
Another way to fix this is to train your multi-speaker architecture on LJS first and then fine-tune it to IEMOCAP. This will prevent you from decreasing the dimensionality of the prenet layer.
If you train train your multi-speaker architecture on LJS, use as many weights as you can from the pre-trained LJS model in this repo.

You mean if we train on a small dataset, we need to decrease the dimensionality of prenet layer. Here is my mind, it limits the decoder part getting too more information form ground truth(reference.)
And it can use more information from encoder. Am I right ?

This trick is used in direct speech-to-speech translation paper. https://arxiv.org/abs/1904.06037
Decreasing the prenet bottleneck size would compress the teacher forcing input and increase the information gap between the teacher forcing input and target at training, so the model has to exploit text more to reconstruct the target. However, I'm not sure whether this relates to dataset scale or not.

Thank you. I will take a try. Because I might meet similar problem.

[kkokdari]

@xin71 @patrick-g-zhang Hello! Can you update the result of this trial? Appreciation!

[xliucs]

Could not really get it working. My guess IEMOCAP is too small to train a tacotrone model.

[rafaelvalle]

@xin71 What did you try?

[xliucs]

@xin71 What did you try?

I tried combining IEMOCAP with VCTK and LJ datasets and added a speaker embedding on the tacotone to test out if it can generate synthesized speech from different speakers. The training kinda worked on VCTK, but it did not work when I added IEMOCAP. My guess is that IEMOCAP has limited data for each speaker, but did not really investigate into that much.