Tacotron2: It's working now---using Chinese corpus

[maozhiqiang]

the plot like

the train is very slowing

[rafaelvalle]

That is great to know, @maozhiqiang!
What exactly is slow? the training iteration or the time it takes to learn attention?
What is your setup?

[begeekmyfriend]

I am glad to say I have transplanted some code from Rayhane Mamah's version into Keith Ito's one and it can get to convergence on small dataset (10h) within 8K steps. I will release my fork in several days since I want to find the way in minimal modification. I will notify you as well as Keith Ito at that time. But I think I can show some clues about the modification that the architecture seems a little different with Keith Ito's one https://github.com/Rayhane-mamah/Tacotron-2/issues/4#issuecomment-375676828. The AttentionWrapper class in Tensorflow is inappropriate for Tacotron model since the query of attention should be the hidden states of 2-layer decoder LSTM while in AttentionWrapper the default query is set as the hidden state of extra wrapped attention RNN. The attention RNN is redundant because it does the same thing that decoder LSTM does. Moreover the architecture mentioned in Tacotron paper shows that the query should be the hidden states of decoder LSTM. That is why Rayhane Mamah substitute TacotronDecoderCell for AttentionWrapper that can help get to convergence.

[rafaelvalle]

@begeekmyfriend let's move this comment and discussion here: https://github.com/NVIDIA/tacotron2/issues/12

[maozhiqiang]

@rafaelvalle I training useing one GPU ,so the training is so slowing !and the attention is not until the twenty thousand step begins to align！

[rafaelvalle]

Training time is probably dominated by computing mel-spectrograms on the CPU and can be amortized by pre-processing the mel-spectrograms and loading them from disk.

We're looking into speeding up the model such that the attention is learned faster.

[rafaelvalle]

@maozhiqiang What is your setup and how many seconds per iteration on average on it?

[rafaelvalle]

FYI: We've added the option to load mels from disk.

[maozhiqiang]

@rafaelvalle thanks! The following is training log
train_log.log

[rafaelvalle]

@maozhiqiang What GPU do you have?

[maozhiqiang]

@rafaelvalle P5000 16G

[rafaelvalle]

To me, the iterations per second seem comparable to what I've seen with Taco 2 implementations.
It should be less of a hassle once we improve the model to learn attention faster.

[maozhiqiang]

@rafaelvalle thanks!

[rafaelvalle]

@maozhiqiang please re-run your experiments on the branch below and and let us know what your convergence time is. In this branch, the model attends to the full mel-spectrogram instead of the prenet drop out mel: this should increase convergence time considerably.
https://github.com/NVIDIA/tacotron2/tree/attention_full_mel

[maozhiqiang]

@rafaelvalle thanks , I will try this

[rafaelvalle]

@maozhiqiang great. We'll soon add zoneout as well.

[root20]

@maozhiqiang Could you share what Chinese corpus you have used?

[maozhiqiang]

sorroy ! The data is not open!

[rafaelvalle]

Closing. Please re-open if the issues haven't been solved.

[begeekmyfriend]

@maozhiqiang Would you please open your modification on this project for Chinese mandarin on your repo?

[maozhiqiang]

@begeekmyfriend yes! i uploaded ! please go to https://github.com/maozhiqiang/tacotron2_pytorch.git

[NewEricWang]

Hi, @maozhiqiang , could you share some synthesized speech samples in Chinese tacotron2 project? What is the unit you used? How do you control the tone of the synthesized speech?

[jinfagang]

@maozhiqiang The link is missing

[tsungruihon]

@maozhiqiang the link is missing too.

[tsungruihon]

@maozhiqiang could you share you hparams.py detail ?