Tacotron2: Trainning data for wavenet_vocoder

[deli666]

I trained this model with my own dataset and want to train wavenet_vocoder after this. But I am confused what is the training data for wavenet_vocoder? Mel_spectrogram generated by this model or generated by sftf?
Besides, what do you mean by "When performing Mel-Spectrogram to Audio synthesis with a WaveNet model, make sure Tacotron 2 and WaveNet were trained on the same mel-spectrogram representation"?

[harlyh]

1) While you train Tacotron, Audio is converted to Mels and the Mel+Label(text) pairs are fed to Tacotron. After successful training you should have a model, which can generate Mels from Text.
Now for wavenet you feed the synthesised Mels+Audio samples to Wavenet. This eventually should produce a model, which is able to Synthesize Mels into ncie Audio.
So to summarise:

• Tacotron training needs Audio+Text input and will start synthesizing Mels from input text, which you can either convert to low quality audio using Griffin-Lim, or use the Mels for next step, that is
• Wavenet training, which takes the Tacotron synthesized Mel + Audio pairs as input to produce a model which can synthesize nice sounding speech from Tacotron Mels.

2) It just means that if, for example, you want to use pre-trained Wavenet model, then the Mel parameters used for Tacotron training must match the Mel params that were used while Wavenet was trained.
Or to put it differently: between Tacotron and Wavenet training sessions you mustn't change Mel parameters.

Hope this explains it

[deli666]

Thank you so much for your answer!
So do you mean that the input training data for wavenet vocoder should be synthesized by pretrained tacotron model?
If so, one wavenet vocoder model can only synthesize .wav generated by exactly one tacotron model. If I train the tacotron model with more steps, I need to train wavenet vocoder once again.
Do I understand that correctly?
Best,

[harlyh]

Well, haven't actually considered training Wavenet until I'm not satisfied with the Tacotron synthesis. Also I haven't tried using pre-trained Wavenet. But considering the intended workflow I do believe that Wavenet would produce the best results only when synthesising speech using the output from the same Tacotron model as was used during Wavenet training. However, I'm not saying that you can't do that. For me it just wouldn't make much sense :)

[rafaelvalle]

Please check this issue where we explain how to use mels from a Tacotron that was trained on data that does not match the same processing as wavenet.

https://github.com/NVIDIA/tacotron2/issues/41

[rafaelvalle]

Closing due to inactivity. Please re-open if necessary.