Tacotron2: How to synthesize audio from checkpoint?

[mligema]

Finally I got the _python -m multiproc train.py etc..._ to work. Simple question: How do I now synthesize audio form a specific checkpoint?

In github.com/keithito/tacotron it was pretty simple: _python eval.py --checkpoint 'path_to_checkpoint_', but here I don't see any documentation on generating wav's.

[Clement-Hui]

You can check out the inference.ipynb . It provides an easy way to inference the audio, but you have to download pretrained waveglow module or code your own way of making the spectrogram output to audio, maybe via wavenet vocoder.

[Yeongtae]

@mligema
Use https://github.com/Yeongtae/tacotron2/blob/master/inference.py

[mligema]

@Yeongtae thanks! It is generating silent wavs. Any suggestions? Here's my melspectrogram

It could be that my training hparams are the problem.

[Clement-Hui]

How many steps did you train? Model should converage around 10k steps.

[mligema]

95000 steps, with 3.5h of total audio files.

[Clement-Hui]

Please refer to #254 and #256. In short, you should try training the model with the English pretrained model as initial weights. Also, you should set attention dropout to a higher value like 0.4. Also, the attention should be a diagonal line across the graph

[Clement-Hui]

Also you can try using the original inference.ipynb. maybe that would help.

[mligema]

So I trained with the Engish pretrained model, with dropout 0.4, to step 35000. Then I synthesized this voice using the inference.ipynb, but it sounds far too "rough" https://www.dropbox.com/s/1fltkjplbn5dlsb/step35000.mp3?dl=0

Any ideas how to improve the quality of synthesized sound?

[Yeongtae]

So I trained with the Engish pretrained model, with dropout 0.4, to step 35000. Then I synthesized this voice using the inference.ipynb, but it sounds far too "rough" https://www.dropbox.com/s/1fltkjplbn5dlsb/step35000.mp3?dl=0

Any ideas how to improve the quality of synthesized sound?

Test my master branch.
It uses preprocessing for audio datasets and normalization for Mel-spectrogram.

https://github.com/Yeongtae/tacotron2
https://github.com/Yeongtae/tacotron2/blob/master/preprocess_audio.py

In addition, you can use validation metrics for monitoring seq 2 seq speech synthesis model on the Tensorboard.
https://github.com/Yeongtae/tacotron2/tree/monitoring_metric

If attention alignment diagonality is close to 1, it means that the seq 2 seq model has been learned well.

[Clement-Hui]

What is the data used? Maybe the data does not have enough samples

[mligema]

@Clement-Hui my dataset consists of 3277 wav files, 1.6GB, single-channel, 22050Hz, 352 kbps, and they are between 5-20 sec long.
Okey @Yeongtae I will try train a model with your fork tacotron2, I'll let you know if I have results.

[Yeongtae]

In my master branch, there is no monitoring metric. It is in the monitoring metric branch.

[Clement-Hui]

What is the method used to convert Mel spectrogram to wave? Is it waveglow or others? Try using wavenet vocoder.

[rafaelvalle]

Closing due to inactivity.