Tacotron2: Audio examples?

[ghenter]

Very cool work, this! However, it would be ideal to also provide examples of input text + output audio from a trained system, alongside held-out examples from the database. This will give an impression of what kind of results the code is capable of generating with the LJSpeech data, and is standard practise in the text-to-speech field.

Aside from synthesising held-out sentences from LJSpeech, Google's speech examples for Tacotron 2 provide another set of challenging text prompts to generate.

Are there any plans to do this? Or are synthesised speech examples already available somewhere?

[nikitos9000]

My result of training this repo on LJSpeech dataset on a single GPU for ~98 hours — https://www.dropbox.com/s/k9w7wwpgte3gnqb/tacotron2_ljspeech_step60000.wav?dl=0

The lack of WaveNet vocoder makes the difference.

[rafaelvalle]

The wavenet vocoder does make a huge difference, and we’re in the process of open sourcing that as well.

[rafaelvalle]

@nsmetanin can you share training curves, etc? i assume people will be interested in looking at that.

[HaoyuanLuo]

@nsmetanin Nice result, Thanks for you sharing. I'm very intersted in how much time it takes to synthesis a 5 seconds wav file. Would you like to share it with us? And what's your hardware resources to get the time results?

[nikitos9000]

@HaoyuanLuo it takes 1.5s for model inference stage + 0.04s for STFT (mel to spectrogram) stage + 6.3s for Griffin-Lim (spectrogram to wav) stage = ~8s to synthesize the sample above (5s of audio) on Tesla M60

My configuration is Tesla M60 with 8Gb memory (I had to limit batch size to 24 to fit the training into GPU)

And the mel outputs, mel outputs postnet and alignment for the sample above are like these, respectively:

[nikitos9000]

@rafaelvalle 1.5s was only the execution time for mel_outputs, mel_outputs_postnet, _, alignments = model.inference(sequence) line, which was re-ran multiple times to get confident numbers.

[rafaelvalle]

Can you put it into perspective and say how long the audio is?

[maozhiqiang]

@nsmetanin how mang steps to alignments! like you show !, I training 5000 steps the mels and alignments like this

[begeekmyfriend]

@rafaelvalle Can you provide python source code for audio synthesis? Jupyter notebook is really awkward for me to use...

@nsakharnykh Your result sounds very like voice of Sarah Kerrigan who has become Queen of Zerg. Really...

[rafaelvalle]

You can get the python code by editing the ipython notebook in a text editor or using github's ipython notebook display feature here

[begeekmyfriend]

@rafaelvalle It generates nothing when I concatenate inference.ipynb into a python file. The only output log shows Warning! Reached max decoder steps and the it is over. The checkout point file is under the output directory. Does it mean it is unable to convergence? By the way I ran the inference code on Ubuntu server without desktop.

[begeekmyfriend]

I used librosa to store the wav file synthesized from checkpoint model of 23K steps. I do not get the alignment image. It seems it failed to convergence on my 10h dataset without augmentation. Well in my experience, it should get to convergence at about 20K steps.

wav = waveform[0].data.cpu().numpy()
librosa.output.write_wav("test.wav", wav.astype(np.float32), hparams.sampling_rate)

eval-23000.zip

[ghenter]

@maozhiqiang @begeekmyfriend Perhaps you should open separate issues on GitHub to dig into the problems each of you are seeing? It feels like alignment/convergence failures and output-generation warnings are separate issues from the current lack of reference audio examples from a system that has been successfully trained on the default data.

[rafaelvalle]

@ghenter Thanks for pointing that out, Gustav. @begeekmyfriend and @maozhiqiang please create a separate issue where we can discuss. There are some things that might improve your training time.

[ghenter]

The wavenet vocoder does make a huge difference, and we’re in the process of open sourcing that as well.

It's exciting to hear that you are working to open-source more of your WaveNet vocoder. However, while the vocoder makes a great difference for signal quality, the Tacotron 2 front-end in this repository is responsible for aspects like pronunciation and prosody.

I think it would be good if everyone (users as well as repo maintainers) would consider posting Griffin-Lim-generated synthetic speech (with matching input text) from their trained systems, so that others know what prosody and pronunciation performance to expect from this code. Sharing the corresponding training curves etc. would be useful, too.

[rafaelvalle]

The pytorch bindings for faster-than-real-time wavenet are here:
https://github.com/NVIDIA/nv-wavenet/pulls
No need to listen to griffin-lim.

[rafaelvalle]

@nsmetanin we're currently making changes to the repo that will considerably reduce the numbers of iteration until the model learns attention.

[rafaelvalle]

@nsmetanin changes have been pushed to master. can you please train again and share the same plots? On LJS and with the default setup the model should learn attention in less than 30k iters.

[ZuoChenFttS]

@rafaelvalle Thanks for your open-source.
By using the default setup, It seems it failed to convergence in my experiment using LJSpeech.
In the repo, it seems never change learning rate. Do you have any experience to share about learning rate? Accoding to the original paper in section 3.1：a learning rate of 10^-3 exponentially decaying to 10^-5 starting after 50,000 iterations. But, in your reply here, you can learn attention in less than 30k iters.(maybe learn attention not means convergence?)
Thanks again.

[rafaelvalle]

FYI: We added an audio example to the readme. You'll need to download the file to listen to it.

[rafaelvalle]

@ZuoChenFttS learning attention does not mean convergence. Learning rate schedules are data and model dependent. If you see spikes in the training loss or the training loss stops going down, resume from an earlier training loss with lower learning rate.
Please post a new issue and we can try to help you.

[begeekmyfriend]

@rafaelvalle The spectrograms of your audio example looks very good. I guess you have not trained with wavenet right? How many iterations did you train out such model and what is the loss value in the end?

[rafaelvalle]

This sample is from a Tacotron 2 model trained for 22500 iters on 8 GPUs.

[begeekmyfriend]

Wow that is a miracle for me to know so little iters needed. I will try it out soon on your demo.

[rafaelvalle]

N.b.: 8 GPUs.

[rafaelvalle]

@begeekmyfriend here's your miracle: ok attention after 8k iterations on a single gpu with batch size 48 :-)
Make sure you pull from master.

[begeekmyfriend]

@rafaelvalle I am glad you made it. But I also expect you can achieve convergence on ~10h small training dataset.

[rafaelvalle]

@begeekmyfriend please give it a try and let us know.

[rafaelvalle]

@begeekmyfriend this ~10h small training dataset you talk about... where is it available?

[begeekmyfriend]

@rafaelvalle My GPU is unavailable in some days. You can cut some of the data of LJSpeech into 10h and try to get convergence within 10K steps. It happens on Keith Ito's version now https://github.com/keithito/tacotron/issues/170#issuecomment-390873725

[rafaelvalle]

I thought you were talking about a specific dataset. We've been able to train on 10h datasets and are currently trying to get a model working on 1h of data.

[begeekmyfriend]

@rafaelvalle Have you tried 5h, 3h or 2h dataset?

[rafaelvalle]

No.

[MXGray]

@rafaelvalle
Hi. Just want to clarify something.
You said the sample WAV is from this repo trained at 22500 iters with 8 GPUs; and
Was batch size 384 (48 x 8), or something else? Please advise. Thanks!

[rafaelvalle]

Closing due to inactivity. Please reopen if necessary.

[ErfolgreichCharismatisch]

Tutorial: Training on GPU with Colab, Inference with CPU on Server here.