Tensorflowtts: Can I retrain only FastSpeech or Tacotron without retraining MB-MelGAN?

[seicaratteri]

Hello! First of all, thank you so much for the work: it's incredibile! 🥇

I want to retrain the system on an a different voice, with hours of recordings I will make. In order to achieve so, I was thinking of retraining FastSpeech2 / Tacotron. I have a doubt: do I need to re train as well MB-MelGAN? or does it just work as universal vocoder?

One last question: is there a general better way to train a new voice? Maybe fine-tuning over LJSPEECH, for example, rather than training from scratch?

Thank you in advance! :)

[OscarVanL]

Yes, I asked a similar question and the maintainers told me that it's not necessary to train a MelGAN model if you use one that has been trained on many speakers. This becomes a universal vocoder and can synthesise speech from any person.

See the first few replies on my question here.

Specifically, on this page, the model at the bottom multiband_melgan.v1_24k is a good Universal Vocoder.

[seicaratteri]

Hello @OscarVanL, first of all thanks a lot for your answer! I was looking at the issue you posted but I still had the doubt so that's why I asked :) thank you a lot for clarifying!

Regarding the other question, do you know (or does anybody) if it's better to train a model from scratch (given that I will have around 24h of studio recordings) or if I should fine tune over one of the existing ones?

And one last question: when I hear or I try to generate samples with fast speech 2 + mb-melgan there is always some metallic artifact noise where the pauses of the speech are, and this does not happen in the examples provided in the official paper. Does it require more training to get rid of these or do I need to use another vocoder? Thanks a lot in advance! :)

[ZDisket]

@seicaratteri If you have a lot of data then it's better to train from scratch. And I believe the FastSpeech2 model is responsible for the metallic artifacts on pauses, as I never had them on my mb-melgan eval samples.

[seicaratteri]

Thanks a lot @ZDisket for your answer! 🥇 I still have a doubt about fast speech 2: by listening to the samples here https://fastspeech2.github.io/fastspeech2/ ( the official website of the paper ) there are none of these artifacts. What could be the cause of if it is not the vocoder?

[dathudeptrai]

Thanks a lot @ZDisket for your answer! I still have a doubt about fast speech 2: by listening to the samples here https://fastspeech2.github.io/fastspeech2/ ( the official website of the paper ) there are none of these artifacts. What could be the cause of if it is not the vocoder?

both fs2 and vocoder can make a metallic artifacts :)))

[seicaratteri]

Thanks a lot @ZDisket for your answer! I still have a doubt about fast speech 2: by listening to the samples here https://fastspeech2.github.io/fastspeech2/ ( the official website of the paper ) there are none of these artifacts. What could be the cause of if it is not the vocoder?

both fs2 and vocoder can make a metallic artifacts :)))

Thank you very much for your answer! So you would say that the reason for no metallic artifacts in the samples from the original paper is that they used Parallel WaveGAN as vocoder rather than MB-MelGAN? :)

[ronggong]

@seicaratteri maybe you could try the espnet2 implementation https://colab.research.google.com/github/espnet/notebook/blob/master/espnet2_tts_realtime_demo.ipynb, they have parallel wavegan implemented. Parallel wavegan seems to me has a better speech quality than the MB-MelGan, but also slower.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.