Tensorflowtts: Suggestion add universal vocoder

[machineko]

It can be hard to test mel-generated from models without universal vocoder.

You can pin me to this issue when I've got some free time I'll check https://github.com/kan-bayashi/ParallelWaveGAN and see which smaller models can be considered universal vocoders and if mb-melgan can be then I'll try to train this repo version else we should consider adding smth like Waveglow or SqueezeWave(?).

[ZDisket]

@machineko The VCTK model from converted weights is already a pretty good universal vocoder in my experience, although most of my tests are on bad models since this repo's FastSpeech2 isn't as resilient against bad datasets as ming024's. Although a mb-melgan model based on the clean clips in LibriTTS would be great, but that will take a lot of compute.

[dathudeptrai]

@machineko You can test with Griffin-Lim on this repo to sanity check. I also convert the mb-melgan VCTK weight from kan's repo to this implementation with some trick, if you like i can share :D.

@ZDisket I think fine-tune VCTK pretrained on target dataset is better than just using it :D. The problem with fastspeech2 is that the model overfits energy/f0, this problem can be solved to add more training data. As we know, for ljspeech, even it overfit but it's still so good compared with tacotron-2. It also generates good mel for korea/chinese dataset. It's a pros but also a cons :)). If your dataset is enough and good, everything ok, you can get the fastspeech2 model that even better than tacotron-2 (my private experiment). But if ur data is not enough, the model overfit too much, then the generate mel is not good when inference test data. So I think we should modify a f0/energy prediction module to adapt to a bad dataset. This is a thing I missed since all my experiment is not yet with a bad dataset. In the future, I will try to train with a noise dataset and consider that as my challenge :D.

[ZDisket]

@dathudeptrai

I think fine-tune VCTK pretrained on target dataset is better than just using it

I would do that if it didn't become pure noise after 10k steps, although it might be my dataset being small. As for FastSpeech2, can you check what he did with his implementation? I can give you a dataset, models and audio samples.

[dathudeptrai]

@ZDisket You can easily see in his tensorboard that the f0/energy loss is so high, that is because he used un-normalize value to train I think :D. And also he uses MAE for f0/energy predictor, not MSE (u can try my implementation with MAE for f0/energy), I didn't try because MSE can generate a good result, but maybe that can help you. The important thing i think it that, how the model adapts with noise f0/energy in training. LJSpeech, chinese/korea dataset are a good dataset so f0/energy is good enough, but ur dataset, I'm not sure about that. I will try to train ur dataset in the near future since that is a funny dataset ^^. BTW, is that ur only dataset, you want to use that dataset for ur product or you have other datasets?

[ZDisket]

@dathudeptrai I have many datasets, but I have sent you the one I'm talking about by email.

[machineko]

Dunno if fs2 have problems with overfit on my fork i have not seen this.
In my experiments f0/energy isnt overfitting also u can easily do a lot of stuff to stop it like changing loss weights, init on pretrained model and just fine tune or try Double descent curve training https://arxiv.org/pdf/1812.11118.pdf
I remember reading few newer papers about overfiting and there is a lot to learn on this still :P

[dathudeptrai]

@machineko how large your dataset is?. and can you share ur samples?

[machineko]

@dathudeptrai ~9000 examples can't share but will upload multi-speaker MFA pull request in few h and u can retrain it on small subset of Libri/VCTK my first training was on ~1.9k examples and still didn't overfit on f0/energy.

[machineko]

@dathudeptrai Im back from work stuff and ready to train libritts would love to get this VCTK mb-melgan to add to logger :)

[dathudeptrai]

@machineko thank :D. Looking for ur PR :D

[machineko]

@dathudeptrai " I also convert the mb-melgan VCTK weight from kan's repo to this implementation with some trick, if you like i can share :D" -> Can u share :P?

[dathudeptrai]

@machineko ur email ?, BTW, to make the conversion work, i need modify the structure a bit, i hope we will have a mb-melgan trained with this repo. But we can use the converted model to check the performance of universal vocoder :D.

[machineko]

@dathudeptrai [email protected] also https://docs.github.com/en/github/building-a-strong-community/about-team-discussions maybe we can add this to repo for better comunication :)

[dathudeptrai]

@machineko i know that :)) but let wait a bit :)), when the right time come :)) i will do that (soon !! ).

[machineko]

there is also https://github.com/vercel/vercel/discussions/3874 but its in beta

[machineko]

I've tried vctk_multi_band_melgan.v2 from ParallelWaveGAN and it works awful compared to waveglow (next run i'll retrain model using 0-8000hz freq config to push output of models through waveglow and see results)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.