Tensorflowtts: Long sentences issue with FS2

[dathudeptrai]

seem my fastspeech2 implementation can't handle long sentence in some dataset such as KSS. FOr Ljspeech and other dataset from other person report that it's still fine. I'm thinking about the maximum length in the training set that my FS2 need to be able to handle long sentences. In my private dataset, it always fine. Maybe 15s is enough.

[dathudeptrai]

I think flow/glow model handle long sentence better than fastspeech. I heard some people got very good model from Glow-tts https://github.com/jaywalnut310/glow-tts even better than tacotron2 and handle long sentences very good on single speaker but not multi-speaker (didn't work for multi speaker). I think this glow-tts + our mb-melgan can get very robust end2end model :D. Has anyone to combine this glow-tts and our mb-melgan ?

@Joovvhan i think you are trying to combine glow-tts + mb-melgan, can you share some insight ?.
cc: @machineko :)))

[ZDisket]

@dathudeptrai I tried that Glow-TTS and while it does handle long sentences flawlessly (I can share notebooks if you want), I can tell from the generated audio that it sounds like it's been split up then audio spliced back together, not a trick one can't do manually(I can share a demo that uses FastSpeech2, PWGAN, and this method). I personally still recommend Tacotron2-DDC: https://erogol.github.io/ddc-samples/, which can naturally align a 1 minute and 49 second input.

[dathudeptrai]

@ZDisket the very very very long sentence is not a real problem on deploy TTS. We just need the model can synthesis around 15-20s (this is the fact when you try to deploy ur model with low latency, this is the end step, the step to make money :v). Actually, our tacotron model can handle very very long sentence as tacotron2-ddc does without ddc trick (you can try on the colab.). i'm still wondering about glow-tts + mb-melgan audio quality :D.

[machineko]

@Joovvhan i think you are trying to combine glow-tts + mb-melgan, can you share some insight ?.
cc: @machineko :)))

I've tried by only with waveglow and audio quality was a lot better than FS2 but for single speaker, multi-speaker can't coverage in my case

[tekinek]

Maybe glow-tts is more sensitive to dataset quality compare to Tacotron or Fastspeech. I guess it could be a very nice addition to TensorflowTTS if the author's claim is true for most of the single speaker datasets. I yet to try it with my dataset because of busy GPU for a while.

It seems Mozilla TTS is working on glow-tts, too.
https://github.com/mozilla/TTS/issues/500#issue-680150289

[machineko]

Maybe glow-tts is more sensitive to dataset quality compare to Tacotron or Fastspeech.

Dunno it was training fine on single speaker from same dataset :)

[dathudeptrai]

Maybe glow-tts is more sensitive to dataset quality compare to Tacotron or Fastspeech. I guess it could be a very nice addition to TensorflowTTS if the author's claim is true for most of the single speaker datasets. I yet to try it with my dataset because of busy GPU for a while.

It seems Mozilla TTS is working on glow-tts, too.

https://github.com/mozilla/TTS/issues/500#issue-680150289

I hope it is a fork rather than intergrated glow tts to mozilla since the model is the same :))) and mozilla use pytorch too :3. BTW, reproduce glowtts on tensorflow is hard, very hard. Maybe just need write a function to make glowtts work with our vocoder is enough. Glowtts code is official from author so i think it should be the best implementation interms of quality.

[dathudeptrai]

Maybe glow-tts is more sensitive to dataset quality compare to Tacotron or Fastspeech.

Dunno it was training fine on single speaker from same dataset :)

yes, literally, glow/flow is sensitive with dataset since it is designed based on pure math (probability) :))).

[machineko]

@dathudeptrai Yes I know :D Just saying dunno if its cause of my dataset its pretty good in quality and single speakers are working fine :)

[tekinek]

BTW, reproduce glowtts on tensorflow is hard, very hard. Maybe just need write a function to make glowtts work with our vocoder is enough. Glowtts code is official from author so i think it should be the best implementation interms of quality.

@dathudeptrai
This is nice:) save effort for unavailables.

Implementing and training a new model is costly especially in terms of time, so it is good to see portings between repos. I had a hard time to make my mozilla mb-melgan work with TensorflowTTS FastSpeech due to pre-emphasis was applied to waveform in vocoder but not in FS here. Actually, preemphasis=0.98 makes my speech more clear. I recommend its addition to this repo.

[ZDisket]

@tekinek Do you have any experience using Mozilla/TTS mel generation models with TensorflowTTS vocoder?

[tekinek]

@ZDisket Not yet. TensorflowTTS vocoder fails with my dataset (discriminator does not learn, I don't know if it is solved in recent updates), but Mozilla vocoder is just fine. So I wanted to use vocoder from Mozilla and FastSpeech from TensorflowTTS.

[ZDisket]

@tekinek

TensorflowTTS vocoder fails with my dataset (discriminator does not learn, I don't know if it is solved in recent updates)

You can always train kan-bayashi's repo then convert weights, that's what I'm doing for mb-melgan currently.

[tekinek]

@ZDisket Thanks, I will try it. Mozilla Tacotron2-DDC really makes good speech, but, with my dataset, works just until reduction factor 2. Let me see how Glow-tts works.

[ZDisket]

@tekinek

but, with my dataset, works just until reduction factor 2

I'm interested. Can you explain in detail?

[dathudeptrai]

@tekinek

but, with my dataset, works just until reduction factor 2

I'm interested. Can you explain in detail?

maybe with r = 1 it's failed.

[dathudeptrai]

@tekinek I think kan's vocoder is the most stable implementation (note that mozillatts is still use kan's code.), i just feel tensorflow is not yet stable as pytorch. It's weird when the implementation work with all my dataset, kss, baker, libritts, ljspeech but didn't work on ur dataset and @ZDisket dataset. I recommend you use kan's repo to training and convert it to this tensorflow implementation to faster training and ez to deploy :)). Parallelwavegan is supported and you can convert weight right now in the parallel_wavegan branch :D.

I really want to fix all things related with tensorflow but since i can't reproduce the bug discriminator can't learn so i can't help, :(.

[ZDisket]

@dathudeptrai Well, it could be because I always finetune, never train from scratch (except for FastSpeech2). Combining the pwgan discriminator with multi-band melgan generator might be a worthy investment.
Although training from kan's repo is showing promising results (afteredit: it's bad)

[tekinek]

@ZDisket Ok, I am still lacking insights; this is just my observation.

In my case, tactron2-ddc does gradual_training by [[0, 7, 64], [1, 5, 64], [50000, 3, 32], [150000, 2, 32], [200000, 1, 32]], which means, you know, the reduction factor reduces at certain steps gruadually.

Look at my tensorborad, in which every big drop in decoder loss happens when reduction rate decreases by one (sorry, these colors are for restored trainings with some dataset tweaks and change in gradual_training settings). But as soon as reduction rates drops to 1 at 200k, stopnet loss increase suddenly and hardly going back normal. In test synthesis, speech becomes slower and alignment breaks for some cases. However, training it until more than 350k ultimately makes it better, but I am still uncertain how good.

Hope this helps.

[tekinek]

@dathudeptrai
As for the FS2 "issue" with long sentence, it seems that FS2 needs training with long sentences to synthesize longer speech stably. With my dataset, which doubles the average sentence length of KSS as shown in https://github.com/TensorSpeech/TensorFlowTTS/pull/130#issuecomment-668990753, FS2 is just fine.

My recommendation for KSS+FS2 is to try fine-tuning with an augmented KSS dataset, like artificially generate long utterances by concatenating two sentences with ";" and corresponding audio files with a short silence. Recently I have realized that many long sentences in my datasets are indeed not sentences but short paragraphs where commas were incorrectly used for real sentence boundaries. This makes another problem during inference that Tacotron2 model somehow becomes confused when it encounters a comma in a sentence. In some cases, model just stops there or make too long silence. (FS2 seems fine in such cases)

[ZDisket]

@tekinek You still having trouble training vocoders? See if the pwgan and mb-melgan hybrid that was just merged into https://github.com/TensorSpeech/TensorFlowTTS/pull/197 do the job, it can be used to finetune on regular mb-melgan models.

[tekinek]

@ZDisket thanks for this information; I will give it a try. I've just started training mb-melgan from Kan Bayashi's repo:

[ZDisket]

@tekinek Alrighty, I'll be interested in your results since I got a way better result than kan's repo with just 10% of the training time by fine tuning the mb-melgan generator on my dataset

[tekinek]

@ZDisket And here is mb-melgan from Mozilla's repo. Eval speech is fine so far at 700k, but I am still unsatisfied. Did pwgan generage better speech in your experiment? (though slower). I've never tried fine-tuning. I think there is no open pretrained vodoer model nor text2mel for my language, and have no idea how fine-tune a vocoder on different dataset (with different language) performs.

I think these vocoders are too sensitive to factors like noise in dataset, fmin, fmax, pre-emphasis etc, which makes it difficult to find a good one.

[dathudeptrai]

@ZDisket And here is mb-melgan from Mozilla's repo. Eval speech is fine so far at 700k, but I am still unsatisfied. Did pwgan generage better speech in your experiment? (though slower)

I think these vocoders are too sensitive to factors like noise in dataset, fmin, fmax, pre-emphasis etc, which makes it difficult to find a good one.

you can training mb-melgan G + pwgan D as @ZDisket did in his experiment. I do not know if there is a difference between mozilla's mb-melgan and kan's mb-melgan since mozilla mb-melgan implementation is based on kan's repo (i just glance the code and there is no difference between the model or training process).

[ZDisket]

@tekinek

Did pwgan generage better speech in your experiment? (though slower)

PWGAN didn't really generate better speech according to my ears, I just used to favor it because of its more stable training. Note that I never (successfully, tried "fast-training" WaveRNN implementation and was disappointed) trained a vocoder from scratch, always finetune (I have no patience)

I've never tried fine-tuning. I think there is no open pretrained vodoer model nor text2mel for my language, and have no idea how fine-tune a vocoder on different dataset (with different language) performs.

That shouldn't be an issue. Vocoders turn mel spectrograms, which look pretty similar for human speech regardless of language, into audio. I've successfully used the pretrained WaveGlow model from NVIDIA for speech on a Spanish Tacotron 2 speaker (I'll train Spanish FS2 when I find time), there's no reason why finetuning shouldn't be able to adapt.

@dathudeptrai

do not know if there is a difference between mozilla's mb-melgan and kan's mb-melgan since mozilla mb-melgan implementation is based on kan's repo

According to the DDC post, his uses a different discriminator:

MB-Melgan Vocoder with Multiple Random Window Discriminator
As a vocoder, I use Multi-Band Melgan [11] generator. It is trained with Multiple Random Window Discriminator (RWD)[13] different than the original work [11] where they used Multi-Scale Melgan Discriminator (MSMD)[12].

The main difference between these two is that RWD uses audio level information and MSMD uses spectrogram level information. More specifically, RWD comprises multiple convolutional networks each takes different length audio segments with different sampling rates and performs classification whereas MSMD uses convolutional networks to perform the same classification on STFT output of the target voice signal.

In my experiments, I observed better RWD yields better results with more natural and less abberated voice.

[tekinek]

Hi @ZDisket Thanks. It seems fine-tuning should be preferred in most cases of training a vocoder.

My training of mb-melgan from Kan Bayashi's repo is ready. Is there any instruction on how to use it with TensorflowTTS FS2? so that I could save some effort.

[ZDisket]

@tekinek

Is there any instruction on how to use it with TensorflowTTS FS2? so that I could save some effort.

I recall there was a notebook to convert weights from kan's repo to this one so you can use it like it was a model trained here, although I lost it(I think @dathudeptrai has it) because I discovered and switched to training multiband pwgan, which is able to give good results after just 500 steps of finetuning and I usually mostly train it to 5 or 50k: https://github.com/TensorSpeech/TensorFlowTTS/tree/master/examples/multiband_pwgan#finetune-multi-band-melgan--pwgan-disc-with-ljspeech-pretrained-on-other-languages

You can hear some audio samples and here (somewhat bottlenecked by attention problems)

[tekinek]

@dathudeptrai sorry, but do you still have that notebook to convert weights from Kan's mb-melgan to this repo? and does it still work?

[dathudeptrai]

i created pull request for that notebook a long time ago in Kan's repo. Pls try it first (https://github.com/kan-bayashi/ParallelWaveGAN/blob/master/notebooks/convert_melgan_from_pytorch_to_tensorflow.ipynb). If it didn't work, pls tell me, i will help.

[tekinek]

Hi @dathudeptrai
I've tried this notebook to convert Kan's mb-melgan. But I see that converted tensorflow model does not match the original one, and synthesized speech (mel is from my Tac2 and FS2) is nothing but a noise.

FYI: My mb-melgan config is same as:
https://github.com/kan-bayashi/ParallelWaveGAN/blob/master/egs/ljspeech/voc1/conf/multi_band_melgan.v2.yaml

[dathudeptrai]

Hi @dathudeptrai
I've tried this notebook to convert Kan's mb-melgan. But I see that converted tensorflow model does not match the original one, and synthesized speech (mel is from my Tac2 and FS2) is nothing but a noise.

FYI: My mb-melgan config is same as:
https://github.com/kan-bayashi/ParallelWaveGAN/blob/master/egs/ljspeech/voc1/conf/multi_band_melgan.v2.yaml

give me a colab.

[tekinek]

@dathudeptrai sorry, here I cannot access colab:(

[dathudeptrai]

@dathudeptrai sorry, here I cannot access colab:(

so let give me a notebook :)).

[tekinek]

@dathudeptrai please check your e-mail.

[dathudeptrai]

@dathudeptrai please check your e-mail.

sorry, did you send "[email protected]" ?

[tekinek]

@dathudeptrai yes, got it from repo page.

[dathudeptrai]

@dathudeptrai yes, got it from repo page.

Can you re send ?

[crux153]

@dathudeptrai Here's some progress update.

I tried training Tacotron2 and FastSpeech2 using another dataset. (It requires a Korean form to be submitted in order to get access, and redistribution is forbidden) It consists of 6,500 utterances and has length of around 9 hours, which is smaller than KSS but contains more longer samples.

With linformer disabled (v1), new FS2 model synthesized long samples without problem!

However when linformer is used with v2, same issue occurred again but it sustained longer than model trained with KSS. KSS model failed to synthesize words from around 7-8 seconds, while the new model had problem around 12-13 seconds. Maybe this is somehow related with maximum length of sentence in train dataset?

Above two mels shows same sentence synthesized with v1 and v2. See how it gets weaker when linformer is applied.

| | KSS | New Dataset |
| --------------------- | -----| ------------- |
| v1 (without linformer) | ❌ | ✅ |
| v2 (with linformer) | ❌ | ❌ |

I guess maybe adjusting hparams like encoder/decoder hidden size may help solving this problem. I'm going to try fine-tuning from LJSpeech pretrained first, and will try tuning hparams if it doesn't work.

Anyway, here's my Tensorboard with new dataset.

[Hongpeng1992]

it seems glowtts do not work better than fastspeech2 . I will upload the chinese version glowtts soon

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.