Tensorflowtts: FastSpeech2 implementation

[dathudeptrai]

Hi, i just fast implemented FastSpeech2 to check the contribution of F0 embedding and Energy embedding. See PR #45 for detail.

[dathudeptrai]

This is a tensorboard of FastSpeech1 and FastSpeech2. Base on this, F0 and energy embeddings really help. But that is is a case we have a groundtruth F0 and energy, i'm wondering how fastspeech2 adapt with predicted F0/Energy :D.

[superhg2012]

@dathudeptrai why f0 and energy are both integer types?

[dathudeptrai]

f0 and energy is real value, but f0_quantize and energy_quantize is integer. We use quantize value to learn embeddings. But when training, we learn to predict real value f0/energy.

[superhg2012]

@dathudeptrai sorry, I missed some codes in dataset

[dathudeptrai]

I changed some code and now i can reproduce the result of fastspeech2. This is my sample at 80k steps fastspeech2-modify with batch_size 16.

80k_steps.zip

[ZDisket]

@dathudeptrai Sounds great. Does FastSpeech2 train faster than Tacotron2? When will you be releasing that model?

[abylouw]

@dathudeptrai I might be missing something, but I cannot find the AudioDataset class that is supposed to be in TensorflowTTS/tensorflow_tts/datasets/

[dathudeptrai]

@abylouw i will update full code tonight :)), still tuning model

[dathudeptrai]

@superhg2012 can u point out what is the checkerboard artifacts in speech domain ?. @ZDisket soon :))

[dathudeptrai]

This is my audio samples of FastSpeech2 at 100k steps with batch_size 16.
fastspeech2.100k.zip. Those samples in the valid set (i copy it in training set to valid to be able compared with original demo pages.), let compared with original demo pages :D https://fastspeech2.github.io/fastspeech2/. There is still a gap but my fastspeech2 now outperforms Fastspeech1 considerably :D. Let see the performance with further training, fastspeech2 paper use batch_size 48 and training with 160k steps :D.

[abylouw]

Those samples sound good.

This is my audio samples of FastSpeech2 at 100k steps with batch_size 16.
fastspeech2.100k.zip. Those samples in the valid set, let compared with original demo pages :D https://fastspeech2.github.io/fastspeech2/. There is still a gap but my fastspeech2 now outperforms Fastspeech1 considerably :D. Let see the performance with further training, fastspeech2 paper use batch_size 48 and training with 160k steps :D.

[dathudeptrai]

@superhg2012 i didn't meet this problem, i think. BTW, this thread should focus about FastSpeech2 rather than vocoder :D.

[dathudeptrai]

hi all, i just updated FastSpeech2 samples on https://dathudeptrai.github.io/TensorflowTTS/. Compared with Tacotron-2 V1, i think it's comparable. The rest question is, when i will release the pretrained model and the code to reproduce it :D.

[ZDisket]

@dathudeptrai

The rest question is, when i will release the pretrained model and the code to reproduce it

Right now?

[dathudeptrai]

@ZDisket :D, this week :)). BTW, what do you think about the performance ?

[ZDisket]

@dathudeptrai
The performance sounds identical to Tacotron2v1 to me, except for a very slight metallic noise. But I think that may go away with more training. By the way, how well does it align compared to Tacotron2v1? Do you have to keep generating them to get a good result?

[dathudeptrai]

Finish reproduce FastSpeech2 https://github.com/dathudeptrai/TensorflowTTS/tree/master/examples/fastspeech2. The model can be further improve with phonemes and use durations from MFA :)).

[rishikksh20]

@dathudeptrai I have some questions regarding this implementation. As per your commit

* Mean-Var normalization for F0/Energy.
* Calculate Charactor F0/Energy based on its durations.
* Remove outliers of F0/energy.
* Use one Conv1D with large kernel_size as F0/Energy Embeddings
  rather than learning bin embeddings.

Instead of predicting raw F0 and energy, you normalize both and instead of quantizing in bins followed by one hot code as mentioned in paper you apply Conv1d operation to learn embeddings, is any specific reason for that?
Also, why Calculate Character F0/Energy based on its duration and delay update for f0/energy embeddings? Isn't this complicate training process?
I have also implemented FastSpeech 2 and getting some decent output, but if above-mentationed things do improve speech quality or any good specific reason behind them then I will also adapt them into my code.

[dathudeptrai]

I have also implemented FastSpeech 2 and getting some decent output, but if above-mentationed things do improve speech quality or any good specific reason behind them then I will also adapt them into my code.

I think you already know if above thing can improve a quality or not :D. I will explaine below :D.

1. Mean-Var normalization for F0/Energy.

Log or mean/var normalization are ok :)), but when training with log + mixed precision i get nan loss so i used mean-var.

1. Calculate Charactor F0/Energy based on its durations.

When you pass Encoder Hidden States to Length regulator, the feature of each mel belong to each charactor is the same but the F0/energy is different. So when you sum it and pass though Selt attention, the selt attention will bias to F0/energy rather than Encoder Hidden states. Imagine that in this case the length regulator features of each charactor is just a constant :))). For those reasons, the model will depend too much on the f0/energy embedding which we cann't predict good on the test set :)).

1. Use one Conv1D with large kernel_size as F0/Energy Embeddings

Because I think the conv1D with learge kernel_size apply into the real value of f0/energy is better, conv1d can adapt with the noise of f0/energy in the inference process while if you use bins, who know how ur model can adapt with f0/energy predicted ? I saw if f0/energy shift one bin everything is ok, but if it shift >= 2 bin, the audio is not good, even worse than FastSpeech1.

1. delay update for f0/energy embeddings.

The Encoder hidden state features should be have enough infomation to recover mel-spectrogram as FastSpeech1 without F0/energy embeddings. In this case, i think f0/energy embedding is a support features, i want the model learn this correlation so i delay update for f0/energy.

1. but if above-mentationed things do improve speech quality or any good specific reason behind them then I will also adapt them into my code.

Don't forget copy right my ideas and reference this repo :)).

So, everything i did almost because i want to reduce a contributions of f0/energy into a mel-spectrogram reconstruction. And i want to reduce it because i think we cann't have a good f0 in the inference and the model cann't adapt when the predicted f0 shift too much compared with groundtruth :)). BTW, i think the results are comparable with original samples and i'm not a fan of "implement exactly what paper said" since i'm a KAGGLER :)).

[abylouw]

Below are my eval and train losses at 176k steps.

2020-06-23 19:39:49,443 (:165) INFO: (Steps: 176000) eval_duration_loss = 0.0404.
2020-06-23 19:39:49,447 (:165) INFO: (Steps: 176000) eval_f0_loss = 0.3211.
2020-06-23 19:39:49,450 (:165) INFO: (Steps: 176000) eval_energy_loss = 0.0876.
2020-06-23 19:39:49,452 (:165) INFO: (Steps: 176000) eval_mel_loss_before = 0.1978.
2020-06-23 19:39:49,455 (:165) INFO: (Steps: 176000) eval_mel_loss_after = 0.1967.

2020-06-23 19:43:28,363 (:210) INFO: (Step: 176200) train_duration_loss = 0.0079.
2020-06-23 19:43:28,366 (:210) INFO: (Step: 176200) train_f0_loss = 0.0072.
2020-06-23 19:43:28,368 (:210) INFO: (Step: 176200) train_energy_loss = 0.0057.
2020-06-23 19:43:28,371 (:210) INFO: (Step: 176200) train_mel_loss_before = 0.0966.
2020-06-23 19:43:28,373 (:210) INFO: (Step: 176200) train_mel_loss_after = 0.0933.

This is on a relatively small dataset of about 3000 utterances of duration 3.5 hours. I use phonemes and duration from HMM based ASR forced align. The predictions look relatively good, as attached

The synthesized speech is intelligible, but has a metallic vocoding quality (I used the pretrained MB MelGAN for synthesis), my question is, is the metallic quality due to the limited amount of training data?

[dathudeptrai]

@abylouw can you share ur samples, is it better than faspeech1 ?, maybe we need fine-tune mb-melgan

[abylouw]

@abylouw can you share ur samples, is it better than faspeech1 ?, maybe we need fine-tune mb-melgan

I have not trained the particular voice with fastspeech1. Attached are synthesized samples together with GT. It is an Afrikaans voice.
samples.zip

[dathudeptrai]

you should fine-tune MB-melgan too since the dataset is difference. @abylouw

[abylouw]

you should fine-tune MB-melgan too since the dataset is difference. @abylouw

Can you please explain what you mean by fine-tuning?

[dathudeptrai]

@abylouw i mean you load the pretrained mb-melgan then continue training on ur dataset :3.

[abylouw]

@abylouw i mean you load the pretrained mb-melgan then continue training on ur dataset :3.

Okay, thank you

[abylouw]

@abylouw i mean you load the pretrained mb-melgan then continue training on ur dataset :3.

I think I am missing something, can a checkpoint be loaded from just h5 files (the discriminator and generator)? or are there some other required files? I get an error when I try to load from

train.multiband_melgan.v1/checkpoints/ckpt-940000

where I saved the discriminator and generator h5 files.

NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for train.multiband_melgan.v1/checkpoints/ckpt-940000

[dathudeptrai]

@abylouw you can load weight here https://github.com/dathudeptrai/TensorflowTTS/blob/master/examples/multiband_melgan/train_multiband_melgan.py#L415.

generator.load_weights("generator-940000.h5")
discriminator.load_weights("discriminator-940000.h5")

i will fix readme to avoid this confusion :)), ckpt is used to resume and continue to training (it include step, epoch infomation) while h5 is a weight used for fine-tuning (step and epoch = 0.)

[abylouw]

@abylouw you can load weight here https://github.com/dathudeptrai/TensorflowTTS/blob/master/examples/multiband_melgan/train_multiband_melgan.py#L415.

generator.load_weights("generator-940000.h5")
discriminator.load_weights("discriminator-940000.h5")

i will fix readme to avoid this confusion :)), ckpt is used to resume and continue to training (it include step, epoch infomation) while h5 is a weight used for fine-tuning (step and epoch = 0.)

Thank you. Does the training then follow the normal procedure, i.e. up to 200000 steps generator and from there generator + discriminator?

[dathudeptrai]

Yes normal :)))

[apsears]

What are your thoughts on the increasing eval loss? Should we change the loss to something else? It is clear that longer training improves voice quality, but the metric does not show that.

[dathudeptrai]

@apsears you mean fastspeech2?

[apsears]

Yes, for fastspeech2 -- except for the first few thousand steps, eval loss is increasing. In most projects, that means you stop early, but we all know that's not a good idea. Is this unique for FastSpeech2 here, or does this happen for other models in the repo as well? Is this a consequence of the limitations of the non-autoregressive approach?

[dathudeptrai]

@apsears the loss has no meaning with fastspeech2 right now. The reason why eval loss is increasing (but not too much) is because we cann't predict a good F0/Energy in the real inference so the mel value will be different. Example if the groundtruth mel is Y while ur prediction is 1.5Y (cause by energy predicted is higher than grountruth energy) then the audio generated still good but the loss will be high. But it's not a problem, you can hear my audio samples on fastspeech2, it's really good compared with tacotron-2, the shape of the mel clearer than fastspeech1.

[LLianJJun]

Hi~
I am training fastspeech2 using my data.
The amount of training data is about 100 hours.
On Tesla P40 (Memory ~22G)
It has been around 24 hours since started training.

The picture below is the current tensortbord capture.
Need more training yet? (currently 170k steps.)
When I check the synthesized sound, the quality is slightly lower,
but I don't know if the cause is fastspeech2-model or mb-melgan-model.

[dathudeptrai]

@LLianJJun can you provide a mel predicted on the real inference ? I mean the figure :) and did you finetune mb_melgan or use pretrained model?

[LLianJJun]

@dathudeptrai

It is a mel predicted figure on the real inference.
I didn't train using the pretrained model mb_melgan, It is 900k steps.
It is other language (not English.)

The voice is great overall. However, husky voices are generated from certain words.

[dathudeptrai]

@LLianJJun There are some things you can do to improve ur result:

1. Ur dataset seems too large compared with LJSpeech so i think futher training will improve it
2. Change delay_f0_energy_step to 1 in https://github.com/TensorSpeech/TensorflowTTS/blob/master/examples/fastspeech2/conf/fastspeech2.v1.yaml#L69. and hard code https://github.com/TensorSpeech/TensorflowTTS/blob/master/tensorflow_tts/models/fastspeech2.py#L122 to tf.constant(1.0) to make sure it work correctly :)).

Ur mel looks good but i think it can be better :))).

[janson91]

hi,
How can I generate duration_files?
Before I run python examples/fastspeech2/train_fastspeech2.py, I have run tensorflow-tts-preprocess, tensorflow-tts-compute-statistics, tensorflow-tts-normalize for Preprocessing.

Thank you

[dathudeptrai]

@janson91 you need extract duration from tacotron2 model or other tool :))). see https://github.com/TensorSpeech/TensorflowTTS/tree/master/examples/tacotron2#step-4-extract-duration-from-alignments-for-fastspeech

[LLianJJun]

@dathudeptrai
Thank you for your comment. ^^

[janson91]

@janson91 you need extract duration from tacotron2 model or other tool :))). see https://github.com/TensorSpeech/TensorflowTTS/tree/master/examples/tacotron2#step-4-extract-duration-from-alignments-for-fastspeech

Thank you for your reply. After I extract duration, there is a new problem when I try to train fastspeech2 by
https://github.com/TensorSpeech/TensorflowTTS/tree/master/examples/fastspeech2#step-2-training-from-scratch

TypeError: examples/fastspeech2/train_fastspeech2.py:130 _one_step_fastspeech2 *
TypeError: apply_gradients() got an unexpected keyword argument 'experimental_aggregate_gradients'

Keyword argument 'experimental_aggregate_gradients' of apply_grandients() should be boolean type according to
https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Optimizer#apply_gradients
but here are :

130:self.optimizer.apply_gradients(zip(gradients, self.model.trainable_variables), 5.0)

My tensorflow version is 2.2 and python is 3.6.

Thank you

[dathudeptrai]

@janson91 my optimizer here https://github.com/TensorSpeech/TensorflowTTS/blob/master/tensorflow_tts/optimizers/adamweightdecay.py#L124. Can you try modifty to:

return super(AdamWeightDecay, self).apply_gradients(zip(grads, tvars), **kwargs)

BTW, pls pull newest code :D.

[abylouw]

@janson91 you need extract duration from tacotron2 model or other tool :))). see https://github.com/TensorSpeech/TensorflowTTS/tree/master/examples/tacotron2#step-4-extract-duration-from-alignments-for-fastspeech

Thank you for your reply. After I extract duration, there is a new problem when I try to train fastspeech2 by
https://github.com/TensorSpeech/TensorflowTTS/tree/master/examples/fastspeech2#step-2-training-from-scratch

TypeError: examples/fastspeech2/train_fastspeech2.py:130 _one_step_fastspeech2 *
TypeError: apply_gradients() got an unexpected keyword argument 'experimental_aggregate_gradients'

Keyword argument 'experimental_aggregate_gradients' of apply_grandients() should be boolean type according to
https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Optimizer#apply_gradients
but here are :

130:self.optimizer.apply_gradients(zip(gradients, self.model.trainable_variables), 5.0)

My tensorflow version is 2.2 and python is 3.6.

Thank you

@janson91 I got the same error when I used ---mixed_precision 1, with --mixed_precision 0 everything worked as expected

[dathudeptrai]

@janson91 @abylouw doing as my mentenioned above fixed this problem :)) see my commit https://github.com/TensorSpeech/TensorflowTTS/commit/4d725deec866ffb8aafcccc025c41de3507080ad

[janson91]

@janson91 you need extract duration from tacotron2 model or other tool :))). see https://github.com/TensorSpeech/TensorflowTTS/tree/master/examples/tacotron2#step-4-extract-duration-from-alignments-for-fastspeech

Thank you for your reply. After I extract duration, there is a new problem when I try to train fastspeech2 by
https://github.com/TensorSpeech/TensorflowTTS/tree/master/examples/fastspeech2#step-2-training-from-scratch
TypeError: examples/fastspeech2/train_fastspeech2.py:130 _one_step_fastspeech2 *
TypeError: apply_gradients() got an unexpected keyword argument 'experimental_aggregate_gradients'
Keyword argument 'experimental_aggregate_gradients' of apply_grandients() should be boolean type according to
https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Optimizer#apply_gradients
but here are :
130:self.optimizer.apply_gradients(zip(gradients, self.model.trainable_variables), 5.0)
My tensorflow version is 2.2 and python is 3.6.
Thank you

@janson91 I got the same error when I used ---mixed_precision 1, with --mixed_precision 0 everything worked as expected

It worked, thx

[janson91]

@janson91 @abylouw doing as my mentenioned above fixed this problem :)) see my commit 4d725de

Thank you, but maybe my graphics card don't support mixed_precision.
https://forums.developer.nvidia.com/t/issue-about-no-suitable-gpus-detected-when-using-mixed-precision-graph-optimizer/75458

[LLianJJun]

@dathudeptrai
As you advised, changing the hardcoded values ​​improved quality. (delay_f0_energy_step, self.is_use_f0_energy)
I am wondering why the values ​​are hardcoded. Is there any other reason?

[dathudeptrai]

@LLianJJun in fact you don't need hard code =)). it should work, let see the code here https://github.com/TensorSpeech/TensorflowTTS/blob/master/examples/fastspeech2/train_fastspeech2.py#L91. Just want to sure =))). If delay_f0_energy_step = 1 is the best config, i will remove delay_f0_energy_step and is_use_f0_energy :D. BTW can you share ur sample, and ur mel-spectrogram image :D.

[l4zyf9x]

@LLianJJun Your eval mel_loss_after is pretty good. Do you use ground truth f0 energy, or predicted f0 energy in eval step? And how do you extract durations?

[dathudeptrai]

@LLianJJun Your eval mel_loss_after is pretty good. Do you use ground truth f0 energy, or predicted f0 energy in eval step? And how do you extract durations?

Hmm, eval_mel_loss use f0/energy grountruth in my implementation but the important is that his duration/f0/energy loss really good, there is no overfit problem so the exact value loss on evaluation is not important here. I just want to see his mel-predicted plot and an autio samples to prove that my implementation work well on other languages/datasets. The performance on ljspeech also proved :)).

[janson91]

@janson91 you need extract duration from tacotron2 model or other tool :))). see https://github.com/TensorSpeech/TensorflowTTS/tree/master/examples/tacotron2#step-4-extract-duration-from-alignments-for-fastspeech

I want to train fastspeech2 model on other languages, and it seems that I need to train tacotron2 model first.
But the training speed of tacotron2 is very slow.
So can I get the duration in other ways, or can I extract duration from an undertrained tacotron2 model.
Thank you

[dathudeptrai]

@janson91 hmm, that is the thing we are trying to solve. We are implementing alternative model to learn alignment. Right now you can extract duration from MFA (https://montreal-forced-aligner.readthedocs.io/en/latest/) or Tacotron-2. Training tacotron-2 is slow, it need train around 2-3 days on 2080TI to get a good duration for fastspeech2 :)).

[janson91]

@dathudeptrai ok, thank you. Btw, if the duration is extracted from Tacotron-2, would it be better? Or how different are the effects of the two methods?

[dathudeptrai]

@janson91 i think MFA is better but it's not support for all languages :)).

[ZDisket]

@janson91 @dathudeptrai MFA+FS2 performs extremely well for me in another repo, I'm interested in how it'd be here.

[dathudeptrai]

@ZDisket we know that MFA, phoneme can improve the quality. The reason why i'm not yet support phoneme because i don't have much time :). And there is a reason why training is on examples folder, that is because it's just examples use case of this framework. Now this framework support tflite, that mean it's end2end process, from text to production :)). Phoneme or MFA is the only step that user should do it by theirself, i don't want our framework become something like CookBook. Anyway, PRs are always welcome :)), tell me if you want to support phoneme and MFA for this repo :)) @ZDisket

[ZDisket]

@dathudeptrai How long did it take you to train your FastSpeech2 model? I can probably glue enough code together to make it happen.

[dathudeptrai]

@ZDisket 12 hours :D, you can make the Pull Request, i can train :D

[ZDisket]

@dathudeptrai I can train it myself.

[dathudeptrai]

@ZDisket ok, can you make the PR, so i can review and code with you faster :'(.

[ZDisket]

@dathudeptrai I will when I have something. By another repo, I meant ming024's implementation. It performs perfectly on every dataset from my dataset creation tool.

[dathudeptrai]

@ZDisket i knew, can you share ur sample ?. To be honest, this pytorch impelementation (https://github.com/rishikksh20/FastSpeech2) is better than the repo you are using IMO.

[ZDisket]

@dathudeptrai Audio samples? I got something better, a notebook

[dathudeptrai]

@ZDisket i don't know if the audio samples of ming024's is good or not interm of quality. He use pretrained nvidia waveglow which is very large model and performance is very good and that is why a audio sample may smoother. But IMO, the mel is not good, the audio sample has no stress, i mean intonation. Maybe you want to try https://github.com/rishikksh20/FastSpeech2.

[ZDisket]

@dathudeptrai In my experience, more training when finetuning often makes the models sound more like their speakers, but also it's very easy to overfit. Here's a video to compare intonation.
But MFA+FS2 performs very well when compared to Tacotron2. The model that's loaded by default in the notebook? I had given up on training Tacotron2 on that dataset due to fatal alignment problems. Rishikksh20's implementation looks nice, but trying to set up training and synthesis would be a waste of time which I could use on other things.

[dathudeptrai]

@ZDisket looks good :D. i will try to use MFA and phoneme asap.

[ZDisket]

@dathudeptrai

i will try to use MFA and phoneme asap.

So do I leave the implementation up to you? Or still want help?

[dathudeptrai]

@ZDisket still want to help :( i need implement some module before try to train mfa and phoneme

[ZDisket]

@dathudeptrai Then I'll start working on it.

[dathudeptrai]

@ZDisket thanks very much

[abylouw]

I have used phonemes as input and the durations are from an HTK forced alignment system. The quality is better than our other DNN based systems.

I think using phonemes as input one can get away with less data. My first voice I trained with FS2 was with 3.5 hours.

[ZDisket]

@dathudeptrai Does your FastSpeech2 take durations in seconds or frames?

[abylouw]

@dathudeptrai Does your FastSpeech2 take durations in seconds or frames?

The durations are in frames.

[ZDisket]

@abylouw I see, how do I convert seconds to frames? I think it was multiplying the seconds by (sampling_rate/hop_length)

[abylouw]

@abylouw I see, how do I convert seconds to frames? I think it was multiplying the seconds by (sampling_rate/hop_length)

That is correct, sampling_rate/hop_length x frame_length_in_sec

[l4zyf9x]

@abylouw I guess converting from seconds to frames, you must round it after calculating duration=sampling_rate/hop_length x frame_length_in_sec. So how do you ensure time resolution between total frame and sum of durations?

[abylouw]

@abylouw I guess converting from seconds to frames, you must round it after calculating duration=sampling_rate/hop_length x frame_length_in_sec. So how do you ensure time resolution between total frame and sum of durations?

That is correct, I round after the calculation. To get the number of frames the same (or very close) I used hop_length/sampling_rate as the target rate in the forced alignment system. Therefore, the frame rate in the forced alignment system is the same as the FS2 system. There might still be one or two frames different, then I pad or drop frames like is done for f0 here:

https://github.com/TensorSpeech/TensorflowTTS/blob/a9a48a7573d9712b6341e06d5f3e64eaa26fe747/tensorflow_tts/bin/preprocess.py#L251-L254

Afterwards I added an assert to make sure that the sum of the duration frames equals mel, energy and f0 by modifying here:

https://github.com/TensorSpeech/TensorflowTTS/blob/a9a48a7573d9712b6341e06d5f3e64eaa26fe747/tensorflow_tts/bin/preprocess.py#L261

[dathudeptrai]

I think the FastSpeech2 implementation now stable, let move forward to MFA + phoneme implementation :D