Tacotron-2: As for Griffin Lim vocoder, the 2nd generation performs worse than the 1st generation

[begeekmyfriend]

@Rayhane-mamah I know you have substituted CBHG for convolution for the post net. But the evaluation results sound no better than the 1st generation. There is analysis from @rafaelvalle https://github.com/keithito/tacotron/issues/194#issuecomment-413747358 It is said there is information loss going from mel-spectrograms to spectrograms. What do you think of it?

I am now considering G&L vocoder for fast synthesis even the quality from WaveNet is the best. I have also improved something on Ito's version that inspired by you, really appreciated for that https://github.com/keithito/tacotron/issues/198

[Rayhane-mamah]

Hey @begeekmyfriend yeah I think what the comment is discussing is that directly inverting mels using G&L is very lossy. That's why I added the mel -> linear network in the first place. As for using CBHG for that, I personally in my experiments found that CBHG takes off fragments better than T2 encoder-like architecture (CBHG uses multitude of differents "grams" so it makes good linear spectrogram reconstruction I think). It would be good to try making a proper comparison when I get the chance. I am also concerned about the preprocessing parameters (fmin and fmax). One could also try figuring out best use case for those.

Glad my work helped achieve something haha :) If speed is a concern @begeekmyfriend, you can try using parallel wavenet which goes faster than G&L (for sure) and has better audio quality ;) That takes quite some time to implement though..

[begeekmyfriend]

Is there any difference on mel outputs between T1 and T2?

[Rayhane-mamah]

The target itself? Except for minor preprocessing differences, there shouldn't be any other inconsistencies.

As for how the model predicts those mels? yeah there's plenty of differences. Loss function is different, as well as the model architecture. overall predictions seem close though.

[begeekmyfriend]

I mean the mel predictions. Even though you added CBHG as post net, there is still difference in effects from G&L vocoder.

[Rayhane-mamah]

Yes there will naturally always be differences when using G&L to approximate the phase information. I mean, you can even try creating linear spectrograms from real audio, then use G&L on top of them, it will have fragments :) The CBHG outputs however do not sound much different from real spectrograms inverted with G&L. Do I make sense?

[begeekmyfriend]

All right, please wait for my T1 evaluation these days, I will show you the difference.

[Rayhane-mamah]

Alright, looking forward to see your awesome results as usual :) depending on your findings I will most probably adapt the post processing network so thanks in advance for that :)

[StevenZYj]

That's not the truth from my experience. I've conducted experiments on the performance of T1 and T2 both under the circumstances of linear prediction, and the quality of predicted audios from T2 easily outperforms those from T1. Also I found CBHG better than T2's encoding part as post processing network based on the loss of my evaluation set.

[begeekmyfriend]

Here are the evaluation results of both T1 and T2. It sounds that the quality of T2 results more steady, but we can still here little noises in them...
T1_VS_T2.zip

[begeekmyfriend]

The above spectrogram is from THCHS-30 female speaker from T1. And the below one is from a single male speaker from T2. It looks that the above one less blurry and the result is with little noise.
By the way both of THCHS-30 and my private male speaker dataset are all right. It seems there should be different hyper parameters applying to the male and female voices.
@Rayhane-mamah I suggest you trying male speakers for some experiments.

[Rayhane-mamah]

Hey thanks for sharing!

Isn't that first plot a melplot?
Blurriness is explained by the MSE really, I think it doesn't have much impact on G&L though.

Yes you are right, different parameters are neede for male speakers ( On Tacotron side) we are discussing on Slack to figure out ways to get best performance with male speakers. Thanks for asserting that :)

[begeekmyfriend]

Both of them are linear plot. In fact I do not find any demo of male evaluation results that produced by G&L, all of the male evaluation demos are from WaveNet. Maybe there should be other optimized hyper parameters for male speakers.

[begeekmyfriend]

@Rayhane-mamah The staff who is the single male speaker providing my private corpus agreed to share his 22 hour long voice recording with you. Can you leave your email that I can upload the dataset to the google cloud driver and you can use this dataset to help me achieve better performances? What I want is the linear spectrograms, (mel one also can be) not WaveNet.

The dataset is in Chinese mandarin and the Chinese adaption has been made in my fork You can modify this line for file location.

Thank you in advance!

[Rayhane-mamah]

Hey again @begeekmyfriend, yes absolutely, here's my address:
rayhane.[email protected]

I believe we can work that out ;)

[begeekmyfriend]

The above one is a female speaker from your version, the below one is from Keith Ito's, we can see your spectrograms are not clear (sorry that it hard for me to express in English) enough compared with Ito's.
female_t1_vs_t2.zip
The source code of T1 is here I have added location sensitive attention model into the original code.

[Rayhane-mamah]

Yeah they're blurry, it has been said in T2 paper, they claim it's due to L2 loss, so one might maybe try replacing it with L1 loss and see what it gives? As for audio quality, there isn't too big a difference really, is that T1 using r=5?

[begeekmyfriend]

Yes T1 is using r=5. But why there is L2 loss and L1 better?

[Rayhane-mamah]

I'm not sure but I guess it's for stability purposes, paper authors are the ones to ask here :) L1 is usually used for its robustness against outliers mainly. Plus, since T2 is supposed to work with Wavenet, mels blurriness isn't a big deal apparently..

[begeekmyfriend]

For what I know L1 results in the sparsity of the weight parameters and L2 can smooth the parameters. I guess square calculation might be lossy for some values?

[Rayhane-mamah]

I think you're making a small confusion here. What you're discussing is the case with L1 regularization and L2 regularization, not loss functions :) The reason, I believe, that makes mels blurry when using L2 loss function, is because L2 gives priority to big errors. if the model is having some values in correct prediction range, and some others which are off, it will focus on correcting those "off" predictions.

By observing the mel plots, you can always see that most of the spectrogram is in range, but details are not entirely there L2 loss doesn't really force the model to make neat predictions. In contrast to L1, when near convergence, the gradients become too small and not much information is learned afterwards. L1 keeps the same slope even when close to convergence..

For regularization cases, L2 helps spreading out the learned information across all weights. L1 on the other hand creates sparsity by setting some parameters to 0. Here's a good blog post I read a while ago, pretty useful.

[begeekmyfriend]

I have tried L1 loss but the result still sounds blurry.
step-38000-eval-waveform-linear.zip

before = tf.reduce_mean(tf.abs(self.mel_targets - self.decoder_output))
after = tf.reduce_mean(tf.abs(self.mel_targets - self.mel_outputs))
linear_loss = tf.reduce_mean(tf.abs(self.linear_targets - self.linear_outputs))

By the way, I have shot a shared link of my mandarin dataset on Google cloud driver with you, did you receive it?

[begeekmyfriend]

Hey, I have found the key point. It does nothing with the NN architecture, the loss function or the regularization, but the audio signal processing. We forgot to add pre-emphasis on linear spectrograms. Let me show you the difference.
preemphasis_or_not.zip

The above one is without pre-emphasis on linear spectrograms and the below one is. We can hear the below one added pre-emphasis clearer. It is only 1K steps so it must be the signal processing determine the quality of audio but not the model. So the pre-emphasis does play an important role on audio signal processing.

Here is the PR that would fix this issue https://github.com/Rayhane-mamah/Tacotron-2/pull/170

[StevenZYj]

That's true. If ignore the network and just do stft and istft and then use G-L to synthesize the audio, you can't get the quality as the original audio. It appears that G-L relies on pre-emphasis to get normal audio, but WaveNet does not.

[begeekmyfriend]

@StevenZYj According to Keith Ito's version, preemphasis should be also added on mel spectrograms.

[tugstugi]

@begeekmyfriend This explains why my Mongolian 6 hours dataset produces poor quality in Tacotron. I have a DC-TTS (arXiv:1710.08969) implementation producing better samples -> it uses preemphasis! I will now retrain Tacotron with your changes. Thanks