Tacotron-2: import the spectrogram into r9y9 wavenet vocoder and some samples

[YanHengxu]

I downloaded the recently updated code, and trained 280,000 steps. The effect is OK, but when I input the generated mel spectrogram into r9y9's training 50k step model, the effect is not good. The following is my generation. Audio, can you tell me why this is? And when is wavenet_vocoder training? Thank you very much.
wavs.zip

[Rayhane-mamah]

Is your wavenet trained on real mel spectrograms?

Does the model give good performance on real mel spectrograms?

On Mon, 23 Apr 2018, 08:03 YanHengxu, [e-mail] wrote:

I downloaded the recently updated code, and trained 280,000 steps. The
effect is OK, but when I input the generated mel spectrogram into r9y9's
training 50k step model, the effect is not good. The following is my
generation. Audio, can you tell me why this is? And when is wavenet_vocoder
training? Thank you very much.
wavs.zip
https://github.com/Rayhane-mamah/Tacotron-2/files/1936982/wavs.zip

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Rayhane-mamah/Tacotron-2/issues/29, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AhFSwMdJGGR8V5CsDLJQquHISHD0I2Yhks5trXzHgaJpZM4TfWLT
.

[YanHengxu]

My wavenet is trained on real mel spectrograms and the wavenet_vocoder model performs very good on real mel spectrograms

[YanHengxu]

This result is trained on real mel spectrograms by r9y9's wavenet_vocoder model
checkpoint_step000500000_ema-3.wav.zip

[Rayhane-mamah]

As I thought.

From Tacotron 2 paper:
"However, when trained on ground truth features and made to synthesize from predicted features, the result is worse than the opposite. This is due to the tendency of the predicted spectrograms to be oversmoothed and less detailed than the ground truth – a consequence of the squared error loss optimized by the feature prediction network. When trained on ground truth spectrograms, the network does not learn to generate high quality speech waveforms from oversmoothed features."

You do not want to train your wavenet on real spectrograms, instead you want to train it on GTA synthesized mels from the Feature Prediction Network.

So you should probably run the synthesize.py on your trained tacotron model to generate GTA mels. Then use those mels to train the wavenet.

By doing the above you should get human like audio quality. My train and synthesize algorithms for wavenet vocoder will be out this friday.

[YanHengxu]

Thank you very much!

[Rayhane-mamah]

Anytime, if you encounter any further problems, please feel free to ask!

[neverjoe]

I think the difference of _normalize(S) lead wavenet vocoder to get bad quality wav, in my experience wavenet vocoder is so sensitive to the melspetrum scale.Now I got reasonable wav with wavenet vocoder and my taco2 model with 500k step, the wavenet vocoder is trained on real mel feature.
ljspeech_model_step_680K_10.zip

[duvtedudug]

Are you referring to normalising to [-4, 4]?

I've found that if you use wavenet_vocoder scaling then taco2 does not align. But taco2 aligns very fast with [-4, 4].
@Rayhane-mamah is this your reason?

Probably best solution is to use an intermediary preprocessing step between networks? Just make a numpy script to rescale e.g. the [-4, 4] to [0.01, 1]?

[Rayhane-mamah]

@duvtedudug I was not aware that model does not learn to align with wavenet_vocoder scaling.. I mainly scale to [-4, 4] to enable for more detail in the output mel spectrogram (in the model's vision). Purely based on experience, outputs scaled to large domains tend to make T2 converge faster and better..

I have no permanent solution for the moment, so @duvtedudug 's solution should do the trick for now. @neverjoe it would be really helpful if you could provide some distribution plot of your potential mels distribution (used to train Wavenet). Thanks :)

[neverjoe]

@Rayhane-mamah https://github.com/Rayhane-mamah My solution is very
easy, when you use Taco2 to generate your mel, you should denormalize it
with Taco2's denormalizing function, and then send to r9y9's wavenet
vocoder with it's own normalizing function. It works for me. Remember it,
wavenet vocoder is very very sensitive to the way of preprocessing.

Rayhane Mama [e-mail]于2018年5月4日周五 上午1:08写道：

@duvtedudug https://github.com/duvtedudug I was not aware that model
does not learn to align with wavenet_vocoder scaling.. I mainly scale to
[-4, 4] to enable for more detail in the output mel spectrogram (in the
model's vision). Purely based on experience, outputs scaled to large
domains tend to make T2 converge faster and better..

I have no permanent solution for the moment, so @duvtedudug
https://github.com/duvtedudug 's solution should do the trick for now.
@neverjoe https://github.com/neverjoe it would be really helpful if you
could provide some distribution plot of your potential mels distribution
(used to train Wavenet). Thanks :)

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/Rayhane-mamah/Tacotron-2/issues/29#issuecomment-386367107,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABsFmOWp37WHHZ9yz90UqYihtYDHH_Toks5tuzmBgaJpZM4TfWLT
.

>

Sent by Inbox

[ABC0408]

hello, @Rayhane-mamah, I run this tacotron model generated GTA mels, when I use those mels to train the wavenet（r9y9）, instead of the real mels, I met this error：
File "E:\TTS\wavenet_vocoder\train.py", line xxx, in collate_fn
assert_ready_for_upsampling(x, c)
File "E:\TTS\wavenet_vocoder\train.py", line xxx, in assert_ready_for_upsampling
assert len(x) % len(c) == 0 and len(x) // len(c) == audio.get_hop_size()
AssertionError
Should I comment this line？like：# assert_ready_for_upsampling(x, c)

[Rayhane-mamah]

@ABC0408 if you want to train with GTA, I recommend you stick with our Wavenet model which handles time inconsistencies.

[sunnnnnnnny]

what is GTA?

[duvtedudug]

what is GTA?

One of the biggest video game franchises of all time ;)

No, it's 'ground-truth-aligned'. Meaning the state of the network follows the ground truth, 'real' spectrogram's timing. Yielding a predicted spectrogram that is the same length of hops as the real spectrogram. This is useful if you want to then train the Wavenet with generated specs rather than 'real' specs.