Tensorflowtts: Loss explodes at 16K when training MB-MelGAN

[crux153]

Training MB-MelGAN with Korean dataset causes all loss to explode at around 16k. I'm training on V100 with mixed precision turned OFF, and training with --generator_mixed_precision 1 even yields NaN loss at the same point (I forgot to save that, might reproduce if needed)

Prediction at 15K

Prediction at 20K (which is pure noise)

I'm using the same configuration given (the dataset itself is 48khz, but I downsampled them to 22.05khz). Here's Kaggle link for the dataset I'm using. https://www.kaggle.com/bryanpark/korean-single-speaker-speech-dataset

I think this is similar with #33. Tacotron2 training seems to work just fine with same dataset, I'll try to train MelGAN-STFT with same dataset.

[ZDisket]

Are you finetuning or training from scratch? When does the discriminator start in your config?

[crux153]

@ZDisket I'm training from scratch. As I'm using the same config in this repo (https://github.com/TensorSpeech/TensorflowTTS/blob/master/examples/multiband_melgan/conf/multiband_melgan.v1.yaml), discriminator training starts from 200k, which means I'm seeing the loss explosion before discriminator training begins.

[ishine]

reduce the learning rate

[dathudeptrai]

@crux153 well, i also got this behavior on private dataset but rarely (ljspeech fine). I think there is some samples on dataset make stft loss expode somehow. You can solve this by clip, let replace 2 lines https://github.com/TensorSpeech/TensorflowTTS/blob/master/tensorflow_tts/losses/stft.py#L98-L99 by:

x_mag = tf.clip_by_value(tf.math.sqrt(x_mag ** 2 + 1e-7), 1e-7, 1e3) 
y_mag = tf.clip_by_value(tf.math.sqrt(y_mag ** 2 + 1e-7), 1e-7, 1e3)

[crux153]

@ishine @dathudeptrai I'll try them out. Thanks!

[dathudeptrai]

@crux153 you may also try change https://github.com/TensorSpeech/TensorflowTTS/blob/master/examples/multiband_melgan/conf/multiband_melgan.v1.yaml#L69 to 16384. BTW, how about the performance of tacotron2 for korea ?

[crux153]

@dathudeptrai It generates audible speeches even though I only trained it until 40k. Currently I'm using GL for vocoder, and MelGAN LJSpeech model suffers highly metallic noise if not fine-tuned (as expected). I'm training MB-MelGAN to use with Tacotron, and will continue to train Tacotron2 (and FastSpeech2) after that. I'll open a PR for Korean (and models) if I can get good results :)

[dathudeptrai]

@crux153 thanks :)) i will help you because i am plan to support more and more languages :)).

[crux153]

@crux153 well, i also got this behavior on private dataset but rarely (ljspeech fine). I think there is some samples on dataset make stft loss expode somehow. You can solve this by clip, let replace 2 lines https://github.com/TensorSpeech/TensorflowTTS/blob/master/tensorflow_tts/losses/stft.py#L98-L99 by:

x_mag = tf.clip_by_value(tf.math.sqrt(x_mag ** 2 + 1e-7), 1e-7, 1e3) 
y_mag = tf.clip_by_value(tf.math.sqrt(y_mag ** 2 + 1e-7), 1e-7, 1e3)

Successfully trained MB-MelGAN to 200k after applying @dathudeptrai's fix!

I'm training discriminator now. I think this issue can be closed :) Maybe it'll help other people if we add a note to this fix to /examples/multiband_melgan/README.md?

[dathudeptrai]

@crux153 i will commit a fix directly to stft.py :)). BTW, ur loss too small compared with ljspeech :)), seem this dataset very clear or very small ^^. Looking forward ur samples and PR :)).

[crux153]

@dathudeptrai The dataset I used is 12+ hr dataset with 12,853 utterances, which is quite big. I think the loss seems quite similar to your ljspeech (my tensorboard only shows til 200k). We'll see when it reaches 1000k :)