Tacotron2: Overfitting: validation loss stuck.

[ksaidin]

Hello,
I am experimenting with 3.5 hours of data set. (16kbit, 22khz, 9.5s av.duration, 1300 audio files)
I am using pretrained models which are provided by this repo.

My hparams;
p_attention_dropout=0.5,
p_decoder_dropout=0.5,
learning_rate=4e-5,
batch_size=16,

After 15k iteraton: Train loss 0.25. It is producing almost original like outputs.

The only thing I couldn't improve is a validation loss, it stuck since 4000 iteration.

All suggestions are welcome! Thanks beforehand.

[rafaelvalle]

Very nice attention map!
My guess is that you're model will start overfitting the training data, which implies you either need to augment your data or regularize the model more strongly, e.g. increase weight decay or decrease the prenet dimensions.

[ksaidin]

Thank you for your recommendations @rafaelvalle !

I've been tuning weight decay and prenet dimensions with no success, I will go and try inflating my data.

[ksaidin]

After I artificially boosted my training data, (up to 6,5 hrs) I've managed to decrease my validation loss to ~0.25. Now I have roughly equal training and validation losses.

My attention plot after 18.5K iterations as below. I am not sure if I am reading it correctly. What are those horizontal lines on the map?

[CookiePPP]

@ksaidin
The horizontal lines show periods where the model will be generating off of the same piece of text.
These horizontal lines will typically be around commas or periods where the model should be pausing.
You'll notice these lines are blurred, this would be for 2 reasons.

1. The model cannot tell the difference between spaces (" ") and commas (",") yet, so when two are neighbouring it will just send a mix of both.
2. It is hard to align along 200 timesteps where most of the inputs are either silence or breathing, very little information to help the model figure out where it is.

Once your model stops improving, you can drop your learning rate and see if the model is able to align along those long stretches.
To my memory, the model needs about 1e-4 learning rate to align along pauses (though ideal learning rate is batch size dependent)

Also be aware that small periods of low alignment can make the entire model unstable when you actually go to use it.

During Validation

Same text without any Teacher Forcing (like how you'd use the model for new text)

[ksaidin]

Thank you @CookiePPP for explanation.

I was too worried about inner silence, because there are thousands of audio files and usually the silence lengths inside are not same. I had significant improvements after @Yeongtae 's trimming code. I was inspired by that work and I've worked out my version of removing inside silence. I've just started training my dataset after then, I don't how much it helps.

[CookiePPP]

@ksaidin
The attention plot you showed has 2 seconds of time spent on the same text piece, so I think that you will find some improvement removing some of the inner silence.
Good Luck regardless!

[ksaidin]

Thank you @CookiePPP , Attention plot's aligned faster this time,

Why do you think this anomaly in train loss. It jumped high up for some time despite nothing's changed in parameters. I've just let it run and it normalized later on.

[rafaelvalle]

@ksaidin how did you augment the data?

[ksaidin]

@rafaelvalle , in two ways: I've created new audios by parsing from different starting points than older ones and I've added random noise.

[ksaidin]

Status report, after 27k iteration I have ~0.22 training loss and ~0.21 validation loss. I have annealed my learning rate to 5e-5. Weight decay stays 1e-6 since I have my validation loss is decreasing alongside with training loss equally.

My attention map has disconnections though output is stable, I guess I will keep decreasing my learning rate.

[ksaidin]

Another anomaly, grad norm jumps as high as 1270! somewhere around 33k iteration. This experiment is showing me really weird graphs. Anyone can explain?
Btw, all other graphs are acting normal.

[ksaidin]

@rafaelvalle , is there a minimum limit for learning rate? My learning rate approaches 1e-6 and it seems my grad.norm calculation is broken.

I was planning to run this experiment for 20K to 30K more iterations on 1e-6. Now I've stopped to understand what's going wrong.

[rafaelvalle]

Around 1e-6 you might start getting numerical precision issues that can make your loss explode.

[ksaidin]

Thank you @rafaelvalle !

Conclusions on training small dataset:

I've got better results after data augmentation than penalizing model complexity. Weight regularization definitely helps reduce overfitting but data augmentation evens losses.

I kept annealing my learning rate manually; reduced 10% when I see no regress for ~2000 iterations, It worked for me till I reach 1e-5.

My final graphs; (1e-5 at 30K iteration)

Thank you.

[EuphoriaCelestial]

@rafaelvalle , in two ways: I've created new audios by parsing from different starting points than older ones and I've added random noise.

If I augment the data by adding noise, will it affect the output audio quality? I used to do so with speech-to-text data, but I think its impractical here

[rafaelvalle]

Make sure your noise is gaussian with zero mean and unit variance. In expectation, the added noise will be equal to zero and hence your noise addition will be unbiased.

[shaktikshri]

@ksaidin , I might be late to ask this but what was your execution time for 15k steps? I have a very similar dataset as yours (5.4hrs). My stats are at 14sec/step on V100 GPU, batch size 128. but i've added some latent variables which are part of the model. Trying to figure out if i have an extremely slow implementation or is it just normal?

[EuphoriaCelestial]

@ksaidin , I might be late to ask this but what was your execution time for 15k steps? I have a very similar dataset as yours (5.4hrs). My stats are at 14sec/step on V100 GPU, batch size 128. but i've added some latent variables which are part of the model. Trying to figure out if i have an extremely slow implementation or is it just normal?

that was slow, I trained my model on RTX 2080ti, batch_size=32, each step take around 1,5s

[ksaidin]

@shaktikshri I don't understand what's your worry exactly about? If your training data average length is 2 seconds then you will have a smaller time per step than having training data average length of 18 seconds. Am I missing something?

[EuphoriaCelestial]

@ksaidin can you please share your data augment code?

[pathnirvana]

I am training with a small Sinhala (indic language of Sri Lanka) language dataset of 1.5 hours and observe overfitting as expected. (I am in the process of creating a bigger dataset but it will take few weeks)

However unlike @ksaidin 's results above I do not get good alignment. Also when I synthesis speech with a some text in the training set with 500 epoch checkpoint I hear garbage - my thinking was that I should here something intelegible since I was using a prompt from the training set. Please see the results below and let me know what I might be doing wrong.

important details

1. I use the same audio format as LJ dataset and all the audio hparams are the same
2. however I modify the symbols to symbols = [_pad] + list(_special) + list(_punctuation) + list('abcdefghijklmnopqrstuvwxyzDT') and transliterate sinhala characters based on the pronounciation to match these symbol range. - no cleaners used
3. for synthesis I am using pretrained waveglow v5 model