Tacotron-2: What is the reason that less dataset is needed to train on your implementation?

[begeekmyfriend]

I have tried both your and Keith Ito's implementation but only 10h+ dataset is needed to train into convergence on your version while there should be 20h+ dataset on Ito's one. I know that your version is deprived from Ito's one. So I wonder which module you changed that results in the effect?

[Rayhane-mamah]

Hello @begeekmyfriend, that's a nice question actually! Must be one of life's mysteries :)

Actually, despite using the same cleaners and project structure (I love how he arranges his code), there are many differences in our models and in our preprocessing.

As a consequence I am not really sure about the reason, but it is important to note that we use much more regularization than in keith ito's work (L2, dropout, zoneout, etc.) Which would be a hint if keith ito's work diverges due to overfitting on small datasets.

Please also note that we do not train using the same loss functions and that our target distributions are not the same, which also creates a whole lot of differences between our works.

So, the short answer is I have no clear idea about the reason, but I would love to know if someone could explain it to us!

[begeekmyfriend]

I think I have found one of the factors that differs the time to achieve convergence. I modified the _process_utterence function in datasets/preprocessor.py like that:

def _process_utterance(mel_dir, linear_dir, wav_dir, index, wav_path, text):
        ...
        #M-AILABS extra silence specific
        # if hparams.trim_silence:
        #       wav = audio.trim_silence(wav)

        #Mu-law quantize
        if is_mulaw_quantize(hparams.input_type):
                #[0, quantize_channels]
                ...
        elif is_mulaw(hparams.input_type):
                #[-1, 1]
                ...
        else:
                #[-1, 1]
                ...

        # Compute the mel scale spectrogram from the wav
        mel_spectrogram = audio.melspectrogram(wav).astype(np.float32)
        mel_frames = mel_spectrogram.shape[1]

        #Compute the linear scale spectrogram from the wav
        linear_spectrogram = audio.linearspectrogram(wav).astype(np.float32)
        linear_frames = linear_spectrogram.shape[1]

        #sanity check
        assert linear_frames == mel_frames

        time_steps = mel_frames * audio.get_hop_size()
        assert time_steps >= mel_frames * audio.get_hop_size()
        assert time_steps % audio.get_hop_size() == 0

        # Write the spectrogram and audio to disk
        audio_filename = 'speech-audio-{:05d}.npy'.format(index)
        mel_filename = 'speech-mel-{:05d}.npy'.format(index)
        linear_filename = 'speech-linear-{:05d}.npy'.format(index)
        ...

As you can see I have commented out pad_lr method for the training audio and I will explain why I did it. And then I conducted three experiments by contrast.

• Set hp.input_type as raw and kept pad_lr working;
• Set hp.input_type as raw and commented pad_lr out;
• Set hp.input_type as mulaw-quantize and commented pad_lr out;

On the first try it failed to convergence. And on the second it began to convergence at 4K steps. On the third time it achieved convergence at only 2K steps.

In my opinion the trim of silence really helps alignments. As we can see in Mu-law quantized input mode, we quantized the audio into 256 channels and then called audio.start_and_end_indices method to trim the silence in the header and the tailer accurately to unit of sample point. So it reached convergence fastest. In raw input mode we do not trim silence but the original audio does not contain much silence at both ends so it still can get to convergence. But when we add pad_lr method and switch to raw input mode, it actually impair the purity of the training audio. So it can never get to convergence due to this reason.
In fact, pad_lr does not help training. We can just set timestep as mel_frames * hop_size at will here since timestep is useless in the subsequent training. I think we should comment out those useless code.
By the way, I tried to transplant audio.start_and_end_indices into Keith Ito's version but up to now it seems no good for less training dataset to get convergence. I will keep track on the porting.

[PetrochukM]

@begeekmyfriend The pad_lr method is run on top of the wav not the mel-spectrogram; therefore, it would in no way affect the Tacotron-2 spectrogram convergence.

[begeekmyfriend]

@PetrochukM trim_silence does the same thing with audio.start_and_end_indices but the latter trim accurately to sample points. As you can see I have commented out trim_silence in these experiments. Moreover I have tried on THCHS30 and it got convergence (at about 4K steps with r=3 batch_size=32) but not under these modification. By the way https://github.com/Rayhane-mamah/Tacotron-2/issues/39 shows that reduction of batch size would result in failure to convergence.

@PetrochukM I do not think so. The pad_lr does affect the input of melspectrogram since the input is the wav. I have conducted these experiments for twice to confirm that with my 10h+ dataset. So can you.

[PetrochukM]

@begeekmyfriend I believe that trim_silence affects the convergence. Earlier you mentioned you commented out pad_lr not trim_silence.

Show me in the code where pad_lr affects the mel_spectrogram please because it looks like it does not after reading the code.

[begeekmyfriend]

@PetrochukM you can see the input of audio.melspectrogram is wav[start : end]. And you can print the start and the end to see the changes.
By the way I have showed that I commented out trim_silence on the third floor.

[PetrochukM]

@begeekmyfriend Literally, l, r = audio.pad_lr(wav, hparams.fft_size, audio.get_hop_size()) pad_lr is run after that line.

Are you focused on pad_lr or trim_silence?

Previously, you said:

As you can see I have commented out pad_lr method for the training audio and I will explain why I did it. And then I conducted three experiments by contrast.

I assume you meant you commented out trim_silence.

[begeekmyfriend]

@PetrochukM Yes, the pad_lr cancels the good effect of audio.start_and_end_indices.

[PetrochukM]

pad_lr is on line 115 while mel_spectrogram is on line 104, they do not affect each other.

I give up.

[begeekmyfriend]

Well maybe there is something wrong... I am nearly insane this morning. Let me cool down and keep observation...

[begeekmyfriend]

Emm... The pad_lr is used to calculate timestep for training metadata which is useless in the data feeder. So commenting them out should not affect the result of training. But I really got convergence at 2K steps twice in my latest experiments which never appeared before. One is with predict_linear as False and another with predict_linear as True.

[begeekmyfriend]

Hey I find a bug in audio.trim_silence where there is librosa.effects.trim with default arguments of frame_length=2048 and hop_length=512 while our hyper parameters with fft_size=1024 and hop_size=256. I would fix into the right arguments and see whether it would get convergence within 2K steps under raw input mode.

[begeekmyfriend]

Succeeded to get convergence at 2K steps in raw input mode with librosa.effects.trim fixed. @Rayhane-mamah I will shoot a PR for you.

[a3626a]

If you think pad_lr cancels out good effects of trim, I suggest you to pass center=false to librosa.stft. With this parameter, pad_lr is not required.

https://librosa.github.io/librosa/generated/librosa.core.stft.html

[begeekmyfriend]

@a3626a I was wrong. Actually pad_lr does nothing for audio pre-processing and training since the time step in metadata would not be taken by the feeder, It would still get to convergence if you set the calculated time step as zero.

[a3626a]

Right. pad_lr does not affect spectrograms.

[begeekmyfriend]

@Rayhane-mamah I have conducted series of ablation studies to find the reasons and here is the modification based on Keith Ito's version https://github.com/keithito/tacotron/issues/170. It confirms me that the architecture of decoder and the normalization of mel-spectrograms are the key components which may determinate whether we can use less dataset to get to convergence or not.

By the way I am still use L1 loss and LSTMBLockCell for minimal modification in this study. In your guess, L2, dropout, zoneout and cumulative attention states are not included but it may improve or speed up convergence. For your information.

[Rayhane-mamah]

@begeekmyfriend, Great observations!! I actually confirm your results.

I am not really sure about L2 regularization, I know it reduces overfit for sure, but can't confirm if it helps convergence. On the other hand, cumulative attention is essential for convergence, without it the model gets stuck from time to time. It's predictable considering that weights cumulation is typically adding more information to the network that without it, which makes predictions easier.

[begeekmyfriend]

@Rayhane-mamah Yes, you are right. The cumulative attention states is essential and I have verified it under r==2 https://github.com/keithito/tacotron/issues/170. And I have also noticed that Keith used less filters and kernel size in the attention convolution layer than you did. What is more, the bias in _location_sensitive_score is also dispensable for convergence and good alignment.
You seems did everything right in the code. Congratulations!!

[drasted]

hi @begeekmyfriend ，小白一枚，最近刚好在找tacotron的源代码资料，看到了你的帖子 ，厉害啊，用THCHS30的数据集训练出不错的效果；想问下，对于初学者，想先出个demo看看效果，是直接用您https://github.com/begeekmyfriend/tacotron/commits/tacotron2 这个链接下的代码 配合数据集 就可以大致有一个雏形了吗？ 差不多训练多长时间能等到有输出的啊（用的云端GPU😂）

[begeekmyfriend]

@drasted My fork is only for ablation study and the results of synthesis has not been tested yet. You had better not use it.

[v-yunbin]

@begeekmyfriend why mine sill not converge on 6000 steps with 9.6hr thchs30 data? but your 10hr data has converged on 2000 steps?

[Rayhane-mamah]

@begeekmyfriend thanks for all the contributions, your help is very valuable!

@v-yunbin it's okey if you don't get alignments right away, I usually get them around 16k :)