Tacotron2: Little gap between words on the alignment plot

[dnnnew]

Hi,
My alignment plot has a little gap between words.

batch size - 64
filter_length=2048,
hop_length=275,
win_length=1100,

And it sounds bit like reading separated words.
Is there anyone who had same issue before.

Thanks.

[rafaelvalle]

The alignment looks promising. What dataset is this? Are there no gaps in the dataset itself?

[dnnnew]

Dataset is Mongolian, female, 15.3k samples and 35 hours.
Alignment after 200k iters:

I think there are no gaps in the dataset itself.

Here is Alignment of 250k iters https://github.com/Rayhane-mamah/Tacotron-2 same dataset (different sentence).

Sample wav:
12703_020.wav.zip

[dnnnew]

It isn't solved yet. I am testing another hparams, changing dataset's parts.

[rafaelvalle]

Let us know. I'll keep it open until we fix it.

[Engineering-Course]

Hi @dnnnew,
Do you find out what causes the gaps?
I also meet the same problem on my dataset.

[dnnnew]

@Hi, Engineering-Course.

Not yet.
I guess it might be depend on my dataset.

My dataset's wav file has multiple sentence or incomplete sentence. So I am working on my dataset (realignment wav and text) now then will train again. I'll report result.

[dnnnew]

@Engineering-Course
please let me know If you find out what causes the gaps.

[Engineering-Course]

However, the wav files on my dataset have the complete sentence and no silence.
Does sampling_rate matter? @rafaelvalle
The sampling rate of my wav is 48000. Then I change the sampling rate in hparams.py from 22050 to 48000.

[Engineering-Course]

BTW, What's your tool to visualize the wave like this? @dnnnew

[dnnnew]

Adobe Audition @Engineering-Course.

[Engineering-Course]

Adobe Audition @Engineering-Course.

Thanks~

[dnnnew]

However, the wav files on my dataset have the complete sentence and no silence.
Does sampling_rate matter? @rafaelvalle
The sampling rate of my wav is 48000. Then I change the sampling rate in hparams.py from 22050 to 48000.

I trained LJ-Speech dataset (same hparams which I trained my dataset) and it looked no gaps.

Since the wav files on your dataset have the complete sentence and no silence, maybe should we change attention network size?

[Engineering-Course]

I also have trained LJ-Speech dataset and it sounds great. And I only change the sampling rate in hparams to train my dataset.
What's your suggestion for attention network size?

[dnnnew]

I also have trained LJ-Speech dataset and it sounds great. And I only change the sampling rate in hparams to train my dataset.

I changed these
filter_length=2048,
hop_length=275,
win_length=1100 then my dataset learn attention (get aligned but with gaps).

What's your suggestion for attention network size?

Before You told me about that the wav files on your dataset have the complete sentence and no silence, I never thought about it.

I'll try it later. (maybe increase)

[Engineering-Course]

Thanks. I'll try it.

[tsungruihon]

may i ask how to generate alignment image? i use tensorborad --logdir=output/log_dir but it just not show me the alignment image

[delgerdalai]

@tsungruihon

https://github.com/NVIDIA/tacotron2/blob/f2c94d94fd64041bd8fdbd908beceea35e82ace0/hparams.py#L13

Alignment image is generated every 1000th iteration in the TensorBoard -> Images.

[rafaelvalle]

@dnnnew did you try training with the default params in the repo?

[dnnnew]

@rafaelvalle, I did. But after 200k iters (415 epochs) with default params, it didn't learn attention.
I changed params like above (from https://github.com/Rayhane-mamah/Tacotron-2), then it looks like this image just in 16k iters.

[rafaelvalle]

Please share with us the exact hparams you used to reach alignment in 16k iters.

[dnnnew]

I changed

_hparams.py_

batch_size=36

_symbols.py_
_letters = 'АБВГДЕЁЖЗИЙКЛМНОӨПРСТУҮФХЦЧШЪЫЬЭЮЯабвгдеёжзийклмноөпрстуүфхцчшъыьэюя'

I trained it on 2 GPU.

[dnnnew]

Also I changed only

_filter_length=2048_

But it didn't learn attention.

[tsungruihon]

@dnnnew may i ask if the train.csv has numbers, like 123456, does the _characters in symbols.py should contain these six numbers?

[delgerdalai]

@tsungruihon

You have to convert number to words.
1 => one
2 => two
etc ....
using characters in symbols.py

or

you can write custom cleaner function like that
https://github.com/NVIDIA/tacotron2/blob/master/text/cleaners.py
https://github.com/Yeongtae/tacotron2/blob/master/text/korean.py

[dnnnew]

@tsungruihon

Also, You have to expand abbreviations to words.

[rafaelvalle]

@dnnnew for sanity checking: when you ran for 200k and it did not learn attention, did you use the cyrillic symbols?
also, i notice you're loading mels from disk. did you use the same mels to train the model from both repos?

[dnnnew]

@rafaelvalle

when you ran for 200k and it did not learn attention, did you use the cyrillic symbols?

Yes

i notice you're loading mels from disk. did you use the same mels to train the model from both repos?

No,
I just used the loading mels from disk to reduce CPU load.
I got mels generation codes from here https://github.com/NVIDIA/tacotron2/blob/f2c94d94fd64041bd8fdbd908beceea35e82ace0/data_utils.py#L37 and I generated preprocess.py.
_preprocess.py generates mels with current hparams, generated mels was used only for this repo._

[tsungruihon]

@dnnnew May i ask you for some idea about my alignment image ? Or how could i judge the effectiveness by the alignment image that i have ?

[rafaelvalle]

@dnnnew did you try training taco2 from this repo with mels produced from Rayhane-mamah repo?

[tsungruihon]

@rafaelvalle @dnnnew Cause i use chinese corpus, i changed the hparams.py into below:

 hparams = tf.contrib.training.HParams(
        ################################
        # Experiment Parameters        #
        ################################
        epochs=500,
        iters_per_checkpoint=1000,
        seed=1234,
        dynamic_loss_scaling=True,
        fp16_run=False,
        distributed_run=False,
        dist_backend="nccl",
        dist_url="tcp://localhost:54321",
        cudnn_enabled=True,
        cudnn_benchmark=False,

        ################################
        # Data Parameters             #
        ################################
        load_mel_from_disk=False,
        training_files='filelists/amber/amber_audio_text_train_filelist.txt',
        validation_files='filelists/amber/amber_audio_text_val_filelist.txt',
#        text_cleaners=['english_cleaners'],
        text_cleaners = ['basic_cleaners'],
        ################################
        # Audio Parameters             #
        ################################
        max_wav_value=32768.0,
        sampling_rate=48000,
        filter_length=4096,
        hop_length=600,
        win_length=2400,
        n_mel_channels=160,
        mel_fmin=125,
        mel_fmax=7600 

And in the 7000 steps, the alignment image like this below:

Does it make sense? Or i should need to modify hparams.py again ?

[dnnnew]

@dnnnew did you try training taco2 from this repo with mels produced from Rayhane-mamah repo?

@rafaelvalle No. I didn't.

[rafaelvalle]

@dnnnew give it a try.

[dnnnew]

@rafaelvalle Ok, I'll try it. Thanks.

[tugstugi]

@dnnnew @delgerdalai
You can still keep the default parameter of this repo and train a Mongolian dataset. You have to cheat a little bit: load the pretrained LJSpeech model but without the embedding weight (because the Mongolian and English symbols differ). With a 5h Mongolian Bible dataset, I got already alignment starting from the step 2500 with the Noam LR scheduling.

[dnnnew]

@tugstugi

Super cool :). I'm really glad to hear that.
Here is a alignment plot of MBS dataset with default params (26k iters).
It seems becoming with the gaps, so I stop training.

Does your alignment plot has a little gaps or not?

[tugstugi]

@dnnnew this repo uses scipy.io.wavfile.read and assumes that WAV files are int16 which is not good :) The WAV files of the Mongolian Bible should be float32. Have you fixed that before training the Mongolian Bible dataset?

So far I remember, the alignments looked OK. I can check tomorrow my dev machine whether the plots looked OK or not.

[dnnnew]

@tugstugi

First, I disabled this line.
https://github.com/NVIDIA/tacotron2/blob/f2c94d94fd64041bd8fdbd908beceea35e82ace0/data_utils.py#L43
But I got error on some files of MBS.

Then, I saved as MBS dataset to int16 😥, and didn't change code.

Thank you

[tugstugi]

@dnnnew somehow I removed the logs. Found only an alignment image of the 3750th step:

[dnnnew]

Nice, it looks no gaps.

I've just started training with pertained LJ weights but without Noam LR scheduler.
How important Noam LR scheduler?

Thanks again.

[rafaelvalle]

@tugstugi thanks for sharing these. With the current setup on LJS there are no gaps in the alignments. For this reason, I suspect that the gaps you are seeing are related to your text and data pre-processing.

It is important to anneal the learning rate. Noam LR scheduler is one of many options.

[tugstugi]

@dnnnew I prefer the Noam LR scheduling specially if I am using a pretrained model. It converges much faster than with a fixed big LR. From the 5000th step, the generated Mongolian audio using Griffin-Lim should be already intelligable.

PS: I forgot to mention that the generated Mongolian audio files didn't have any wrong pauses and sounded naturally.

[dnnnew]

@tugstugi , @rafaelvalle
LJS pretrained weights and after 8k iters. (default params)

LJS-pretrained_8k_step.wav.zip

Without pertained weights and after 202k iters. (audio params was changed)

202k_step_without_pretrain.wav.zip

Sound files are generated by Waveglow(500k , 130 epochs, "filter_length": 2048,"hop_length": 275,"win_length": 1100, n_channels: 256).

Thank you all.
Баярлалаа танд 👍

[tugstugi]

@dnnnew I have checked your attached training sample. Actually your speaker makes many unnatural pauses after every few words like "гэх мэдээлэл", "хэвлэлээр цацагдсан нь" and "ор үндэсгүй гэж".

Who is the speaker? The voice sounds familiar.

[dnnnew]

Yep :(.
Also, we made a mistake when we prepare dataset. Our training sample has multiple sentence, or incomplete sentences. We are working on realignment sound and text.

Нарамчимэг гэсэн байхаа, үндэсний радиод ажиллаж байсан нэг эгчээр уншуулсан. Бид нар мэдээ уншдаг модел-тэй болох гээд нээлттэй кодуудыг туршиж үзээд.

[tugstugi]

@dnnnew Is your waveglow trained on your dataset? Could you try something for me? Feed your Mongolian Tacotron output to the LJSpeech trained waveglow and post the result here :)

PS: send me message over FB. I have some private questions.

[dnnnew]

Yes, waveglow is trained on my dataset.

Tacotron model (LJS pertained weights + 23k iters /training is ongoing/) + LJS pretrained waveglow / my waveglow(500k iters).

Here is result. 😮 super
result.zip

_sentences_
_text = "Монгол Улсын Боловсролын Их сургуулийн хичээлийн тавдугаар байр ашиглалтад орж, өнөөдөр нээлтээ хийлээ."
text = "Дунд тусгалын Цөмийн Зэвсгийг Хязгаарлах гэрээнээс Амеркийн нэгдсэн улс гарах болсон нь Вашингтон-Москвагийн харилцааг муутгаад байгаа."_

PS: Sure

[tugstugi]

so LJSpeech waveglow works better than Mongolian trained waveglow even for Mongolian texts? :D I haven't trained waveglow for the Mongolian Bible dataset, so I am also using the LJSpeech trained waveglow. Works better than expected for the Mongolian male voice.

[dnnnew]

I read that LJS waveglow is trained 870epochs, my version is trained 130 epochs.
Is it possible fine tuning LJS waveglow with my dataset?

[tugstugi]

@dnnnew sure, you can finetune but the problem is waveglow's train code and their provided pretrained model don't match. So you have to fix it first.

[yondu22]

what does this plot mean, can someone explain it?
At each step of the decoder, we are trying to decode one frame of the spectrogram. However, it is not entirely clear what encoder information should be used at each step of the decoder. It can be assumed that this correspondence will be direct. For example, if we have an input text sequence of 200 characters and the corresponding spectrogram of 800 frames, then there will be 4 frames per character. However, you must agree that speech generated on the basis of such a spectrogram would be completely devoid of naturalness. We pronounce some words more quickly, some more slowly, somewhere we pause, and somewhere we don’t. And to consider all possible contexts is not possible. That is why attention is a key element of the entire system: it sets the correspondence between the decoder step and the information from the encoder in order to obtain the information necessary to generate a specific frame. And the larger the value of attention scales, the more “attention should be paid” to the corresponding part of the encoder data when generating a spectrogram frame.