Tacotron2: Problems in inference mel-spectrogram with tacotron ?

[leminhnguyen]

I have tried to inference mel-spectrogram with my own pre-trained model. But I don't understand why with the same text the mel-spectrogram inferred by tacotron have different size

For example: I have the snippet below

def get_sequence(text):
    sequence = np.array(text_to_sequence(text, ['basic_cleaners']))[None, :]
    sequence = torch.autograd.Variable(torch.from_numpy(sequence)).to(device).long()
    return sequence

text = "some text here"
text2seq = get_sequence(text)
for _ in range(10):
    mel_outputs, mel_outputs_postnet, _, alignments = model.inference(text2seq)
    print(mel_outputs_postnet.shape)

And the result:

torch.Size([1, 80, 176])
torch.Size([1, 80, 177])
torch.Size([1, 80, 179])
torch.Size([1, 80, 174])
torch.Size([1, 80, 177])
torch.Size([1, 80, 174])
torch.Size([1, 80, 180])
torch.Size([1, 80, 180])
torch.Size([1, 80, 179])
torch.Size([1, 80, 181])

As you can see, the third dimension changed randomly after each iteration. And I don't know why this happen
Someone can explain this more details ? Thanks in advance

[CookiePPP]

The third dimension is the time dimension, aka, [1, 80, 181] is the same as a mel-spectrogram with 181 frames, which translates to around 2.3 second of audio.

the model will produce different outputs each time you run it.
(different outputs in all aspects, length/duration, audio quality, accuracy and everything in-between can vary between generations with the same input text)

This is because the dropout in the prenet is always active. Training=True
https://github.com/NVIDIA/tacotron2/blob/master/model.py#L99

The outputs should all be reasonable and match the input text, so just pick whatever and you'll be fine.

[leminhnguyen]

@CookiePPP Thank you for your time, great answer. I have a additional question. If I have the model A trained with sampling_rate = 16k, hop_length=200 and win_length=800, the model B trained with sampling_rate = 22k, hop_length=256 and win_length=1024. So the size of predicted mel-spectrograms from two models are different significantly or not depend on the config ?

[CookiePPP]

hop_length/sampling_rate = hop_time_in_milliseconds
200/16000 = 0.0125s per frame
256/22050 = 0.0116s per frame
The 16Khz model will require less frames per second, which will normally increase stability/naturalness but decrease audio quality. However the difference between 12.5ms and 11.6ms is tiny so you can ignore it.

800/16000 = 0.050s per frame
1024/22050 = 0.046s per frame
Again, very minor difference, you can ignore it.

---

Yeah, those models should produce almost identical spectrograms and be usable on the same vocoder.
You may want to change model A to use 16000, 186, 736 if you plan to use the WaveGlow or HiFi-GAN pretrained models.

[leminhnguyen]

@CookiePPP Oh bro, lucky for me that you did know Hifi-GAN. I'm using mel-spectrogram from tacotron for fine-tuning in Hifi-GAN. And I have the problem in aligning mel-spectrogram from tacotron and original audio. In Hifigan they have the snippet below.

mel = np.load(os.path.join(self.base_mels_path, os.path.splitext(os.path.split(filename)[-1])[0] + '.npy'))
mel = torch.from_numpy(mel)
if len(mel.shape) < 3:
    mel = mel.unsqueeze(0)
if not mel.shape[1] == self.num_mels:
    mel = mel.transpose(1, 2)
if self.split:
    frames_per_seg = math.ceil(self.segment_size / self.hop_size)
    if audio.size(1) >= self.segment_size:
         mel_start = random.randint(0, mel.size(2) - frames_per_seg - 1)
         mel = mel[:, :, mel_start:mel_start + frames_per_seg]
         audio = audio[:, mel_start * self.hop_size:(mel_start + frames_per_seg) * self.hop_size]
     else:
          mel = torch.nn.functional.pad(mel, (0, frames_per_seg - mel.size(2)), 'constant')
          audio = torch.nn.functional.pad(audio, (0, self.segment_size - audio.size(1)), 'constant')

As you can see, first they get the predicted mel-spectrogram from tacotron, then they choose a random number (between 0 -> mel.shape - frames_per_seg) and trying to map the mel-spectrogram and audio by using this number. So if the size_of_mel * hop_size > the_number_of_frames_in_audio, it can be mismatch between the predicted mel and the audio. So do you have any suggestion ?

[CookiePPP]

So if the size_of_mel * hop_size > the_number_of_frames_in_audio, it can be mismatch between the predicted mel and the audio.

You shouldn't have spectrograms with longer duration than the input audio files if you're generating the spectrograms correctly. You're supposed to generate them using teacher forcing (which is just like outputs you'd get in the training/validation stage)

https://github.com/NVIDIA/tacotron2/blob/master/train.py#L134
y_pred[1] will contain a FloatTensor of shape [batch_size, n_mel_channels, mel_timesteps]

---

https://github.com/Yeongtae/tacotron2/blob/master/GTA.py

Have a look at this GTA.py file for an already built solution. It's old but it should have everything you need.

[leminhnguyen]

@CookiePPP Many thanks for your helpful links. I have some questions:

1. The teacher forcing is an already built-in feature in tacotron ?
2. If I have pre-trained model and infer the mel-spectrogram by that model then the generated mels are in teacher-forcing ?
3. If not, how can I config to use teacher-forcing for training and inferring ?
4. The predicted mel-spectrogram which are used as input for wavenet or hifigan training should be generate during training/validation stage or after we trained completely and got the model for inferring ?

[CookiePPP]

(Image taken from paper "A New GAN-based End-to-End TTS Training Algorithm")

---

When you're Training tacotron2, each mel spectrogram is predicted frame-by-frame in order from left to right.
When you predict frame num11, you use the frame num10 as the input. By doing this tacotron2 has a very simple task of predicting a single frame at a time given perfect information about the previous frame.

You can see a visual representation for training above.

• y1 is the 1st mel spectrogram frame taken from the original audio. y1 would be a target/goal of the model.
• ŷ1 is tacotron2 attempt to recreate y1

That is teacher forcing in a nutshell. Predict the next frame using the real current frame as input.

---

Next, Testing (inferring) (prediction).

When you go to produce original audio from your model, you won't have any real frames to use to guide/teacher-force the model.
Instead, the model uses it's own output as it's input. As you can see in the top half of the image.
This mode isn't used for training because it is sometimes very unstable, if the model is even slightly unsure of it's output then it will produce a slightly blurred or incorrect output, and then it will use it's blurry/incorrect output as it's input. Repeat that a few times and the model very quickly starts outputting rubbish.

The code you posted above

...
    mel_outputs, mel_outputs_postnet, _, alignments = model.inference(text2seq)
...

is done without any teacher forcing. Because of this the outputs will be completely different from the target audio's.

You can see what type of mode the model is running in by the function you use to call it.
outputs = model(y) will run in teacher-forcing mode and will require the target spectrogram as one of the elements inside y.
outputs = model.inference(y) will run in 'free-running' mode and only requires the text.

---

1.
Teacher forcing is build into this repo already.
Calling model(y) will run the model with teacher forcing. You need the original audio to use this, I recommend just using GTA.py I linked above to make it easier for you.

2.

If I have pre-trained model and infer the mel-spectrogram by that model then the generated mels are in teacher-forcing ?

If you use model.inference(y) at any point then your outputs will not be aligned to the original audio/spectrogram and the model will not be running in teacher-forcing.

3.

If not, how can I config to use teacher-forcing for training and inferring ?

Look at GTA.py or train.py, they both have examples of the set up required.
You wouldn't infer with teacher-forcing as you would just reproduce the input spectrogram and there's no point.

4.

The predicted mel-spectrogram which are used as input for wavenet or hifigan training should be generate during training/validation stage or after we trained completely and got the model for inferring ?

You should generate predicted mel-spectrogram after the model is trained, but do not use model.inference for it.

You could generate the mel-spectrograms while the model is training/validating, but most people like to do it after training to make sure everything comes from the same model and so they can run the model in eval() mode which turns off dropout and BatchNorm (if you don't know what these are then don't worry about it).

[leminhnguyen]

@CookiePPP I’m eternally grateful for your supporting to the newbie likes me. Today I have read the GTA.py file recommended by you. I don't know whether you have run it or not, but for me I think they have a problem in their code. I'll point that in the section below

This is their code: (I just comment out a few lines and add a print statement print(wav_path, input_lengths[k], output_lengths[k], mel.shape) )

for k in range(batch_size):
    wav_path = org_audiopaths[k]
    mel_path = mel_paths[k]+'.npy'
    # map = "{}|{}\n".format(wav_path,mel_path)
    # f.write(map)
    # To do: size mismatch
    diff = output_lengths[k] - (input_lengths[k] / hparams.hop_length)
    diff = diff.data.data.cpu().numpy()
    mel = mel_outputs_postnet[k,:,:output_lengths[k]-diff]
    print(wav_path, input_lengths[k], output_lengths[k], mel.shape)
    # if diff != 0: print(wav_path, input_lengths[k], output_lengths[k], mel.shape)
    # np.save(mel_path, mel)

The result:

Warm starting model from checkpoint 'outdir/checkpoint_16000'
FULL/wavs/0005941_0.wav tensor(199) tensor(888) (80, 0)
FULL/wavs/0010459_0.wav tensor(192) tensor(904) (80, 0)
FULL/wavs/0013751_0.wav tensor(175) tensor(877) (80, 0)
FULL/wavs/0000785_0.wav tensor(173) tensor(903) (80, 0)
FULL/wavs/0000206_0.wav tensor(172) tensor(887) (80, 0)
FULL/wavs/0012955_0.wav tensor(168) tensor(893) (80, 0)
FULL/wavs/0007443_0.wav tensor(168) tensor(892) (80, 0)
FULL/wavs/0014254_0.wav tensor(167) tensor(912) (80, 0)
FULL/wavs/0000731.wav tensor(167) tensor(854) (80, 0)
FULL/wavs/0000511_0.wav tensor(163) tensor(890) (80, 0)
....

As you can see the third dimension of sliced mel almost zeros.
I ran with: python GTA.py -o FULL/ -c outdir/checkpoint_16000 and the model trained with sampling_rate=16k, hop_size=200 and win_size=800. But I don't think the problem caused by these config. I think the problem at the line

diff = output_lengths[k] - (input_lengths[k] / hparams.hop_length)

In their code, they commented the output_lengths[k] as the original mel-spectrogram. But the input_lengths[k] doesn't have any suggestion, but with the size (eg.: tensor(199)) I believe that is the size of text input. So

1. The expression input_lengths[k] / hparams.hop_length (len(text)/hop_size) doesn't have any sense
2. For example with the first output line:
   Because diff = 888 - (199/888) = 888 so the line mel = mel_outputs_postnet[k,:,:output_lengths[k]-diff]
   will be mel = mel_outputs_postnet[k,:,:888-888] --> mel = mel_outputs_postnet[k,:,:0] that leads to empty predicted mel
3. I think the input_lengths[k] should the number of frames of the audio instead of the text length

Do you have any idea for this problem ??? Again, thank in advance

[CookiePPP]

Yeah, input_length is the text length, that shouldn't be used in any way to determine the output size.

---

I think you can change

            # To do: size mismatch
            diff = output_lengths[k] - (input_lengths[k] / hparams.hop_length)
            diff = diff.data.data.cpu().numpy()
            mel = mel_outputs_postnet[k,:,:output_lengths[k]-diff]
            if diff != 0: print(wav_path, input_lengths[k], output_lengths[k], mel.shape)

to

            mel = mel_outputs_postnet[k,:,:output_lengths[k]]

I don't remember the last time I used this file unmodified, but I think I might've had the same issue.

[leminhnguyen]

@CookiePPP My friend, thank you so much for your support. I'll try that and let you know the fine-tuning result when it's done :smile:

[CookiePPP]

@leminhnguyen If everything works can you upload the modified GTA.py somewhere?
I'll link others to it if this comes up again with someone else. 😄

[leminhnguyen]

@CookiePPP Definitely, bro :smile:

[youssefavx]

Okay guys, I'm a bit of a beginner here so am quite baffled by the complexity. All I want to do is generate .npy mel spectrograms given 1 wave file, just as a start. The GTA.py file is extremely convoluted.

I'm looking for one function like this:

def generate_mel(audio_folder_path, modelpath):
    model = load_model(modelpath)
    for audio_file_path in audio_folder_path:
        mel_spectrogram_path = model(audio_file_path)
    print(f".npy Mel spectrogram generated at {mel_spectrogram_path}")

However, this is how far I've gotten and I'm getting really really confused:

I figured out how to load the model in Colab by doing this (before that I defined all functions from the GTA file in Colab while pointing my directory inside the tacotron2 repo. For fellow beginners, I also had to edit the n_symbols parameter to 148 (it was previously 149) in the hparams.py file after downloading the tacotron2 model because I got an error about the 'shape' of the model when loading.):

import os
import time
import argparse
import math
from numpy import finfo
import numpy as np
from hparams import create_hparams

hparams = create_hparams('')

#if hparams.distributed_run:
init_distributed(hparams, 1, 0, 'group_name')
torch.manual_seed(hparams.seed)
torch.cuda.manual_seed(hparams.seed)
model = load_model(hparams)

Then...

1. When I do this: model('/content/sp1.1.4.lr.wav'), I get this error:

ValueError: too many values to unpack (expected 5)

1. I examine this further, and see that this is how it's used in GTA.py:

x, _ = batch_parser(batch)
_, mel_outputs_postnet, _, _ = model(x)

Hm... what's the x?

I look back further...

What's batch_parser?

It seems if I run this code the batch_parser error goes away, great!

if hparams.distributed_run or torch.cuda.device_count() > 1:
    batch_parser = model.module.parse_batch
else:
    batch_parser = model.parse_batch

Now...

NameError: name 'batch' is not defined

What's batch?

Well it seems:

for i, batch in enumerate(train_loader):

What's train_loader?

It comes from this function:

def prepare_dataloaders(hparams):
    # Get data, data loaders and collate function ready
    trainset = TextMelLoader(hparams.training_files, hparams) # trainset.__getitem__(index) = (text, mel), text in [num_char], mel in [num_mel, ceil((len(audio)+1)/hop_length)]
    valset = TextMelLoader(hparams.validation_files, hparams)
    collate_fn = TextMelCollate(hparams.n_frames_per_step) #
    train_sampler = DistributedSampler(trainset) \
        if hparams.distributed_run else None
    train_loader = DataLoader(trainset, num_workers=1, shuffle=False,
                              sampler=train_sampler,
                              batch_size=hparams.batch_size, pin_memory=False,
                              drop_last=True, collate_fn=collate_fn)
    return train_loader, valset, collate_fn, trainset

When I run:

train_loader, valset, collate_fn, trainset = prepare_dataloaders(hparams)

I get train_loader... it works.

But...still, what's train_loader?

I do train_loader.__dict__ and I get:

{'_DataLoader__initialized': True,
 '_DataLoader__multiprocessing_context': None,
 '_IterableDataset_len_called': None,
 '_dataset_kind': 0,
 '_iterator': None,
 'batch_sampler': <torch.utils.data.sampler.BatchSampler at 0x7f5153151898>,
 'batch_size': 64,
 'collate_fn': <data_utils.TextMelCollate at 0x7f511a0952e8>,
 'dataset': <data_utils.TextMelLoader at 0x7f511a095780>,
 'drop_last': True,
 'generator': None,
 'num_workers': 1,
 'persistent_workers': False,
 'pin_memory': False,
 'prefetch_factor': 2,
 'sampler': <torch.utils.data.sampler.SequentialSampler at 0x7f5153151198>,
 'timeout': 0,
 'worker_init_fn': None}

Hmm... dataset! What's inside?

Let's see...

train_loader.dataset.__dict__

I get:

{'audiopaths_and_text': [['kss/wavs/3/3_1334.wav', '왜 오늘따라 그렇게 시무룩해 보여?'],
  ['kss/wavs/4/4_1017.wav', '숙제를 끝내는 데 두 시간이 들었어요.'],
  ['kss/wavs/4/4_5058.wav', '좋은 부모가 되는 건 절대 쉬운 일이 아니에요.'],
  ['kss/wavs/3/3_2561.wav', '실질적인 경험을 통해 제 지식을 넓혀 나가고 싶어요.'],
  ['kss/wavs/4/4_2383.wav', '저는 현대적인 스타일을 좋아해요.'],
  ['kss/wavs/2/2_0730.wav', '어머니는 심장이 약하세요.'],
  ['kss/wavs/4/4_1469.wav', '간단히 설명해 봐.'],
  ['kss/wavs/4/4_2192.wav', '어젯밤 대형 항공 사고가 있었어요.'],
  ['kss/wavs/3/3_3680.wav', '한 달에 평균적으로 사이트 방문자 수가 몇 명이나 됩니까?'],
  ['kss/wavs/3/3_3944.wav', '나무에 꽃이 활짝 피었다.'],
  ['kss/wavs/2/2_0916.wav', '적합한 예가 잘 생각이 안 나네요.'],
  ['kss/wavs/4/4_2130.wav', '저는 해마다 여름 휴가 때 해외에 나가요.'],
  ['kss/wavs/3/3_3107.wav', '그는 보석으로 어제 풀려났다.'],
  ['kss/wavs/3/3_1162.wav', '그분은 언제나 열정과 의욕이 넘쳐 보여요.'],
  ['kss/wavs/3/3_1124.wav', '그녀는 음악에만 전념하기로 결심했다.'],
  ['kss/wavs/3/3_3665.wav', '엠피쓰리 파일을 홈페이지에서 다운로드 받으시기 바랍니다.'],
  ['kss/wavs/4/4_2585.wav', '그런 사고는 누구한테나 일어날 수 있어요.'],
  ['kss/wavs/4/4_1368.wav', '눈 때문에 어제는 집 안에 하루 종일 갇혀 있었어요.'],
  ['kss/wavs/4/4_4521.wav', '한국 사람과 일본 사람을 구분할 수 있습니까?'],
  ['kss/wavs/3/3_1468.wav', '그녀는 글쓰기에 비범한 재능을 갖고 있다.'],
  ['kss/wavs/4/4_2360.wav', '갑자기 화면에 아무것도 안 나와요.'],
  ['kss/wavs/3/3_0337.wav', '그들은 여성 운전 금지에 대한 항의의 표시로 삭발을 했다.'],
  ['kss/wavs/4/4_3060.wav', '저는 지난날의 제가 아니에요.'],
  ['kss/wavs/4/4_3393.wav', '그는 끝내 나타나지 않았다.'],
  ['kss/wavs/4/4_1562.wav', '그 그림 벽에 거는 건 어떨까?'],
  ['kss/wavs/1/1_0952.wav', '고기가 다 타서 먹을 수가 없었죠.'],
....

Huh? I thought I was going to feed in one wave file and get a mel spectrogram... I didn't know text was needed... besides that where is this dataset coming from?

Most importantly, how do I feed it my audio file?

Okay, let's look inside:

train_loader = DataLoader(trainset, num_workers=1, shuffle=False,
                              sampler=train_sampler,
                              batch_size=hparams.batch_size, pin_memory=False,
                              drop_last=True, collate_fn=collate_fn)

What's trainset?

trainset = TextMelLoader(hparams.training_files, hparams) # trainset.__getitem__(index) = (text, mel), text in [num_char], mel in [num_mel, ceil((len(audio)+1)/hop_length)]

Very confusing, but okay, what's hparams.training_files?

Well hparams comes from:

hparams = create_hparams(args.hparams)

And there doesn't seem to be a default argument:

 parser.add_argument('--hparams', type=str, required=False, help='comma separated name=value pairs')

:'(... I just want 1 function to give audio and get a mel spectrogram...

[youssefavx]

Now I try and do this:

train_loader.dataset.__dict__.update({'audiopaths_and_text':[['/content/sp1.1.4.lr.wav']]})

And when I do train_loader.dataset.__dict__ it now looks like this:

{'audiopaths_and_text': [['/content/sp1.1.4.lr.wav']],
 'load_mel_from_disk': False,
 'max_wav_value': 32768.0,
 'sampling_rate': 22050,
 'stft': TacotronSTFT(
   (stft_fn): STFT()
 ),
 'text_cleaners': ['english_cleaners']}

But when I try to iterate over it:

 for batch in train_loader:
    x, _ = batch_parser(batch)
    _, mel_outputs_postnet, _, _ = model(x)
    mel_outputs_postnet = mel_outputs_postnet.data.cpu().numpy()

Nothing happens... it's as though it's now empty.

[CookiePPP]

GTA.py is designed to be dropped into the tacotron2 folder. If you've trained a tacotron2 model then you should just be able to run GTA.py and it will dump a txt file and .npy files into the output folder. It will use the training audio files + text that was used to train the model, no extra code should be required.

This is too much text to read through and respond to properly right now. I will write a more detailed response in a bit if it seems useful.

[youssefavx]

I haven't trained a tacotron2 model, am I supposed to train one for HiFi GAN if I want to fine tune HiFi GAN on a speaker's clean audio?

I downloaded the pre-trained model they have listed on the readme.

I tried to run GTA like so:
!python GTA.py -o testing2/ -c tacotron2_statedict.pt

I have no idea where in the code that person is feeding in the directory filelists which has a list of text files which seem to contain paths to audio files and their corresponding text.

Instead of finding the parameter where those files are being fed in, I modified the file kss_train_filelist.txt from a list like so:

kss/wavs/4/4_3999.wav|늦어서 죄송합니다.
kss/wavs/4/4_3580.wav|저울을 영에 맞춰 놓아라.
kss/wavs/4/4_0431.wav|물건을 너무 많이 담아서 봉지가 터질 것 같아.
kss/wavs/3/3_0252.wav|이 꽃은 향기가 좋아요.
...

To this:

testing/sp1.1.4.lr.wav|Potatoes

When I ran it with the above command, I found that a folder was generated called mels and a text file called map.txt

However, when I check both the folder and map.txt they are both empty, unfortunately. I have no idea what the problem is.

Sorry about all that text, no need to read it.

[evelynyhc]

hop_length/sampling_rate = hop_time_in_milliseconds
200/16000 = 0.0125s per frame
256/22050 = 0.0116s per frame
The 16Khz model will require less frames per second, which will normally increase stability/naturalness but decrease audio quality. However the difference between 12.5ms and 11.6ms is tiny so you can ignore it.

800/16000 = 0.050s per frame
1024/22050 = 0.046s per frame
Again, very minor difference, you can ignore it.

Yeah, those models should produce almost identical spectrograms and be usable on the same vocoder.
You may want to change model A to use 16000, 186, 736 if you plan to use the WaveGlow or HiFi-GAN pretrained models.

@CookiePPP Thank you for your time, great answer. I have a additional question. If I have the model A trained with sampling_rate = 16k, hop_length=200 and win_length=800, the model B trained with sampling_rate = 22k, hop_length=256 and win_length=1024. So the size of predicted mel-spectrograms from two models are different significantly or not depend on the config ?

hello, i have the same question, if i want to train the model with 16k, any other parameters should i change? thank you

[evelynyhc]

GTA.py is designed to be dropped into the tacotron2 folder. If you've trained a tacotron2 model then you should just be able to run GTA.py and it will dump a txt file and .npy files into the output folder. It will use the training audio files + text that was used to train the model, no extra code should be required.

This is too much text to read through and respond to properly right now. I will write a more detailed response in a bit if it seems useful.

GTA.py is designed to be dropped into the tacotron2 folder. If you've trained a tacotron2 model then you should just be able to run GTA.py and it will dump a txt file and .npy files into the output folder. It will use the training audio files + text that was used to train the model, no extra code should be required.

This is too much text to read through and respond to properly right now. I will write a more detailed response in a bit if it seems useful.

[ErfolgreichCharismatisch]

Tutorial: Training on GPU with Colab, Inference with CPU on Server here.