Tacotron2: How to train a new model with dataset of diffirent language?

Created on 25 Mar 2020 · 33Comments · Source: NVIDIA/tacotron2

I would like to know if it possible to train a Tacotron 2 model for another language, using another dataset which have the same structure as LJ Speech dataset?
And if it is possible, is there any tutorial to do so?

Source

EuphoriaCelestial

Most helpful comment

1.
Ensure your Audio files are all WAV PCM 16 Bit files. You may have to convert them.

2.
Get your data into a filelist.
https://github.com/NVIDIA/tacotron2/blob/master/filelists/ljs_audio_text_val_filelist.txt
For every file in your dataset you'll have a
FILE PATH and a Quote separated by a |

3.
Split that filelist into 2/3 files, a train list, validation list and test list.
I normally do 95% files into the train, 5% into validation and ignore the test list.

4.
Once you have the filelists ready, update
https://github.com/NVIDIA/tacotron2/blob/master/hparams.py#L28#L29
To point to your train and validation filelists.

5.
Since your using another language, you will need to update the symbols list.
Change,
_punctuation and _letters
to include all the symbols you will use from your language.
6.
update the text_cleaners hparam for your language.

There are 3 cleaners you can try
Located Here

def basic_cleaners(text):

def transliteration_cleaners(text):

def english_cleaners(text):

You should probably update the hparam

        text_cleaners=['english_cleaners'],

        text_cleaners=['basic_cleaners'],

to start with.

7.
Train from Scratch.
https://github.com/NVIDIA/tacotron2#training
or
Training using a pre-trained model
https://github.com/NVIDIA/tacotron2#training-using-a-pre-trained-model

CookiePPP on 25 Mar 2020

👍22

All 33 comments

1.
Ensure your Audio files are all WAV PCM 16 Bit files. You may have to convert them.

3.
Split that filelist into 2/3 files, a train list, validation list and test list.
I normally do 95% files into the train, 5% into validation and ignore the test list.

4.
Once you have the filelists ready, update
https://github.com/NVIDIA/tacotron2/blob/master/hparams.py#L28#L29
To point to your train and validation filelists.

There are 3 cleaners you can try
Located Here

def basic_cleaners(text):

def transliteration_cleaners(text):

def english_cleaners(text):

You should probably update the hparam

        text_cleaners=['english_cleaners'],

        text_cleaners=['basic_cleaners'],

to start with.

7.
Train from Scratch.
https://github.com/NVIDIA/tacotron2#training
or
Training using a pre-trained model
https://github.com/NVIDIA/tacotron2#training-using-a-pre-trained-model

CookiePPP on 25 Mar 2020

👍22

@CookiePPP this seem to be quite detailed, thank you! And I have another question, I tried training with LJ Speech dataset and having 2 problems:

I changed the epochs value in hparams.py file to 50 for a quick run, but it run more than 50 epochs. Do I need to change something else?
In the outdir folder, I only see checkpoint file, I cant find model file anywhere (I guess the model file name ends with .pt right?)

EuphoriaCelestial on 26 Mar 2020

oh, 1 more problem: how can I continue training from lastest check point? Everytime I re-run the training command, it train from start

EuphoriaCelestial on 26 Mar 2020

Not sure
checkpoint file contains the model. It's a dictionary containing
https://github.com/NVIDIA/tacotron2/blob/master/train.py#L115#L118
The iteration, model state and optimizer state.
Use -c PATH/TO/CHECKPOINT.

CookiePPP on 26 Mar 2020

👍2

@CookiePPP ok, so can you give me some guidelines on how to load the model in checkpoint file and do some inference?

EuphoriaCelestial on 26 Mar 2020

and is there anyway to create .pt model file from checkpoint file?

EuphoriaCelestial on 26 Mar 2020

@EuphoriaCelestial
https://github.com/NVIDIA/tacotron2#inference-demo

Inference demo

Download our published [Tacotron 2] model
Download our published [WaveGlow] model
jupyter notebook --ip=127.0.0.1 --port=31337
Load inference.ipynb
Change checkpoint_path to whatever you want and text to your input text.

and is there anyway to create .pt model file from checkpoint file?

I'm not sure exactly what a .pt does, but you should be able to copy the Line from above and make another function that does what you need.

CookiePPP on 26 Mar 2020

I'm not sure exactly what a .pt does, but you should be able to copy the Line from above and make another function that does what you need.

your published model is a .pt file, I tried to run the inference demo file, and it run perfectly when load in the published model, but when I tried to load checkpoint file into checkpoint_path, it return an error:
NameError: name 'load_model' is not defined

EuphoriaCelestial on 26 Mar 2020

I'm not sure exactly what a .pt does, but you should be able to copy the Line from above and make another function that does what you need.

your published model is a .pt file, I tried to run the inference demo file, and it run perfectly when load in the published model, but when I tried to load checkpoint file into checkpoint_path, it return an error:
NameError: name 'load_model' is not defined

ah, nevermind, it was my fault, I didnt restart Jupyter before run

EuphoriaCelestial on 26 Mar 2020

@CookiePPP final question, do I need to re-train Waveglow model if using new tacotron2 model which is trained for another language? And how to do so?

EuphoriaCelestial on 26 Mar 2020

@EuphoriaCelestial
I don't know.
https://arxiv.org/pdf/1912.02461.pdf

4.2.3. Unseen speakers and unseen language
For unseen languages, WaveNet’s performance is comparable to the
case of seen language. Whereas WaveRNN degrades a lot in terms
of performance, as it is sensitivity to the mismatch of seen/unseen
languages.

Research paper failed to train WaveGlow.
Wavenet is good for any Language
WaveRNN is only for one Language per model

And how to do so?

It's same as Tacotron. WaveGlow takes a long time to train so make sure to use the pretrained model as the starting point.

CookiePPP on 26 Mar 2020

Ensure your Audio files are all WAV PCM 16 Bit files. You may have to convert them.

I notice that all audio file in LSJ dataset are 32 bit, 22050Hz, so why do I need my files to be 16bit?
and my files is 44100Hz, do I need to convert it to 22050Hz or just change the sampling_rate value in hparams.py file?

EuphoriaCelestial on 30 Mar 2020

@EuphoriaCelestial
https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.write.html
Scipy (used in this repo) will read 32 bit files as INT between -2147483648 | +2147483647
and 16 bit files as INT between -32768 | +32767

I forgot,
you can just change https://github.com/NVIDIA/tacotron2/blob/master/hparams.py#L35

max_wav_value=32768.0,

to 2147483648 and use 32 bit WAVs.

my files is 44100Hz, do I need to convert it to 22050Hz or just change the

Both options work.

(Option A)
Change
https://github.com/NVIDIA/tacotron2/blob/master/hparams.py#L36#L39

        sampling_rate=22050,
        filter_length=1024,
        hop_length=256,
        win_length=1024,

by 44100 / 22050 (2x) to make it compatible with the 22.05Khz pretrained WaveGlow.

        sampling_rate=44100,
        filter_length=2048,
        hop_length=512,
        win_length=2048,

(Option B)
Resample everything to 22050Hz and leave filter_length,hop_length,win_length default.

(Option C)
Leave everything default and train a 44.1Khz WaveGlow (this will takes weeks/months, not recommended).

CookiePPP on 30 Mar 2020

I choose option A, and use params like below:
################################ # Audio Parameters # ################################ max_wav_value=2147483648.0, sampling_rate=44100, filter_length=2048, hop_length=512, win_length=2048, n_mel_channels=80, mel_fmin=0.0, mel_fmax=8000.0,

and got this error:

FP16 Run: False Dynamic Loss Scaling: True Distributed Run: False cuDNN Enabled: True cuDNN Benchmark: False Epoch: 0 /home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/scipy/io/wavfile.py:273: WavFileWarning: Chunk (non-data) not understood, skipping it. WavFileWarning) Train loss 0 162.724335 Grad Norm 25.745108 1.56s/it /home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/scipy/io/wavfile.py:273: WavFileWarning: Chunk (non-data) not understood, skipping it. WavFileWarning) Traceback (most recent call last): File "train.py", line 290, in <module> args.warm_start, args.n_gpus, args.rank, args.group_name, hparams) File "train.py", line 248, in train hparams.distributed_run, rank) File "train.py", line 132, in validate for i, batch in enumerate(val_loader): File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ data = self._next_data() File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise() File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise raise self.exc_type(msg) RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop data = fetcher.fetch(index) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/hangtg/Music/tacotron2/data_utils.py", line 61, in __getitem__ return self.get_mel_text_pair(self.audiopaths_and_text[index]) File "/home/hangtg/Music/tacotron2/data_utils.py", line 34, in get_mel_text_pair mel = self.get_mel(audiopath) File "/home/hangtg/Music/tacotron2/data_utils.py", line 46, in get_mel melspec = self.stft.mel_spectrogram(audio_norm) File "/home/hangtg/Music/tacotron2/layers.py", line 73, in mel_spectrogram assert(torch.min(y.data) >= -1) RuntimeError: invalid argument 1: cannot perform reduction function min on tensor with no elements because the operation does not have an identity at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:345

EuphoriaCelestial on 30 Mar 2020

@EuphoriaCelestial
I don't know, I leave you to figure that out.

CookiePPP on 30 Mar 2020

but the params I am using is correct for 32 bit, 44100 .wav file, right?

EuphoriaCelestial on 30 Mar 2020

@EuphoriaCelestial
That should work.
I have never used 32 bit before.
Do you think your files might be float32 or int32?
https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.read.html

Maybe you should have max_wav_value=1.0,?
I do not know.

CookiePPP on 30 Mar 2020

@CookiePPP I am able to run training now, but I got this error when it try to save checkpoint

Traceback (most recent call last): File "train.py", line 290, in <module> args.warm_start, args.n_gpus, args.rank, args.group_name, hparams) File "train.py", line 248, in train hparams.distributed_run, rank) File "train.py", line 134, in validate y_pred = model(x) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/hangtg/Music/tacotron2/model.py", line 508, in forward encoder_outputs, mels, memory_lengths=text_lengths) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/hangtg/Music/tacotron2/model.py", line 408, in forward decoder_input) File "/home/hangtg/Music/tacotron2/model.py", line 363, in decode attention_weights_cat, self.mask) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/hangtg/Music/tacotron2/model.py", line 77, in forward attention_hidden_state, processed_memory, attention_weights_cat) File "/home/hangtg/Music/tacotron2/model.py", line 60, in get_alignment_energies processed_query + processed_attention_weights + processed_memory)) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 13413) is killed by signal: Killed.

I noticed that when saving checkpoint, it consume huge amount of RAM, around 29-31GB. Sometimes it can save checkpoint, but sometimes it use all of my RAM and crash like above. I tried reduce batch size to 16, number_of_epochs = 5000, save checkpoint after 500 epochs

EuphoriaCelestial on 30 Mar 2020

@CookiePPP I found this in text/cmudict.py file, what is it? should I change it?
valid_symbols = [ 'AA', 'AA0', 'AA1', 'AA2', 'AE', 'AE0', 'AE1', 'AE2', 'AH', 'AH0', 'AH1', 'AH2', 'AO', 'AO0', 'AO1', 'AO2', 'AW', 'AW0', 'AW1', 'AW2', 'AY', 'AY0', 'AY1', 'AY2', 'B', 'CH', 'D', 'DH', 'EH', 'EH0', 'EH1', 'EH2', 'ER', 'ER0', 'ER1', 'ER2', 'EY', 'EY0', 'EY1', 'EY2', 'F', 'G', 'HH', 'IH', 'IH0', 'IH1', 'IH2', 'IY', 'IY0', 'IY1', 'IY2', 'JH', 'K', 'L', 'M', 'N', 'NG', 'OW', 'OW0', 'OW1', 'OW2', 'OY', 'OY0', 'OY1', 'OY2', 'P', 'R', 'S', 'SH', 'T', 'TH', 'UH', 'UH0', 'UH1', 'UH2', 'UW', 'UW0', 'UW1', 'UW2', 'V', 'W', 'Y', 'Z', 'ZH' ]

EuphoriaCelestial on 13 Apr 2020

I choose option A, and use params like below:
################################ # Audio Parameters # ################################ max_wav_value=2147483648.0, sampling_rate=44100, filter_length=2048, hop_length=512, win_length=2048, n_mel_channels=80, mel_fmin=0.0, mel_fmax=8000.0,

and got this error:

FP16 Run: False Dynamic Loss Scaling: True Distributed Run: False cuDNN Enabled: True cuDNN Benchmark: False Epoch: 0 /home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/scipy/io/wavfile.py:273: WavFileWarning: Chunk (non-data) not understood, skipping it. WavFileWarning) Train loss 0 162.724335 Grad Norm 25.745108 1.56s/it /home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/scipy/io/wavfile.py:273: WavFileWarning: Chunk (non-data) not understood, skipping it. WavFileWarning) Traceback (most recent call last): File "train.py", line 290, in <module> args.warm_start, args.n_gpus, args.rank, args.group_name, hparams) File "train.py", line 248, in train hparams.distributed_run, rank) File "train.py", line 132, in validate for i, batch in enumerate(val_loader): File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ data = self._next_data() File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise() File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise raise self.exc_type(msg) RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop data = fetcher.fetch(index) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/hangtg/Music/tacotron2/data_utils.py", line 61, in __getitem__ return self.get_mel_text_pair(self.audiopaths_and_text[index]) File "/home/hangtg/Music/tacotron2/data_utils.py", line 34, in get_mel_text_pair mel = self.get_mel(audiopath) File "/home/hangtg/Music/tacotron2/data_utils.py", line 46, in get_mel melspec = self.stft.mel_spectrogram(audio_norm) File "/home/hangtg/Music/tacotron2/layers.py", line 73, in mel_spectrogram assert(torch.min(y.data) >= -1) RuntimeError: invalid argument 1: cannot perform reduction function min on tensor with no elements because the operation does not have an identity at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:345

how do you fix this problem? assert(torch.min(y.data) >= -1)

HeyNuts on 19 May 2020

how do you fix this problem? assert(torch.min(y.data) >= -1)

I dont know where your error come from, but I got something similar to this because there is some corrupt file in training data. I just wrote a tool to remove all broken files and it work fine.

EuphoriaCelestial on 19 May 2020

I choose option A, and use params like below:
################################ # Audio Parameters # ################################ max_wav_value=2147483648.0, sampling_rate=44100, filter_length=2048, hop_length=512, win_length=2048, n_mel_channels=80, mel_fmin=0.0, mel_fmax=8000.0,
and got this error:
FP16 Run: False Dynamic Loss Scaling: True Distributed Run: False cuDNN Enabled: True cuDNN Benchmark: False Epoch: 0 /home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/scipy/io/wavfile.py:273: WavFileWarning: Chunk (non-data) not understood, skipping it. WavFileWarning) Train loss 0 162.724335 Grad Norm 25.745108 1.56s/it /home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/scipy/io/wavfile.py:273: WavFileWarning: Chunk (non-data) not understood, skipping it. WavFileWarning) Traceback (most recent call last): File "train.py", line 290, in <module> args.warm_start, args.n_gpus, args.rank, args.group_name, hparams) File "train.py", line 248, in train hparams.distributed_run, rank) File "train.py", line 132, in validate for i, batch in enumerate(val_loader): File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ data = self._next_data() File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise() File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise raise self.exc_type(msg) RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop data = fetcher.fetch(index) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/hangtg/Music/tacotron2/data_utils.py", line 61, in __getitem__ return self.get_mel_text_pair(self.audiopaths_and_text[index]) File "/home/hangtg/Music/tacotron2/data_utils.py", line 34, in get_mel_text_pair mel = self.get_mel(audiopath) File "/home/hangtg/Music/tacotron2/data_utils.py", line 46, in get_mel melspec = self.stft.mel_spectrogram(audio_norm) File "/home/hangtg/Music/tacotron2/layers.py", line 73, in mel_spectrogram assert(torch.min(y.data) >= -1) RuntimeError: invalid argument 1: cannot perform reduction function min on tensor with no elements because the operation does not have an identity at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:345

how do you fix this problem? assert(torch.min(y.data) >= -1)

did you fix it?

Endreje on 22 May 2020

@Endreje

This error occurs if one of the audio samples is larger than max_wav_value.
If max_wav_value is too high then all audio will be quiet when processed,
If max_wav_value is 2¹⁶ times higher than max, then most audio will be silent when processed and spectrograms will look empty/purple (similar issue here),
if max_wav_value is too low then you will get this assert(torch.min(y.data) >= -1) error

Make sure max_wav_value
https://github.com/NVIDIA/tacotron2/blob/0102db28e2e167146b20ab72a9e8dc17d78f3a4f/hparams.py#L35
is actually the maximum value of your wavs.
For 16-bit PCM files, it should be 2¹⁵.
For 32-bit PCM files it should be 2³¹.
If it still has this error then it may be a rounding issue?
It would be safest to convert all your files into 16 bit mono and just using the default value if you cannot find another fix.

CookiePPP on 22 May 2020

@Endreje

This error occurs if one of the audio samples is larger than max_wav_value.
If max_wav_value is too high then all audio will be quiet when processed,
If max_wav_value is 216 times higher than max, then most audio will be silent when processed and spectrograms will look empty/purple (issue here),
if max_wav_value is too low then you will get this assert(torch.min(y.data) >= -1) error

Make sure max_wav_value

https://github.com/NVIDIA/tacotron2/blob/0102db28e2e167146b20ab72a9e8dc17d78f3a4f/hparams.py#L35

is actually the maximum value of your wavs.
For 16-bit PCM files, it should be 215.
For 32-bit PCM files it should be 231.
If it still has this error then it may be a rounding issue?
It would be safest to convert all your files into 16 bit mono and just using the default value if you cannot find another fix.

My data show up as 32bit-float in Audacity, but I still be able to train with max_wav_value=32768.0 because I removed all larger files.
Will it cause any problem in future? Should I change max_wav_value to 2^31 and continue the training or I have to restart all again?

EuphoriaCelestial on 22 May 2020

@EuphoriaCelestial
paste
Top files should be 32-bit PCM, bottom output.wav should be 16-bit PCM.
My Audacity is not showing bit properly, can you check yours?

edit:
I don't think Audacity is good for checking bit depth

edit2:
Don't worry about it, you would have seen big problems by now if it was set wrong.
Either 0.0 loss or assertion errors.

CookiePPP on 22 May 2020

@Endreje
This error occurs if one of the audio samples is larger than max_wav_value.
If max_wav_value is too high then all audio will be quiet when processed,
If max_wav_value is 216 times higher than max, then most audio will be silent when processed and spectrograms will look empty/purple (issue here),
if max_wav_value is too low then you will get this assert(torch.min(y.data) >= -1) error
Make sure max_wav_value
https://github.com/NVIDIA/tacotron2/blob/0102db28e2e167146b20ab72a9e8dc17d78f3a4f/hparams.py#L35

is actually the maximum value of your wavs.
For 16-bit PCM files, it should be 215.
For 32-bit PCM files it should be 231.
If it still has this error then it may be a rounding issue?
It would be safest to convert all your files into 16 bit mono and just using the default value if you cannot find another fix.

My data show up as 32bit-float in Audacity, but I still be able to train with max_wav_value=32768.0 because I removed all larger files.
Will it cause any problem in future? Should I change max_wav_value to 2^31 and continue the training or I have to restart all again?

I used soxi on linux to compare LJ.wav file with my .wav file. I made sure I exported all the data as is in soxi on LJ.wav. The max_wav_value makes great point, Ill look into it.
At the moment, here is the config to my .wav files.

Input File : 'LJ001-0001.wav'
Channels : 1
Sample Rate : 22050
Precision : 16-bit
Duration : 00:00:09.66 = 212893 samples ~ 724.126 CDDA sectors
File Size : 426k
Bit Rate : 353k
Sample Encoding: 16-bit Signed Integer PCM

Input File : 'data/annakarenina_001_tolstoy_64kb_41930.wav'
Channels : 1
Sample Rate : 22050
Precision : 16-bit
Duration : 00:00:04.00 = 88199 samples ~ 299.997 CDDA sectors
File Size : 176k
Bit Rate : 353k
Sample Encoding: 16-bit Signed Integer PCM

NEW EDIT:
I just checked, the length of some of my files is 0. Can this be the cause?
My max wav value is 13740, in hparams.py its 32768.0.

Endreje on 22 May 2020

@EuphoriaCelestial

Top files should be 32-bit PCM, bottom output.wav should be 16-bit PCM.
My Audacity is not showing bit properly, can you check yours?

edit:
I don't think Audacity is good for checking bit depth

edit2:
Don't worry about it, you would have seen big problems by now if it was set wrong.
Either 0.0 loss or assertion errors.

I used soundfile to check bit-depth, turned out all my data is 16 bit PCM. Audacity fooled me all this time.

EuphoriaCelestial on 22 May 2020

👍1

Closing due to inactivity.

rafaelvalle on 25 Jun 2020

I choose option A, and use params like below:
################################ # Audio Parameters # ################################ max_wav_value=2147483648.0, sampling_rate=44100, filter_length=2048, hop_length=512, win_length=2048, n_mel_channels=80, mel_fmin=0.0, mel_fmax=8000.0,

and got this error:

FP16 Run: False Dynamic Loss Scaling: True Distributed Run: False cuDNN Enabled: True cuDNN Benchmark: False Epoch: 0 /home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/scipy/io/wavfile.py:273: WavFileWarning: Chunk (non-data) not understood, skipping it. WavFileWarning) Train loss 0 162.724335 Grad Norm 25.745108 1.56s/it /home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/scipy/io/wavfile.py:273: WavFileWarning: Chunk (non-data) not understood, skipping it. WavFileWarning) Traceback (most recent call last): File "train.py", line 290, in <module> args.warm_start, args.n_gpus, args.rank, args.group_name, hparams) File "train.py", line 248, in train hparams.distributed_run, rank) File "train.py", line 132, in validate for i, batch in enumerate(val_loader): File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__ data = self._next_data() File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise() File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/_utils.py", line 394, in reraise raise self.exc_type(msg) RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last): File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop data = fetcher.fetch(index) File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/hangtg/Music/tacotron2-env/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/hangtg/Music/tacotron2/data_utils.py", line 61, in __getitem__ return self.get_mel_text_pair(self.audiopaths_and_text[index]) File "/home/hangtg/Music/tacotron2/data_utils.py", line 34, in get_mel_text_pair mel = self.get_mel(audiopath) File "/home/hangtg/Music/tacotron2/data_utils.py", line 46, in get_mel melspec = self.stft.mel_spectrogram(audio_norm) File "/home/hangtg/Music/tacotron2/layers.py", line 73, in mel_spectrogram assert(torch.min(y.data) >= -1) RuntimeError: invalid argument 1: cannot perform reduction function min on tensor with no elements because the operation does not have an identity at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:345

I too had this same issue. the problem was that my audio files were getting converted to 32-bit pcm when I was adding silences using sox.
the solution was to add sox -b 16 and things were getting output as 16-bit pcm so the default hparams could be used

pathnirvana on 23 Jan 2021

@Endreje
This error occurs if one of the audio samples is larger than max_wav_value.
If max_wav_value is too high then all audio will be quiet when processed,
If max_wav_value is 216 times higher than max, then most audio will be silent when processed and spectrograms will look empty/purple (issue here),
if max_wav_value is too low then you will get this assert(torch.min(y.data) >= -1) error
Make sure max_wav_value
https://github.com/NVIDIA/tacotron2/blob/0102db28e2e167146b20ab72a9e8dc17d78f3a4f/hparams.py#L35

is actually the maximum value of your wavs.
For 16-bit PCM files, it should be 215.
For 32-bit PCM files it should be 231.
If it still has this error then it may be a rounding issue?
It would be safest to convert all your files into 16 bit mono and just using the default value if you cannot find another fix.

My data show up as 32bit-float in Audacity, but I still be able to train with max_wav_value=32768.0 because I removed all larger files.
Will it cause any problem in future? Should I change max_wav_value to 2^31 and continue the training or I have to restart all again?

I used soxi on linux to compare LJ.wav file with my .wav file. I made sure I exported all the data as is in soxi on LJ.wav. The max_wav_value makes great point, Ill look into it.
At the moment, here is the config to my .wav files.

Input File : 'LJ001-0001.wav'
Channels : 1
Sample Rate : 22050
Precision : 16-bit
Duration : 00:00:09.66 = 212893 samples ~ 724.126 CDDA sectors
File Size : 426k
Bit Rate : 353k
Sample Encoding: 16-bit Signed Integer PCM

Input File : 'data/annakarenina_001_tolstoy_64kb_41930.wav'
Channels : 1
Sample Rate : 22050
Precision : 16-bit
Duration : 00:00:04.00 = 88199 samples ~ 299.997 CDDA sectors
File Size : 176k
Bit Rate : 353k
Sample Encoding: 16-bit Signed Integer PCM

NEW EDIT:
I just checked, the length of some of my files is 0. Can this be the cause?
My max wav value is 13740, in hparams.py its 32768.0.

I had the same issue, even though my stats on sox were identical with LJ Speech. In my case this happened because some files were empty (because of issues in how I was segmenting the audio).

You can verify if you have empty wavs (44 bytes) in your dataset. Or use something like the snippet below to count the number of frames in your files and them remove them from your dataset.

import wave
f = wave.open(f"{root}/{file}", mode="rb")
frames = f.getnframes()
if frames == 0:
        print(f"{root}/{file}")

rhlbhatnagar on 16 Feb 2021

FP16 Run: False
Dynamic Loss Scaling: True
Distributed Run: False
cuDNN Enabled: True
cuDNN Benchmark: False
Epoch: 0
Traceback (most recent call last):
File "train.py", line 290, in
args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
File "train.py", line 208, in train
for i, batch in enumerate(train_loader):
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 517, in __next__
data = self._next_data()
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1199, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
data.reraise()
File "/usr/local/lib/python3.7/dist-packages/torch/_utils.py", line 429, in reraise
raise self.exc_type(msg)
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/content/drive/My Drive/tacotron/data_utils.py", line 61, in __getitem__
return self.get_mel_text_pair(self.audiopaths_and_text[index])
File "/content/drive/My Drive/tacotron/data_utils.py", line 34, in get_mel_text_pair
mel = self.get_mel(audiopath)
File "/content/drive/My Drive/tacotron/data_utils.py", line 39, in get_mel
audio, sampling_rate = load_wav_to_torch(filename)
File "/content/drive/My Drive/tacotron/utils.py", line 14, in load_wav_to_torch
sampling_rate, data = read(full_path)
File "/usr/local/lib/python3.7/dist-packages/scipy/io/wavfile.py", line 264, in read