Tacotron2: reduction window is vital for the model to pick up alignment.

[bfs18]

The hparams.py says n_frames_per_step=1, # currently only 1 is supported, but reduction window is very important for them model to pick up alignment. Using a reduction window can be considered as dropping teacher forcing frames at equal intervals, and thus increases the information gap between the teacher forcing input and the target. Tacotron2 tends to predict the target from the autoregressive input (teacher forcing input at training) without exploiting the conditional text if the information gap is not large enough.
The reduction window can be replaced by a frame dropout trick if it is not continent to implement in the current code. Just set the teacher forcing input frames to the global mean according to a certain percentage.
In implement this in my fork. It can pick up alignment at much earlier steps without warmstart.
my fork

NVIDIA-tacotorn2

[xDuck]

Read up a bit on your implementation and it seems very promising. Going to give it a go with a fork I've been working on that is struggling to learn attention fully. I was looking into applying something similar (but not nearly as elegant) myself.

Can you provide a link to the paper you reference in your fork's README?

[bfs18]

Hi, the paper is available at https://arxiv.org/abs/1909.01145

[onyedikilo]

@bfs18 Hi, tried your fork but somehow I am getting NaN's on gradient.norm and mi loss, any ideas? I trained master successfully with the same data.

[bfs18]

@bfs18 Hi, tried your fork but somehow I am getting NaN's on gradient.norm and mi loss, any ideas? I trained master successfully with the same data.

Hi @onyedikilo

1. You can only use drop frame first, just set use_mmi=False.
2. If you would like to use mmi, make sure that the blank index and vocab size are correctly setted (torch.CTCLoss doc). Besides silent symbols, such as 'SPACE' and punctuation, should be avoided in the CTC target. Finally you may try reducing gaf to 0.1.

[onyedikilo]

@bfs18 Sorry I couldn't understand what you meant with

You can only use drop frame first

Can you explain it in different words?

[bfs18]

Hi @onyedikilo
I add several new options in hparams.py in my fork.

• use_mmi (use mmi training objective or not)
• use_gaf (use gradient adaptive factor or not, to keep the max norm of gradients from the taco_loss and mi_loss approximately equal)
• max_gaf (maximum value of gradient adaptive factor)
• drop_frame_rate (drop teacher-frocing input frames to a certain rate)

When setting use_mmi=False and drop_frame_rate to a value in range (0., 1.), only drop frame trick is used.

[hadaev8]

@bfs18
Why you have this line?
https://github.com/bfs18/tacotron2/blob/master/train.py#L253

[bfs18]

@bfs18
Why you have this line?
https://github.com/bfs18/tacotron2/blob/master/train.py#L253

Hi @hadaev8 , this line has no influence on the numerical values of gradients. When calculating taco_loss, the variables of CTC recognizer is not used, so the gradients of taco_loss with respect to these variables are None. The gradients become zero tensors after adding this line.

[onyedikilo]

I can confirm that the alignment picks up significantly faster with my data set.

[bfs18]

I can confirm that the alignment picks up significantly faster with my data set.

Hi @onyedikilo , thanks a lot for your confirmation.

[hadaev8]

@bfs18
Any ideas why my alignment looks like this with CTC loss?
https://i.imgur.com/17Wz22v.png

[bfs18]

@bfs18
Any ideas why my alignment looks like this with CTC loss?
https://i.imgur.com/17Wz22v.png

Hi @hadaev8 , This is caused by that the CTC loss is over weighted. When CTC loss is over weighted, the model would depend more on the text input to reduce the total loss. It leads to a diagonal alignment combined with the Local Sensitive Attention.

Setting hparams.use_gaf=True and a smaller hparams.max_gaf, such as 0.1, would solve the problem.

[hadaev8]

Well, I read again paper
They say

In Tacotron2, the attention context is concatenated to the LSTM out-put and projected by a linear transform to predict the Mel spectrum.This means the predicated Mel spectrum contains linear componentsof the text information. If we use this Mel spectrum as the input tothe CTC recognizer, the text information is too easily accessible forthe recognizer. This may cause the text information to be encodedin a pathological way in the Mel spectrum and lead to a strict di-agonal alignment map (one acoustic frame output for one phonemeinput) combined with location-sensitive attention. So before the lin-ear transform operation, we add an extra LSTM layer to mix the textinformation and acoustic information.

Should you point where should be this lstm layer?

[bfs18]

Well, I read again paper
They say

In Tacotron2, the attention context is concatenated to the LSTM out-put and projected by a linear transform to predict the Mel spectrum.This means the predicated Mel spectrum contains linear componentsof the text information. If we use this Mel spectrum as the input tothe CTC recognizer, the text information is too easily accessible forthe recognizer. This may cause the text information to be encodedin a pathological way in the Mel spectrum and lead to a strict di-agonal alignment map (one acoustic frame output for one phonemeinput) combined with location-sensitive attention. So before the lin-ear transform operation, we add an extra LSTM layer to mix the textinformation and acoustic information.

Should you point where should be this lstm layer?

Hi, the paper uses a internal Tensorflow implementation. It is a bit different from the open-sourced fork. In the open-sourced fork a ff_layer with relu activation is used to mix the information. It is this line https://github.com/bfs18/tacotron2/blob/8f8605ee0f67f6f571e74725030f16b13e4c7d2d/model.py#L388

[xDuck]

Finally got around to trying out your fork on my modified spectrums and I can confirm it picked up attention much faster! Thanks!

[hadaev8]

@bfs18
Are you author of paper?
Do you know lstm mixer dim?

[bfs18]

Hi @hadaev8

Are you author of paper?

yes.

Do you know lstm mixer dim?

I just use the same dimension as the decoder_rnn_dim, of which the value is 1024.

[bfs18]

Finally got around to trying out your fork on my modified spectrums and I can confirm it picked up attention much faster! Thanks!

Hi @xDuck , I am glad to hear that.

[hadaev8]

@bfs18
I added lstm for mixing decoder outputs like this
https://pastebin.com/SNxAPcUD
but looks like it does not mix it enough.
Alignment crush bit later, but still.
Maybe it should be bi-directional lstm?
Or I doing wrong something?

[bfs18]

Hi @hadaev8 ,
Have you tried a smaller max_gaf?
The diagonal alignment is due to mixed causes. Usually a feed forward layer with a nonlinear activation function would mix the information sufficiently according to my later experiments.
It also leads to a corrupted alignment when gradients from the CTC loss dominates the training. Text information in the decoder_output is too much picked to reduce the total loss in such occasion.

[hadaev8]

@bfs18
Using gaf in distributed setup worse training.
So I trying this approach https://arxiv.org/pdf/1705.07115.pdf
With lstm mixer and feeding mel output to CTC recognizer makes it more stable, but still training crush later.

My gradients indeed suffer.
https://i.imgur.com/amcGgHJ.png
Orange is your original implementation, others are my expiriments.

[bfs18]

@bfs18
Using gaf in distributed setup worse training.
So I trying this approach https://arxiv.org/pdf/1705.07115.pdf
With lstm mixer and feeding mel output to CTC recognizer makes it more stable, but still training crush later.

My gradients indeed suffer.
https://i.imgur.com/amcGgHJ.png
Orange is your original implementation, others are my expiriments.

Hi @hadaev8 ,

The paper is bit complicated. I haven't go through it.

gaf is just a dynamic weight for the mi_loss.
https://github.com/bfs18/tacotron2/blob/8f8605ee0f67f6f571e74725030f16b13e4c7d2d/train.py#L259
You can use a small weight, such as 1e-2 or 1e-3, instead of gaf. You can even use an annealing schedule just like the KL-annealing trick. In my experiments, the gaf become very small after 10k steps.

[xDuck]

Hey @bfs18 Just wanted to let you know your fork is working great with my GST adaption as well based on Google's GST paper. Alignment learning super quickly and my models produce recognizable speech in about 3 hours on a 2070 Graphics card - way faster than before.

[rafaelvalle]

@bfs18 what are the most important changes for the results @xDuck mentioned?

[xDuck]

@rafaelvalle I should mention I am using bark-scale spectrograms with 18 channels and 2 pitch features for my spectrograms along with a lpcnet-forked vocoder (Targetting faster than realtime CPU inference. Currently 1/3 realtime speed on a 2017 macbook pro for synthesis). I have noticed in general that speeds up training a lot too (less features to predict). Samples attached of her after not much training with different GST reference clips. Single-Speaker LJSpeech used - These are from my very first test.

gst_results.zip

Alignment after 3k steps w/ batch size 20

[bfs18]

Hey @bfs18 Just wanted to let you know your fork is working great with my GST adaption as well based on Google's GST paper. Alignment learning super quickly and my models produce recognizable speech in about 3 hours on a 2070 Graphics card - way faster than before.

Hi @xDuck , thanks for your information.

[bfs18]

@bfs18 what are the most important changes for the results @xDuck mentioned?

Hi @rafaelvalle , setting the teacher forcing input to the global mean to a certain percentage is a stable trick, which boosts up alignment learning a lot. The extra CTC loss would speed up alignment learning and reduce bad case. However, it is a bit tricky to tune.

[hadaev8]

I can align this bad boy slaps tacotron with only 2k steps.

Also wondering why you here have it not aligned on the decoder timestep axis.

[bfs18]

Hi @hadaev8

I can align this bad boy _slaps tacotron_ with only 2k steps.

How did you solve your problem?

Also wondering why you here have it not aligned on the decoder timestep axis.

I don't quite get what are you trying to say. I guess you are saying the tail of the alignment is different form the above figures. It's a bit wired. I am also wondering. However, the padding frames are not important.

[hadaev8]

I turn off ctc loss then it became too low.

[CookiePPP]

@bfs18
My mel_predicted looks very over-smoothed, I assume a result of not understanding the hparams. What should I be changing to improve/fix the predicted mels?
(I'm having trouble getting training loss under 0.3 and I assume that's also related)
This is at 130000 steps, annealed learning rate to 1e-5.

(Thank you for the hard work)

[bfs18]

Hi @CookiePPP ,
You may try smaller dropout rate in these 2 lines, e.g. setting p=0.2.
https://github.com/bfs18/tacotron2/blob/8f8605ee0f67f6f571e74725030f16b13e4c7d2d/model.py#L240
https://github.com/bfs18/tacotron2/blob/8f8605ee0f67f6f571e74725030f16b13e4c7d2d/model.py#L246

[CookiePPP]

@bfs18
Alright, I've changed dropout to 0.1 on both lines, I'm restarting training from the start of LR decay.
I'll post back in a few days with the results (if that's of interest to anyone here).
EDIT:


I had to move on before Learning rate fully decayed but the effect is still visible, minor improvement but not enough for a 18Khz 80 Channel Mel-Spec.

[rafaelvalle]

Yes, please post the results of your experiments as they can be valuable to our community.

[Truong-Thanh-Quang]

@bfs18 : When I start training, error apeared.

FP16 Run: True
Dynamic Loss Scaling: True
Distributed Run: True
cuDNN Enabled: True
cuDNN Benchmark: False
Initializing Distributed
Done initializing distributed
Selected optimization level O2: FP16 training with FP32 batchnorm and FP32 master weights.

Defaults for this optimization level are:
enabled : True
opt_level : O2
cast_model_type : torch.float16
patch_torch_functions : False
keep_batchnorm_fp32 : True
master_weights : True
loss_scale : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled : True
opt_level : O2
cast_model_type : torch.float16
patch_torch_functions : False
keep_batchnorm_fp32 : True
master_weights : True
loss_scale : dynamic
Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'",)
Epoch: 0
Traceback (most recent call last):
File "train.py", line 342, in
args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
File "train.py", line 242, in train
y_pred = model(x)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(input, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/apex-0.1-py3.6.egg/apex/amp/_initialize.py", line 197, in new_fwd
*applier(kwargs, input_caster))
File "/home/traithivang/works/tts2/tacotron2/model.py", line 565, in forward
) = self.decoder(encoder_outputs, mels, memory_lengths=text_lengths)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(input, *kwargs)
File "/home/traithivang/works/tts2/tacotron2/model.py", line 411, in forward
decoder_inputs = torch.cat((decoder_input, decoder_inputs), dim=0)
RuntimeError: Expected object of scalar type Half but got scalar type Float for sequence element 1 in sequence argument at position #1 'tensors'
Traceback (most recent call last):
File "train.py", line 342, in
args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
File "train.py", line 242, in train
y_pred = model(x)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(input, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/apex-0.1-py3.6.egg/apex/amp/_initialize.py", line 197, in new_fwd
*applier(kwargs, input_caster))
File "/home/traithivang/works/tts2/tacotron2/model.py", line 565, in forward
) = self.decoder(encoder_outputs, mels, memory_lengths=text_lengths)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(input, *kwargs)
File "/home/traithivang/works/tts2/tacotron2/model.py", line 411, in forward
decoder_inputs = torch.cat((decoder_input, decoder_inputs), dim=0)
RuntimeError: Expected object of scalar type Half but got scalar type Float for sequence element 1 in sequence argument at position #1 'tensors'

Source code is checked out from https://github.com/bfs18/tacotron2.

Please help me. Thanks!

[bfs18]

Hi @quangtt2016 , I guess setting hparams.fp16_run=False would solve your problem, my code is not tested with fp16.

[kannadaraj]

@bfs18 : Thanks for sharing this code.. What do you think i need to do to have fp16 compatibility

bfs/tacotron2/utils.py", line 24, in dropout_frame

dropped_mels = (mels.half() * (1.0 - drop_mask).unsqueeze(1) +
RuntimeError: expected device cuda:5 and dtype Float but got device cuda:5 and dtype Half

[machineko]

@bfs18 Did u have some pretrained model for ur implementation? And also what about smaller dataset training should we change something in hparms?

[kannadaraj]

@bfs18 : Thanks for the wonderful job. I was able to incorporate MMI in GST framework and it reduced the training time substantially. this is the alignment graph after 10K steps.

My only comments it that in distributed multi GPU mode the performance goes bad. Once i turned GAF it started behaving well. With minor modification i was able to train at fp16 AMP that can speedup the process.

[grey-area]

Question: if you modify the original Tacotron2 code to drop the teacher forcing input frame rate, does it pick up alignments faster than it otherwise would?

Put another way: the code in your fork picks up alignments faster than the original Tacotron2. To what extent is this due to dropping teacher forcing frames, and to what extent is it due to other factors?

[Truong-Thanh-Quang]

@kannadaraj: Can you share the code changes for fp16 compatibility?

[bfs18]

Question: if you modify the _original_ Tacotron2 code to drop the teacher forcing input frame rate, does it pick up alignments faster than it otherwise would?

Put another way: the code in your fork picks up alignments faster than the original Tacotron2. To what extent is this due to dropping teacher forcing frames, and to what extent is it due to other factors?

Hi @grey-area , I did not test this carefully. Only dropping teacher forcing frames would boost up alignment learning.

[xDuck]

Question: if you modify the _original_ Tacotron2 code to drop the teacher forcing input frame rate, does it pick up alignments faster than it otherwise would?

Put another way: the code in your fork picks up alignments faster than the original Tacotron2. To what extent is this due to dropping teacher forcing frames, and to what extent is it due to other factors?

I tested dropping just frames haphazardly before integrating this branch to my fork and I saw slight improvement, especially in models where there was not a lot of data or doing a bit of transfer learning with an extremely small dataset. Overall however this branch performed much better than just dropping frames. Also, dropping too many frames will negatively impact your training time.

[Thien223]

@bfs18 Could you please tell us how fast your model is?

How much time does it take to synthesize a normal length sentences?
Thank you in advanced!

[bfs18]

@bfs18 Could you please tell us how fast your model is?

How much time does it take to synthesize a normal length sentences?
Thank you in advanced!

Hi @tdplaza , my modification does not add any overheads at inference stage.

[Thien223]

@bfs18 Thank you for your hard works and kindness of sharing.

We are testing your model on our own dataset, the alignment at 36k steps is as following:

The alignment is faster aligned than original nvidia repo, but the inference results could not be played after synthesizing with your tacotron and nvidia waveglow. Anyway, thank you.

[bfs18]

Hi @tdplaza

The inference result could not be played

This may caused by waveform coding. You may use Audition to play the waveform or save the waveform with soundfile.write(fp, data, sr, 'PCM_16').

[Thien223]

@bfs18 yes, I will check that.

one last question, how could i extract the global mean (as ljspeech_global_mean.npy) for my dataset?

[bfs18]

@bfs18 yes, I will check that.

one last question, how could i extract the global mean (as ljspeech_global_mean.npy) for my dataset?

Hi @tdplaza
You may search calculate_global_mean in train.py

[Thien223]

Thank you so much

[rafaelvalle]

@bfs18 and others: can you rank your modifications according to how much they help with learning alignments faster? I've been following this thread and it seems to me that the main factor is dropping teacher forcing by some small factor, specially in small datasets.

[CookiePPP]

@rafaelvalle
DFR (Drop-frame-rate) was the main improvement I found.
MMI (Auxiliary Recognizer) seems to cause issues with punctuation alignment in my dataset/tests.

---

My dataset is multispeaker with mean 2.3s duration audio files. Highly emotive, approx 30 mins per speaker.

[xDuck]

@rafaelvalle i would also agree that dropping teacher forcing was the main thing that helped my model to learn alignment, although I believe the gradient adaptive factor was also just as beneficial. My work was transfer learning from multi speaker datasets to a new voice with approx 30min of new recordings. I was also using a 20-feature bark spectrogram so I could use an LPCNet vocoder to achieve faster than real-time CPU inference.

[rafaelvalle]

What DFR factor did you use? Do other folks see similar conclusions?

[CookiePPP]

@rafaelvalle
edit: I used values from 0.0 to 0.35, linearly increasing from 0.0 @ 60K iters to 0.35 @ 150K iters then tuning more precisely later on.
I believe it depends on the speaker(s) data.
I got best results with 0.25 initially.
Later added Blizzard2011 dataset (increasing dataset total duration from 51.4 to 67.7 hrs) and best results required decreasing DFR to 0.2.

---

I measure "best results" as follows;
I perform validation in 3 passes,

• 100% Teacher Forcing and 0% DFR
• Same Teacher Forcing and DFR as used in training.
• 0% Teacher Forcing and 0% DFR (effectively inference)

https://github.com/NVIDIA/tacotron2/pull/284
I use metric Average max attention weight with pass 3 (effectively inference), and I find that to be a good indicator of model alignment strength/robustness in real-world conditions.
I also use val_loss on first validation pass to determine spectrogram detail independently of alignment.
"best results" DFR would be the DFR that resulted in highest Average max attention weight during pass 3 after approx 100K iters.

[rafaelvalle]

Thank you for letting us know.

[bfs18]

MMI (Auxiliary Recognizer) seems to cause issues with punctuation alignment in my dataset/tests.

Hi @CookiePPP , non-vocable phonemes, such as punctuation, should be removed from CTC targets when using MMI objectiove. Because CTC loss cannot distinguish blank token and non-vocable phonemes.

[CookiePPP]

@bfs18
I would have used the same CTC targets you defined.
https://github.com/bfs18/tacotron2/blob/master/text/symbols.py#L20

I remember MMI increasing the strength of alignments for letters and phonemes however alignments around commas and periods was much worse, and resulted in the model being very unstable during inference. Any comma or period had a high chance of collapsing the alignment and producing the Warning! Reached max decoder steps warning.

It has been a long time since I used MMI so some of the details are a bit fuzzy (I have the logs somewhere if needed).
I believe the MMI objective just doesn't work with very emotive datasets or maybe my model was underfitting, not sure.
Also, yes I did use GAF.

[xDuck]

For what it’s worth, my dataset did have punctuation mid-speech that was
represented in the audio files and for the most part I have no issues with
it breaking on punctuation. I did notice that it was not as good with
punctuation however, where you could clearly see punctuation in the
alignment graph and it wasn’t perfect at inference. However I ended up
keeping MMI as it wasn’t a significant issue.

On Tue, Apr 28, 2020 at 12:31 PM Cookie [e-mail] wrote:

@bfs18 https://github.com/bfs18
I would have used the same CTC targets you defined.
https://github.com/bfs18/tacotron2/blob/master/text/symbols.py#L20

I remember MMI increasing the strength of alignments for letters and
phonemes however alignments around commas and periods was much worse, and
resulted in the model being very unstable during inference. Any comma or
period had a high chance of collapsing the alignment and producing the Warning!
Reached max decoder steps warning.

It has been a long time since I used MMI so some of the details are a bit
fuzzy (I have the logs somewhere if needed).
I believe the MMI objective just doesn't work with very emotive datasets
or maybe my model was underfitting, not sure.
Also, yes I did use GAF.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NVIDIA/tacotron2/issues/280#issuecomment-620717288,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABICRIPPRE3WXJTNJQENXQTRO4AHPANCNFSM4JIESUGQ
.

[chazo1994]

However I ended up keeping MMI as it wasn’t a significant issue.

Could you share your issue that your are facing with MMI? I trying to train MMI model, Alignment pick up faster even with small DFR, but out mel-spectrogram is so smooth, and taco-loss quite high (I tried with DFR from 0 to 0.2 but result be the same), taco-loss hard ti decrease lower than 0.3.
@bfs18 Could your share your pretrained model with LJSpeech dataset?
I attach my target and predicted mel-spectrogram bellow.
Predicted mel-spectrogram

Target mel-spectrogram

[chazo1994]

Does any one try to implement this paper https://ieeexplore.ieee.org/document/8703406
Which used pre-aligned posterior to guide attetion.
I tried to implement this method, but I I don't know how to map between whitespace character in alignments matrix (phoneme representation) with pre aligned posterior. Because in Forced alignment task don't align any whitespace.

[CookiePPP]

@chazo1994
Updating the dropout like @bfs18 stated earlier may help with Loss.
https://github.com/NVIDIA/tacotron2/issues/280#issuecomment-569564127

---

I had not heard of this paper before, though reading it, very interesting indeed!
I solved most of my alignment issues by using more data and a multispeaker model however I would definitely be interested in recreating the paper and using Guided Attention in later models.

In regards to your problem, I don't fully understand (I still consider myself new to Deep Learning so not too useful), but I will try to help with parts I do understand.

I've also heard guided attention from espnet a few times, though looking further, I believe they just Diagonal-Guided Attention.

Maybe explore FastSpeech/ForwardTacotron
https://github.com/as-ideas/ForwardTacotron#-training-your-own-model
for ways to generate alignments and input them to your loss function?

[chazo1994]

Hi @CookiePPP ,
You may try smaller dropout rate in these 2 lines, e.g. setting p=0.2.
https://github.com/bfs18/tacotron2/blob/8f8605ee0f67f6f571e74725030f16b13e4c7d2d/model.py#L240
https://github.com/bfs18/tacotron2/blob/8f8605ee0f67f6f571e74725030f16b13e4c7d2d/model.py#L246

I would try smaller dropout but over-smoothed mel spectrogram and horizontal line noise is big problems.

[hadaev8]

@bfs18
Why you dropout frames with mean value?

[chazo1994]

I report my results with MMI and DFR:

Drop Frame Rate = 0

• Alignment:
  
• Mels:
  

Drop Frame Rate = 0.1

• Alignment:
  
• Mels:
  
• Loss:
  

Drop Frame Rate = 0.2
_19k Step_

• alignment:
  
• mels:
  
• Loss:
  

_34k_Step_

• Alignment:
  
• Mels:
  

gaf is nan after 30k step (I modified code to train model with mixed_precision).
As you can see, my models converged soon but loss explode after 30k steps.
@bfs18
@rafaelvalle
@CookiePPP

[chazo1994]

@bfs18
In your paper, you decay learning rate by a factor sqrt(4000/step) from 4000 step. But your fork don't have any learning-rate decay code.

[bfs18]

@bfs18
Why you dropout frames with mean value?

Hi @hadaev8 , because this value would not distort the input values to the prenet a lot, then it would not distort the activation values to the following modules.

[bfs18]

Hi @chazo1994 It seems that numerical errors occurred in your running.

In your paper, you decay learning rate by a factor sqrt(4000/step) from 4000 step. But your fork don't have any learning-rate decay code.

I found gradient adaptive factor works better. I use that trick instead.

[hadaev8]

@bfs18
I wonder to try gaussian noise instead of a fixed value.
And thinking should I set the whole frame to a single value or add noise separatly to every frame value.

[rafaelvalle]

@chazo1994 thank you for sharing this, can you share spectrogram reconstruction training and validation loss?

[chazo1994]

@chazo1994 thank you for sharing this, can you share spectrogram reconstruction training and validation loss?

This is my validation loss with MMI and DFR=0.2.
I don't know how to get spectrogram reconstruction?

[lalindra-desilva]

@bfs18 Just trying out your fork for the first time and followed instructions in this thread with ljspeech pretrained model. Running into the following error. Any idea why?

RuntimeError: Error(s) in loading state_dict for Tacotron2: size mismatch for decoder.gate_layer.linear_layer.weight: copying a param with shape torch.Size([1, 1536]) from checkpoint, the shape in current model is torch.Size([1, 1024]).

Appreciate any feedback.

[terryyizhong]

@bfs18 Just trying out your fork for the first time and followed instructions in this thread with ljspeech pretrained model. Running into the following error. Any idea why?

RuntimeError: Error(s) in loading state_dict for Tacotron2: size mismatch for decoder.gate_layer.linear_layer.weight: copying a param with shape torch.Size([1, 1536]) from checkpoint, the shape in current model is torch.Size([1, 1024]).

Appreciate any feedback.

you cannot use the pretrained Tacotron2 model of this branch. The model structure has been modified.

[titocaco]

@bfs18 I am using your fork with my dataset and it just started to align, but I am facing some problems when I try to use inference.ipynb with the tacotron model trained with your fork and the waveglow model; but, when I use this very same waveglow model that I have mentioned with the tacotron model trained with the NVidia repository, I have no problem.

The problem is this one:

AttributeError: 'WN' object has no attribute 'cond_layer'

If I am not wrong, the convert_model.py (from WaveGlow) should be used in this case, right? I have used it, but this error persists.

I need to use WaveGlow.
Any ideas to solve this, please?

[CookiePPP]

@titocaco
Download WaveGlow repo from
https://github.com/NVIDIA/waveglow
and replace the one in the tacotron2 folder?
(WaveGlow from bfs18's repo might be out of date)

[titocaco]

@CookiePPP it works! Thank you! =)

[zhitiankai]

FP16 Run: False
Dynamic Loss Scaling: True
Distributed Run: False
cuDNN Enabled: True
cuDNN Benchmark: False
calculating global mean...
Traceback (most recent call last):
File "train.py", line 341, in
train(args.output_directory, args.log_directory, args.checkpoint_path, args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
File "train.py", line 193, in train
global_mean = calculate_global_mean(train_loader, hparams.global_mean_npy)
File "train.py", line 159, in calculate_global_mean
for i, batch in enumerate(data_loader):
File "/data2/user/ztk/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
return self._process_next_batch(batch)
File "/data2/user/ztk/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
File "/data2/user/ztk/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/data2/user/ztk/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/data2/user/ztk/tacotron2_tibetan/data_utils.py", line 63, in __getitem__
return self.get_mel_text_pair(self.audiopaths_and_text[index])
File "/data2/user/ztk/tacotron2_tibetan/data_utils.py", line 34, in get_mel_text_pair
mel = self.get_mel(audiopath)
File "/data2/user/ztk/tacotron2_tibetan/data_utils.py", line 46, in get_mel
melspec = self.stft.mel_spectrogram(audio_norm)
File "/data2/user/ztk/tacotron2_tibetan/layers.py", line 73, in mel_spectrogram
assert(torch.min(y.data) >= -1)
RuntimeError: invalid argument 1: tensor must have one dimension at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:574

When i run your code https://github.com/bfs18/tacotron2 , i meet this bug,Can you give me some suggestions?

[ErfolgreichCharismatisch]

I should mention I am using bark-scale spectrograms with 18 channels and 2 pitch features for my spectrograms along with a lpcnet-forked vocoder

Can lpcnet help me with this issue: https://github.com/NVIDIA/tacotron2/issues/463?

[xDuck]

Yes, Tacotron 2 + LPCNet should get you to be able to perform inference on
CPU but the best speeds I was able to achieve were about 2x real-time on a
current gen intel CPU with AVX2 support.

On Wed, Mar 24, 2021 at 6:06 AM Erfolgreich charismatisch <
@.*> wrote:

I should mention I am using bark-scale spectrograms with 18 channels and 2
pitch features for my spectrograms along with a lpcnet-forked vocoder

Can lpcnet help me with this issue: #463
https://github.com/NVIDIA/tacotron2/issues/463?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NVIDIA/tacotron2/issues/280#issuecomment-805668409,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABICRIOQ52NG47IWO5X6C7DTFG2RTANCNFSM4JIESUGQ
.

[ErfolgreichCharismatisch]

Interesting. How do you use your nvidia Tacotron2 model with LPCNet?

Yes, Tacotron 2 + LPCNet should get you to be able to perform inference on CPU but the best speeds I was able to achieve were about 2x real-time on a current gen intel CPU with AVX2 support.
…
On Wed, Mar 24, 2021 at 6:06 AM Erfolgreich charismatisch < @.*> wrote: I should mention I am using bark-scale spectrograms with 18 channels and 2 pitch features for my spectrograms along with a lpcnet-forked vocoder Can lpcnet help me with this issue: #463 <#463>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#280 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABICRIOQ52NG47IWO5X6C7DTFG2RTANCNFSM4JIESUGQ .

[xDuck]

You will have to adjust the number of mels (and maybe other Params) used
and feed it bark spectrograms for training from scratch. I made a lot of
modifications that I don’t really remember, but it is not a simple task.

On Wed, Mar 24, 2021 at 7:08 AM Erfolgreich charismatisch <
@.*> wrote:

Interesting. How do you use your nvidia Tacotron2 model with LPCNet?

Yes, Tacotron 2 + LPCNet should get you to be able to perform inference on
CPU but the best speeds I was able to achieve were about 2x real-time on a
current gen intel CPU with AVX2 support.
… <#m_7142798940015613202_>
On Wed, Mar 24, 2021 at 6:06 AM Erfolgreich charismatisch < @.*>
wrote: I should mention I am using bark-scale spectrograms with 18 channels
and 2 pitch features for my spectrograms along with a lpcnet-forked vocoder
Can lpcnet help me with this issue: #463
https://github.com/NVIDIA/tacotron2/issues/463 <#463
https://github.com/NVIDIA/tacotron2/issues/463>? — You are receiving
this because you were mentioned. Reply to this email directly, view it on
GitHub <#280 (comment)
https://github.com/NVIDIA/tacotron2/issues/280#issuecomment-805668409>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABICRIOQ52NG47IWO5X6C7DTFG2RTANCNFSM4JIESUGQ
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NVIDIA/tacotron2/issues/280#issuecomment-805731788,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABICRIJN3URWMU7W455TFCDTFHBZPANCNFSM4JIESUGQ
.

[ErfolgreichCharismatisch]

If you had to do it all over again, how would you start?

PS: Can you share a diff between your files and the vanilla files?

You will have to adjust the number of mels (and maybe other Params) used and feed it bark spectrograms for training from scratch. I made a lot of modifications that I don’t really remember, but it is not a simple task.

[xDuck]

I’ve already mostly abandoned the project after considering my research
“completed”. I do not have the diff accessible anymore, sorry. As for doing
it over again, now there are better alternatives like SqueezeWave, HiFi
GAN, etc. Keep in mind you will trade quality for speed in the vocoders, it
is hard to compare to the quality of WaveGlow.

This project was not designed to run on the CPU (rightfully so, NVIDA makes
GPUs not CPUs), so it might not be what you are looking for - but it does a
damn good job on GPUs.

On Wed, Mar 24, 2021 at 7:22 AM Erfolgreich charismatisch <
@.*> wrote:

Well, that does not exactly sound encouraging ;)

If you had to do it all over again, how would you start?

PS: Can you share a diff between your files and the vanilla files?

You will have to adjust the number of mels (and maybe other Params) used
and feed it bark spectrograms for training from scratch. I made a lot of
modifications that I don’t really remember, but it is not a simple task.

—

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NVIDIA/tacotron2/issues/280#issuecomment-805742441,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABICRIMYDWUMCMRV2SXQ7R3TFHDQ3ANCNFSM4JIESUGQ
.

[ErfolgreichCharismatisch]

Yes. Which setup would you recommend for my goal?

EDIT: I just tried SqeezeWave, but Nvidia is in it yet again, this time in apex. Therefore I get AssertionError: Torch not compiled with CUDA enabled.

I’ve already mostly abandoned the project after considering my research “completed”. I do not have the diff accessible anymore, sorry. As for doing it over again, now there are better alternatives like SqueezeWave, HiFi GAN, etc. Keep in mind you will trade quality for speed in the vocoders, it is hard to compare to the quality of WaveGlow. This project was not designed to run on the CPU (rightfully so, NVIDA makes GPUs not CPUs), so it might not be what you are looking for - but it does a damn good job on GPUs.

[xDuck]

I cannot give you a good answer without knowing everything about what you
want to do. I suggest you do some research and evaluate your options in
your setup.

On Wed, Mar 24, 2021 at 8:18 AM Erfolgreich charismatisch <
@.*> wrote:

Yes. Which setup would you recommend for my goal?

I’ve already mostly abandoned the project after considering my research
“completed”. I do not have the diff accessible anymore, sorry. As for doing
it over again, now there are better alternatives like SqueezeWave, HiFi
GAN, etc. Keep in mind you will trade quality for speed in the vocoders, it
is hard to compare to the quality of WaveGlow. This project was not
designed to run on the CPU (rightfully so, NVIDA makes GPUs not CPUs), so
it might not be what you are looking for - but it does a damn good job on
GPUs.

—

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NVIDIA/tacotron2/issues/280#issuecomment-805774694,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABICRIPOE47XMZXCJKNJ243TFHKALANCNFSM4JIESUGQ
.

[ErfolgreichCharismatisch]

Tutorial: Training on GPU with Colab, Inference with CPU on Server here.