Tensorflowtts: What is an expected average training speed of Tacotron2?

[jun-danieloh]

What I am seeing is about ~11s/it with v100 GPU. I just followed the latest instruction in this github. Did I miss something?

[dathudeptrai]

@jun-danieloh it should be 3-4s / 1it :))). The dataloader use fixed shape for training tacotron2 (use max_length of ur training mel) so maybe because ur max mel length is too long :)). The maximum mel length in ur data should be <= 850.

[nukes]

3-4s/1it is still to slow, i think someone should solve this problem: AR decoding cell is quite inefficient in tensorflow. basically 5-10 days for training tacotron model, which is pretty slow...

[jun-danieloh]

@nukes I totally agree with you. My tacotron2 in PyTorch shows 0.7~0.8 sec/it which is about 5x times faster with the better quality. Training speed is the biggest hurdle for me to use Tensorflow model.

[jun-danieloh]

@nukes Can you add more details about how decoding cell is quite inefficient in TF? I can probably work on this stuff to speed up the training speed myself.

[dathudeptrai]

@nukes I totally agree with you. My tacotron2 in PyTorch shows 0.7~0.8 sec/it which is about 5x times faster with the better quality. Training speed is the biggest hurdle for me to use Tensorflow model.

with r = 1 ?. can you share that repo :)), i can not believe that tacotron2 can train with 0.7~0.8 sec/it with r = 1 even it's mixed precision.

[jun-danieloh]

@dathudeptrai I am not able to share the repo at the moment. It is not mixed precision and r = 5.

[dathudeptrai]

@dathudeptrai I am not able to share the repo at the moment. It is not mixed precision and r = 5.

you can adjust my repo with r = 5 :)) it should run as fast as pytorch version. (https://github.com/TensorSpeech/TensorFlowTTS/blob/master/examples/tacotron2/conf/tacotron2.v1.yaml#L35)

[jun-danieloh]

@jun-danieloh it should be 3-4s / 1it :))). The dataloader use fixed shape for training tacotron2 (use max_length of ur training mel) so maybe because ur max mel length is too long :)). The maximum mel length in ur data should be <= 850.

@dathudeptrai What it the r value for 3-4s/it??

[dathudeptrai]

@jun-danieloh it should be 3-4s / 1it :))). The dataloader use fixed shape for training tacotron2 (use max_length of ur training mel) so maybe because ur max mel length is too long :)). The maximum mel length in ur data should be <= 850.

@dathudeptrai What it the r value for 3-4s/it??

r = 1

[dathudeptrai]

the reason why my tacotron is slower caused by static shape. That mean the decoder steps always equal by maximum length of the training mel-spectrogram. So if ur training data have some samples that its length > 1000 then the training speed will so slow :)). The maximum length should be <= 850 :D.

[jun-danieloh]

@dathudeptrai
Is it static for all batches? Static shape changes every batch or is it fixed at the beginning of the training?

[jun-danieloh]

@dathudeptrai
I think you originally had the code below in tacotron_dataset.py. Did you take this out?

        datasets = datasets.filter(
            lambda x: x["mel_lengths"] <=850
        )

[dathudeptrai]

@dathudeptrai
I think you originally had the code below in tacotron_dataset.py. Did you take this out?

        datasets = datasets.filter(
            lambda x: x["mel_lengths"] <=850
        )

yes, you should do that :)). 750-850 is good.

[jun-danieloh]

the reason why my tacotron is slower caused by static shape. That mean the decoder steps always equal by maximum length of the training mel-spectrogram. So if ur training data have some samples that its length > 1000 then the training speed will so slow :)). The maximum length should be <= 850 :D.

Is there any way to have it dynamic? Is the max length set at the beginning of the training and does it last for the remaining training iterations without any changes?

[dathudeptrai]

the reason why my tacotron is slower caused by static shape. That mean the decoder steps always equal by maximum length of the training mel-spectrogram. So if ur training data have some samples that its length > 1000 then the training speed will so slow :)). The maximum length should be <= 850 :D.

Is there any way to have it dynamic? Is the max length set at the beginning of the training and does it last for the remaining training iterations without any changes?

the default value is dynamic, see here (https://github.com/TensorSpeech/TensorFlowTTS/blob/master/examples/tacotron2/conf/tacotron2.v1.yaml#L55). But it will cause training 2x slower, static shape is much faster. You can switch it to True and only training the mel that its length <= 850. You should read the readme carefully :))) (https://github.com/TensorSpeech/TensorFlowTTS/tree/master/examples/tacotron2#some-important-notes)

[jun-danieloh]

@dathudeptrai I really appreciate for your detailed explanations :) Let me go through README more carefully.

Can I ask one last question? Does static shape mean for every batch or for an entire training set? What I mean by that is the value of max length changes every batch based on the longest one in a batch or is it set at the beginning of the training?

[dathudeptrai]

s it set at the beginning of the traini

it set at the beginning of the training :)). That mean all batch will padded to maximum length :)))

[jun-danieloh]

@dathudeptrai Really appreciate for your detailed explanations :):)

[nukes]

@nukes I totally agree with you. My tacotron2 in PyTorch shows 0.7~0.8 sec/it which is about 5x times faster with the better quality. Training speed is the biggest hurdle for me to use Tensorflow model.

Some people told me the same result as yours:Pytorch is much faster for training tacotron. I saw your discussion with @dathudeptrai . Basically i think use static shape compromises between training data usage and training speed, which still cannot fundamentally solve this problem. I do not exactly know which part in AR decoding slows down the training. but the while loop should be optimized in my opinion. Maybe someone should profile the training process with tf-profiler and optimize it. a optimal way maybe someone could rewrite the decoding loop into tensorflow kernel/ops in c++ level to decoding all the frames within on op( removing the tf.while).

[jun-danieloh]

@dathudeptrai The refer below is not available. Can you point me to other similar commits or posts?

use_fixed_shapes: false    # use_fixed_shapes for training (2x speed-up)
                           # refer (https://github.com/dathudeptrai/TensorflowTTS/issues/34#issuecomment-642309118)

[dathudeptrai]

@jun-danieloh i can confirm that in my case fixed_shapes = True is 2x faster. But there is a man report that fixed_shapes=False is faster, you can refer this comment (https://github.com/tensorspeech/TensorflowTTS/issues/34#issuecomment-642309118)

[jun-danieloh]

@dathudeptrai I can also confirm that fixed_shapes = True shows 2x faster training speed. Let me check why this helps. Does fixed_shapes shows any quality degradations?

[dathudeptrai]

@dathudeptrai I can also confirm that fixed_shapes = True shows 2x faster training speed. Let me check why this helps. Does fixed_shapes shows any quality degradations?

there are no quality degradations here :D