Tensorflowtts: training speed

[jun-danieloh]

Hi, I am using ljspeech dataset, same default setting as repository, no mixed precision and using 1080Ti GPU.

My training is still running since last week. Is it usual to have this slow training speed? Inference pre-trained model is based on 120k but mine is still at ~90k and the inference quality is not as good as one from 120k so I still need to wait.

[train]:  45%|###################5                        | 89081/200000 [170:05:17<219:35:43,  7.13s/it]2020-09-22 08:02:23,395 (base_trainer:138) INFO: (Steps: 89081) Finished 229 epoch training (389 steps per epoch).

[dathudeptrai]

@jun-danieloh can you add bellow code after (https://github.com/TensorSpeech/TensorFlowTTS/blob/master/examples/tacotron2/tacotron_dataset.py#L194-L196):

        datasets = datasets.filter(
            lambda x: x["mel_lengths"] <=850
        )

[jun-danieloh]

@dathudeptrai Let me try this and get back soon.

[jun-danieloh]

@dathudeptrai When max_mel_length set 850, it shows a speed improvement from 7.13/it to 6.15/it with 1080Ti GPU.

[kb-rahul]

@dathudeptrai : I am training on dual V100 gpu. I have set max mel length to 850 as recommended above. I still see 12-14/it. Can you think of reason why? or what I could do to speed up the process?

[jun-danieloh]

@dathudeptrai @kb-rahul
I git pulled the latest code and the latest setting but it is still extremely slow compared to PyTorch model. It is about x5 times slower than PyTorch.

Is it Tensorflow vs PyTorch issue? Or a model implementation issue?

[jun-danieloh]

@dathudeptrai
Also, my training speed doesn't get improved when I moved from 1080Ti to V100. It is almost identical. Is it a known issue?

Why does TensorFlow use the full memory of GPU when it is unnecessary? Even if with the smaller number of batch size, TensorFlow tries to utilize the full memory of GPU always.