Tacotron-2: GPU training time?

[PetrochukM]

Hi There!

How long does it take to train Tacotron-2?

Thanks!

[Rayhane-mamah]

Hello again,

As for training, a training step takes 0.75 seconds (with a batch size of 64). It takes around 12 hours to do 60k steps.

It takes about few thousand steps to get a perfect feature prediction model so I expect the training to last for 24~36 hours. I'll let it training longer just to check if model keeps improving.

Actual stats at 60k steps:

As long as loss will be decreasing significantly I'll let it train.

Note:
Again, these time stats are provided using a Titan XP GPU.

[eyaler]

@Rayhane-mamah training ljspeech on v100 (tf 1.12, cuda10, google vm image) with batch=32, i am getting ~5 sec/step. how can i go about debugging this?