Tacotron2: How are multiple GPUs utilized in distributed run?

[shreyesh-ziro]

Before, I had tried training on single GPU with batch size of 8 (NVIDIA GeForce RTX 2070)

Now, I'm training an English dataset having around 20k sentences. using 4 GPUs, each 16DB. (Tesla T4).
I tried batch size of 64, however i get an OOM error of CUDA, but worked well with batch size 32.

Wanted to know why it doesn't work for batch size 64, considering I have 4 GPU (16GB x 4 = 64GB of GPU in total)?

In distributed run, are the 4GPU running in parallel with batch size 32? (equivalent to 32x4 = 128)
Or are they collectively processing a batch size of 32?

[CookiePPP]

batch_size is per GPU.
64 batch_size using 4 GPU's = 256 audio files per iteration.

[CookiePPP]

I tried batch size of 64, however i get an OOM error of CUDA, but worked well with batch size 32.

have you tried enabling fp16_run?
Running in half-precision might let you increase the batch size to 64 without any other code modifications.

[shreyesh-ziro]

@CookiePPP
I did try enabling f16_run. However, I faced some issue with apex, hence disabled it again and only using distributed_run=True.
I'll give f_16 a try again and update here if any issues or if it works for batch_size=64