Tacotron2: half-precision support on single GPU?

[shaun95]

Thanks for this repo. I have a Pascal-based card (GTX1080) and I switched training to using floating-point-16; to see if it improves training times. I've set fp16_run to True but I get an error.

`
Epoch: 0

Traceback (most recent call last):
File "train.py", line 285, in
args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
File "train.py", line 210, in train
y_pred = model(x)
File "/tacotron2-master/model.py", line 80, in forward
alignment.data.masked_fill_(mask, self.score_mask_value)
RuntimeError: value cannot be converted to type Half without overflow: -inf
`

Since I only have one card I left distributed_run to False.

Running the repo with the default floating-point precision works fine. As others remarked the training times on single GPU is rather slow, on average it takes about 5.5 seconds per iteration with the default single-GPU settings and on LJ-Speech.

[rafaelvalle]

We're looking into the FP16 run error.
Probably most of that performance time is spent on the CPU computing mel-spectrograms and can be amortized by pre-processing the mel-spectrograms and loading them from the disk.

[rafaelvalle]

@shaunmayberry What is your current setup?

[rafaelvalle]

FYI: We've added the option to load mels from disk.

[rafaelvalle]

The FP16 error is a pytorch bug that is currently being fixed. We'll include our own patch soon.

[shaun95]

@rafaelvalle Thanks! I will try the option to load mels from disk after I make the changes needed to produce them to disk. My rig is set up with Pytorch 0.4, using the latest tacotron2 code prior to your updates from today. I am using LJ-Speech and I set fp16_run in hparams to True. I've also reduced the batch size to 24.

[rafaelvalle]

OK. We've added the FP16 patch as well and it should work with your setup.
Let us know if it doesn't.

[shaun95]

@rafaelvalle Thanks for the updates. I tried the update with fp16_run enabled. I am running the training with the mel analysis done on the fly and the batch size set to 24. The training seems to run after skipping the first few iterations. On the first few iterations a message is shown the iteration is skipped for overflow. I paste below the first ~60 lines. I suppose this is OK considering the resolution of fp16?

`

OVERFLOW! Skipping step. Attempted loss scale: 4294967296
OVERFLOW! Skipping step. Attempted loss scale: 2147483648.0
OVERFLOW! Skipping step. Attempted loss scale: 1073741824.0
OVERFLOW! Skipping step. Attempted loss scale: 536870912.0
OVERFLOW! Skipping step. Attempted loss scale: 268435456.0
OVERFLOW! Skipping step. Attempted loss scale: 134217728.0
OVERFLOW! Skipping step. Attempted loss scale: 67108864.0
OVERFLOW! Skipping step. Attempted loss scale: 33554432.0
OVERFLOW! Skipping step. Attempted loss scale: 16777216.0
OVERFLOW! Skipping step. Attempted loss scale: 8388608.0
OVERFLOW! Skipping step. Attempted loss scale: 4194304.0
OVERFLOW! Skipping step. Attempted loss scale: 2097152.0
OVERFLOW! Skipping step. Attempted loss scale: 1048576.0
OVERFLOW! Skipping step. Attempted loss scale: 524288.0
OVERFLOW! Skipping step. Attempted loss scale: 262144.0
Train loss 15 49.825356 Grad Norm 10.964307 5.23s/it
OVERFLOW! Skipping step. Attempted loss scale: 131072.0
Train loss 17 42.958412 Grad Norm 24.459961 5.22s/it
Train loss 18 24.287481 Grad Norm 19.116211 4.87s/it
Train loss 19 13.412683 Grad Norm 17.072796 5.30s/it
Train loss 20 7.944973 Grad Norm 8.451199 4.85s/it
Train loss 21 6.329238 Grad Norm 6.656669 5.53s/it
Train loss 22 5.629976 Grad Norm 5.408318 5.08s/it
Train loss 23 5.232729 Grad Norm 5.742975 5.25s/it
Train loss 24 5.263054 Grad Norm 5.709540 5.20s/it
Train loss 25 4.859667 Grad Norm 4.302123 5.44s/it
Train loss 26 5.378138 Grad Norm 19.306095 4.71s/it
Train loss 27 4.302876 Grad Norm 5.000734 4.85s/it
Train loss 28 4.655708 Grad Norm 3.420239 5.01s/it
Train loss 29 4.100136 Grad Norm 3.279778 4.98s/it
Train loss 30 3.958766 Grad Norm 2.951070 4.90s/it
Train loss 31 4.213562 Grad Norm 9.340719 4.65s/it
Train loss 32 3.921006 Grad Norm 3.553857 5.11s/it
Train loss 33 4.878996 Grad Norm 14.087032 5.18s/it
Train loss 34 4.285587 Grad Norm 9.683391 5.24s/it
Train loss 35 3.880463 Grad Norm 2.702801 4.76s/it
Train loss 36 3.762278 Grad Norm 7.337779 5.09s/it
Train loss 37 3.629325 Grad Norm 3.801917 4.93s/it
Train loss 38 3.828110 Grad Norm 6.207859 5.01s/it
Train loss 39 3.582167 Grad Norm 5.539951 5.12s/it
Train loss 40 3.684500 Grad Norm 2.424948 5.24s/it
Train loss 41 3.895284 Grad Norm 8.835989 5.14s/it
Train loss 42 4.618782 Grad Norm 11.627217 4.55s/it
Train loss 43 3.788434 Grad Norm 3.240130 5.02s/it
Train loss 44 3.820687 Grad Norm 6.515767 5.01s/it
Train loss 45 3.796714 Grad Norm 6.008824 5.05s/it
Train loss 46 3.771316 Grad Norm 4.035465 4.82s/it
Train loss 47 3.584950 Grad Norm 4.370368 4.88s/it
Train loss 48 3.752807 Grad Norm 5.499559 5.12s/it
Train loss 49 3.947842 Grad Norm 2.102493 4.96s/it
Train loss 50 3.786860 Grad Norm 6.217353 4.92s/it
Train loss 51 4.193044 Grad Norm 8.662756 4.73s/it
Train loss 52 3.517472 Grad Norm 1.989968 5.15s/it
Train loss 53 4.172578 Grad Norm 10.428472 4.96s/it
Train loss 54 4.515800 Grad Norm 13.725286 5.09s/it
Train loss 55 3.807115 Grad Norm 6.901080 4.61s/it
Train loss 56 3.750019 Grad Norm 3.148731 4.95s/it
Train loss 57 3.578784 Grad Norm 4.168697 4.97s/it
Train loss 58 3.886656 Grad Norm 4.331501 5.12s/it`

[rafaelvalle]

Yes, that's expected feedback from dynamic loss scaling.

[shaun95]

@rafaelvalle Thanks