Tts: use APEX for Mixed-Precision training.

[erogol]

In order to enable mixed precision training, meld APEX library to TTS. It needs to be easy enough but also it is suggested to use loss scaling to compensate float16 training.

I'd be a nice leap forward to train larger TTS versions like when r=1 or Tacotron2 architecture.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discourse page for further help. https://discourse.mozilla.org/c/tts

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discourse page for further help. https://discourse.mozilla.org/c/tts

[repodiac]

Hi - this issue seems to have been closed "by automation"... are there still any plans on mixed precision with the TTS models (both taco1 and 2)? This would really help with "commodity GPU hardware" for training...

[erogol]

I don't have any plans but I can help for any contributions.

[repodiac]

OK thanks, any help is much appreciated. I can work on this (for a later PR). So, @el-tocino mentioned that you could have your reasons why you or anybody else did not pursue this path - are there any in terms of efforts/yield threshold?

Otherwise, I found this documentation from NVIDIA with detailed descriptions and examples for e.g. PyTorch: https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#amp

[erogol]

Personally I don't have time unfortunately. But I think it is useful for low-resource training. I don't think it is to much coding work but mixed-precision training requires some additional algorithmic tricks like loss scaling to make it work and do some trainings until you see it works as good as the regular training.

[repodiac]

hi again - first of all: can you reopen the issue while we are working on this? Thx.

Secondly, as briefly noted in the PR #414 currenlty only O1 works (both checked with taco1 and taco2 architecture). For O2, I get a bunch of varying errors where half or float (alternating) are expected by certain layers in PreNet or later.

I tried to manually set all the input data to half precision but this only shifted the errors to another part. I unfortunately do not have the means/insights yet to debug and find out why and where it fails. The error message is not "intuitive" so to speak... the NVIDIA taco2 implementation uses O2 optimization throughout the whole architecture and has not made any other code changes (one exception: the initialization of the attention score mask is set to minimum of FP16, I found this in your code as well, changed it, but didn't help!)

Could you maybe check back and try to find the spots in the code where it oviously is not flexible enough to switch or some other stuff is hardcoded?

Motivation: The speed and memory advantages are impressive, I think

• taco1 and taco2 use approx. only 2/3 of memory with O1 opt.
• taco2 seems to have twice the throughput: the current architecture seems to be like that sped is not increased (i.e. training step time decreases), but the GPU has only half the usage, i.e. without O1 it took up to 50% of the resources whereas with O1 it never crossed 30% (rather less). With taco1 I couldn't see any significant changes (?)
  ** really nice, since less throughput also lowers energy consumption, in my case from ~175W to ~70W tops!

[erogol]

If you can send a PR for O2 updates, I can try to check what is failing.

[repodiac]

ok, fine. will do - but wait, probably makes sense to first merge this PR #414 on O1 optimization!? Codewise, O2 is actually the same, just set config setting to "O2"

[repodiac]

dev branches were too much out of sync (PR #414 was abandoned) - I just issued a new PR instead: #444

[erogol]

It think we finished that