Tensorflowtts: NaNs in FS2 posnet

[machineko]

So i've trained Fastespeech2 on my dataset using phones based on MFA extraction (pull request incoming later tomorrow).
But I meet weird NaNs in eval postnet after ~50k steps postnet is only producing nans.
I'm not an expert using tensorflow (last 2.5 years i've use mainly pytorch) so I have question if it is possible that it's caused by fp16 training [later i'll train fp32 model to confirm this but it needs a lot more time] and how to do training more stable on fp16 (maybe some tricks how to debug nans in tensorflow)?

[dathudeptrai]

@machineko mixed precision is still on experimental module of tensorflow so it is not stable yet. In my case it ălways fine in my private dataset and ljspeech. If you get nan, you can training float32, the training speed is still very fast even it is float32. You just need < 1 day to train fastspeech. But if you still want to use float16, you can try this trick: resume training float32 at the step you get nan for around 5k steps. Then resume again with float16 :))

[tekinek]

Same problem #125 with tacotron2 training at 53.5k steps (1080ti, tensorflow 2.2). Restart training from 50k solved my problem so far.

[machineko]

Sure I'll try both of ur solutions later and post updates here :)

Also, its multi-speaker model.

[machineko]

It's looks like fp16 stability turning off and retraining on 32precision didn't have any nans problem. (probably we can fix it excluding some layers from fp16 training and i'll test it in next week)