TTS issue regarding Ubuntu version & Problem with memory consumption

[haqkiemdaim]

Hi @erogol, i got few question to ask and i hope it is relevant to be asked here 😊

I can see there is some different when running TTS training on ubuntu16 compared to ubuntu18.04(latest). Which is ubuntu18.04 run smoothly without any internal error such as librosa error, but when it comes to ubuntu16 it does came up with the error.

Is the version of ubuntu affect the training process?

From what i observe, the training require a lot of memory since it run together with the preprocessing at the same time. (correct me if im wrong)

So is it possible to do all the pre-process before execute the training process? Much more better isn't it ?

[erogol]

Maybe it is just librosa version differences or some backend library differences between ubuntu versions. But there is nothing specific with your OS version. If you feel like preprocessing is useful, you need to implement one. Unfortunately, we dropped the support for preprocessing just recently but you can find some example codes in some other branches.

[haqkiemdaim]

@erogol so you mean that your current latest code does not have preprocessing? Why is that? Is it not necessary to have some preprocessing on audio data?

One more thing, i would like to run the training on my laptop with below specification:

*12GB RAM
*Nvidia GForce 940Mx
*Intel core i5 7th Gen
*SSD

If i run the training for 1000 epoch using above specification, does it will affect my hardware performance?

Since i already tried for a few epochs with time estimation 15 min+ per epoch.

What can you say about this ya ?

[erogol]

no need preprocesses + 15 min is okay + ya

[haqkiemdaim]

Alright @erogol this is one more important question i want to ask :

For now i face one more problem regarding
_CUDA out of Memory.. try to Allocate.._

I tried:
✔ followed the range of batch size 8-12 with 4GB GPU
✔ reduce the batch size as lower as 3, still got error after 4 hours of training with 9
epoch

I know this error occur due to not enough of existing memory (4GB).
What other solution that I can try to execute the training successfully ya ? Since i really need to know the output of my own 5 Hours dataset. This is very important and i really need your help 🙏

[erogol]

if you have very long sentences that might cause OOM error. Try to use shorter sentences. 4GB is really small for this project. Try to find something better.

[haqkiemdaim]

@erogol one more question if you dont mind:

I ran the tarining of 1000 epoch with 3×8 bs but only manage to reach 9th epoch due to CUDA out of memory.

But then, i tried on the model .pth.tar generated by the 9th epoch in the web server and it appears that below erorr arise:

Decoder stopped with 'max_decoder_step'

What would be the error caused by ya?