Tacotron2: Getting killed message depending on text size

[shoegazerstella]

I am using the new model provided a few days ago. It is very fast at loading and it is twice as fast as the old one during inference.
My problem is that I get a killed message with a string whose length is 1233.
I am doing all this on cpu so I modified a bit the code for this. Is there a way I can avoid the killed message maybe changing some audio parameter?
Thanks a lot

[simonefrancia]

I am facing the same problem and I made some tests for this, hoping that this info could help to solve the issue:
I took a fixed sentence ( length 35 characters in my case) and at every iteration of my test, I double the length of input appending the sentence used in the previous step. For example (case of length 6 for "hello\n" ) :

Step 1 ---> input 1 : "hello\n" Step 2 ---> input 2 : "hello\nhello\n" Step 3 ---> input 3 : "hello\nhello\nhello\nhello\n" .... Step N ---> input N : input (N-1) * 2.

Instead of using sentence "hello\n", I used a sentence of length 35.
Using this method with some fine-tuning, I found that giving as input 27 times the sentence of length 35 (so 27x35 = 945 chars) i got a killed, while with 26x35 it's ok.

I am doing this tests in CPU with 8-core i7 and 8GB RAM inside a Docker container.

[loretoparisi]

Any update on this memory issue?

[rafaelvalle]

Is this during inference? The system is probably running out of memory?

[shoegazerstella]

hi @rafaelvalle,
Yes, this happens during inference. I suspect it is a memory issue.
Would it be possible to fix it by maybe changing some audio parameters? I see some connections with this issue. Thanks

[rafaelvalle]

This seems to be a regular OOM error. You can solve it by dividing your text into chunks at end of sentences, for example.

[loretoparisi]

@rafaelvalle just curious, do you have any statistics about text length and memory pressure? It could be useful! Thank you.

[rafaelvalle]

Not from memory. You can get that number by increasing the length of the input until the system runs out of memory.

[loretoparisi]

@rafaelvalle yes it was what we did in https://github.com/NVIDIA/tacotron2/issues/192#issuecomment-487007847
The constraint were 8GB RAM, inference on CPU only, having a OOM with 27 sentences of 35 char length, so 27 x 35 x 8 ~ 7KB of text, assuming no Unicode double byte text.