Rasa: How to speed up mitie training process

[xliuhw]

Hi, I am using mitie backend to train my models for intents and entities. the intents and entities recognition accuracy is better than other backends from my tries. One problem for me is that mitie takes a long time.

I am wondering if anyone has any suggestions on making use of GPUs in RASA?

I tried num_threads:16 on the stable version (0.7.1) on my 24 Cores desktop, it is the same as in one thread. Does RASA support multiple threads for mitie? or Do I need to run on the latest version?

Thanks!
Xingkun

[amn41]

Thanks for creating this issue, this does sound concerning. Could you please try a benchmark using only MITIE's python bindings? See https://github.com/mit-nlp/MITIE/blob/master/examples/python/train_text_categorizer.py for an example.

That will let us isolate if rasa is preventing a speedup or if it's on MITIE's end.

[xliuhw]

Good suggestion! I will have a try and then let you know. Thanks!

[xliuhw]

Finally I got the results calling directly from mitie train_text_categorizer.py for the intents recognition.
It still took a long time but it's nearly 1/3 quicker than calling from RASA as it took 64.1 hours while RASA took 93.5 hours on exact same data.

I suspect there were issues with my data annotations as I notice there are something in the data like

"play x" --> labelled intent: music_play
"play y" --> labelled intent: radio_play
"play z" --> labelled intent: podcast_play

which may have confused the trainer.
I will tidy up my data and try again.

[amn41]

Awesome! I think it's worth testing with a much smaller dataset so we can speed up the debug loop.

[xliuhw]

I am testing with a much smaller dataset and will let you know and send you the test files after it finishes.

[xliuhw]

I've done following tests.

Test1:
Training to recognize 10 categories
num training samples: 1008

RASA: Training time: 49904 seconds. == 13.86 hours
DirectMitieCall: Training time: 36602 seconds. = 10.17 hours

Test2:
Training to recognize 5 categories
num training samples: 324

RASA: Training time: 4554 seconds. = 75.9 minutes
DirectMitieCall: Training time: 5134 seconds. = 85.57 minutes

Test3:
Training to recognize 3 categories
num training samples: 236

RASA: Training time: 2225 seconds. = 37.08 minutes
DirectMitieCall: Training time: 1885 seconds. = 31.46 minutes

Confused thing is we can say that RASA is lower than DirectMitieCall in Test1 and Test3,
but quicker in Test2.

Which test data would you like to have a try? All these test data?

Cheers!

[tmbo]

We have added a new entity recognizer (CRF) which has a similar performance as the MITIE model but trains quite a bit faster. Since, we can not do anything about MITIEs performance (this should be raised on the MITIE project), I'll close this issue.

[gagan-bansal]

Please read detailed explanation about MITIE working. That may solve the long duration training problem.