Rasa: Performance Evaluation of a Trained Model?

[mit4dev]

rasa NLU version: 0.9.0a2

Used backend / pipeline: spacy_sklearn

Operating system: Ubuntu 16.04 x64

Issue: How can I measure the overall performance of the _rasa NLU_ on a trained model e.g. accuracy, precision, recall metrics. I've already inspected the documentation and made some research online but had no luck... Am I missing something?

Thanks in advance..

[PHLF]

If you want something that RASA gives you automatically, it's not possible at this time AFAIK: you need to have a specific tagged dataset (like your training dataset) for evaluating the classification performance of your model. Feed your trained model with this new dataset; count how many intents / entities where well identified and extract some kind of performance metric from that.
You don't have to create a new dataset for this task: you could also split your training data in two sets, use one part for training your model and the other to evaluate its accuracy.
This could be added as an option in RASA (automatic splitting of the training data for cross validation), but I am pretty sure that you can already get some information from the intents / entities model training (I didn't looked that much into that process by the way).

[mit4dev]

I've almost had the same idea of you measuring the performance manually. I just wanted to make sure if there is any official solution for this purpose. Thanks for the quick and clarifying reply.

[tmbo]

@mit4dev we actually want to have this integrated, but it just didn't happen yet. #248

[karnigili]

is there any way to get stats on the test data set (implemented in the lib?)