Rasa: make it easy to track e2e test performance over time

[TyDunn]

Description of Problem:

It is challenging to track how end-to-end tests are performing over time

Overview of the Solution:

"The ability to generate junit.xml files for some of the nlu / e2e story tests. The nlu test generates a json file which can be consumed by our pipeline, but the results of the e2e tests only show the errors in the failed_stories.md file. I’m not tied to junit.xml files, json generated results are fine as well, but would be a good way for us to plot and keep track of how our tests are performing. Happy to chat further about this or put a ticket into the github repo, just thought I’d run it past you first to see if you know if this is something that is in the pipeline"

This is from a user. I am going to chat with them further about this to better understand their workflow

[wochinge]

@RasaHQ/production-squad can we deprio this?

[TyDunn]

@wochinge I want to talk to the user more first. I think I would make case that this would help enterprises use end-to-end tests more

[wochinge]

@TyDunn ok, just checked. It would be just writing the return dictionary of https://github.com/RasaHQ/rasa/blob/26f78e845507d2e3a6a83ff75edd28687d3e8510/rasa/core/test.py#L557 to disk. Let's do it.

[rgstephens]

NLU test json output:

DIETClassifier_errors.json - entity classification errors
DIETClassifier_report.json - entity extraction f1, precision, recall summary
intent_errors.json - intent error details
intent_report.json - intent precision
RegexEntityExtractor_errors.json - errors
RegexEntityExtractor_report.json - details

Story json output:

story_report.json - precision