Rasa: `rasa test` to include action server

[ArjaanBuijk]

Description of Problem:
How do I test that my bot is working?

Currently, rasa test :

• Fully tests NLU, including external entity extractors like duckling
• Does not fully test e2e stories, because it does not actually call the action server.

This means that even when rasa test shows that all the e2e stories pass, it is not a guarantee that the bot is functional. One has to test all the e2e stories manually, which is a lot of work and it slows down the CDD iterations.

Doing tests with the action server included is especialy important during the CDD Fix step, or when upgrading a bot from Rasa OSS 1 to Rasa OSS 2.

A side benefit of including the action server during rasa test is that it would provide a nice method for debugging custom actions and it provides a declaritive mechanism to create full coverage tests of the custom actions. (https://github.com/RasaHQ/rasa/issues/4212#issuecomment-698888514)

Overview of the Solution:
rasa test calls the action server.

Blockers (if relevant):
It is not a blocker, because it is standard testing practice. But, the user must be careful that the action server is running in some kind of local mode or testing mode. Eg. during testing it is not actually updating a production data-base or calls external services.

Slack thread

[TyDunn]

Anything you would add @akelad? We have discussed this in the past

[TyDunn]

Context behind past decision to not execute actions in test stories: https://forum.rasa.com/t/snapshot-based-testing-with-rasa/13318

[akelad]

not much to add to this, but we have had customers bring this up a few times in the past year.

Having a mock action server for testing I believe is common in the enterprise anyways -otherwise you would be modifying "real world things" with your manual tests as well. So I feel like allowing rasa test to include the action server again could make sense

[TyDunn]

I really think we should make running the actions a possibility, especially given another data point that Ella added to the Slack thread yesterday. I'm adding this to the Rasa Open Source 2.1 milestone to push this discussion more