Rasa: make sure that rasa train commands run gracefully with e2e

[Ghostvv]

With e2e we linked training rasa.nlu and rasa.core. Therefore, we need to make sure that rasa train, rasa train nlu and rasa train core run perfectly

Potential pitfalls (just smth came to my mind, there might. be more edge cases):

• [ ] rasa train was called, nlu data didn't change but e2e stories did and added e2e examples -> need to retrain nlu model
• [ ] rasa train core was called, but core stories contain e2e examples -> the command should raise an error that rasa train with appropriate nlu pipeline should be called
• [ ] when nlu config changes, rasa train should retrain core as well
• [ ] do not allow both intent: and user: in training data
• [ ] during loading for prediction core should be able to check that loaded nlu model is the same as during training
• [ ] rasa train was called, but no nlu data are present. However, your stories contain e2e examples. -> We need to train an nlu model.

[wochinge]

@Ghostvv Somebody mentioned that training on e2e is currently broken - does that block implementing a solution or can we move forward with it?

[Ghostvv]

the training on e2e should work now

[joejuzl]

Is the e2ebot example bot meant to be in a working state?
I get this error when using it:

2020-11-19 14:44:03 DEBUG    rasa.core.processor  - Action 'action_session_start' ended with events '[<rasa.shared.core.events.SessionStarted object at 0x153e4b450>, <rasa.shared.core.events.ActionExecuted object at 0x153d87250>]'.
2020-11-19 14:44:04 DEBUG    rasa.core.processor  - Received user message 'hello' with intent '{'id': -4740810205347421666, 'name': 'greet', 'confidence': 0.9999948740005493}' and entities '[]'
2020-11-19 14:44:04 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2020-11-19 14:44:04 DEBUG    rasa.core.policies.ensemble  - Made e2e prediction using user text.
2020-11-19 14:44:04 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(True)` event.
2020-11-19 14:44:04 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_TEDPolicy.
2020-11-19 14:44:04 DEBUG    rasa.core.processor  - Predicted next action 'Welcome to moodbot. How are you feeling today?' with confidence 1.00.
2020-11-19 14:44:04 ERROR    rasa.core.processor  - Encountered an exception while running action 'Welcome to moodbot. How are you feeling today?'.Bot will continue, but the actions events are lost. Please check the logs of your action server for more information.
Traceback (most recent call last):
  File "/Users/josephjuzl/code/rasa/rasa/core/processor.py", line 750, in _run_action
    output_channel, nlg, temporary_tracker, self.domain
  File "/Users/josephjuzl/code/rasa/rasa/core/actions/action.py", line 599, in run
    f"Failed to execute custom action '{self.name()}' "
rasa.shared.exceptions.RasaException: Failed to execute custom action 'Welcome to moodbot. How are you feeling today?' because no endpoint is configured to run this custom action. Please take a look at the docs and set an endpoint configuration via the --endpoints flag. https://rasa.com/docs/rasa/custom-actions
2020-11-19 14:44:04 DEBUG    rasa.core.processor  - Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x153f0c510>]'.
2020-11-19 14:44:04 DEBUG    rasa.core.processor  - Action 'Welcome to moodbot. How are you feeling today?' ended with events '[]'.
2020-11-19 14:44:04 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_TEDPolicy.
2020-11-19 14:44:04 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 0.82.
2020-11-19 14:44:04 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2020-11-19 14:44:04 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2020-11-19 14:44:04 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'test_user4'.

Do bot responses in e2e stories have to still be defined in the domain?

[Ghostvv]

predicting e2e bot utterances in rasa shell doesn't work yet. There is a separate issue for it: https://github.com/RasaHQ/rasa/issues/6410

[joejuzl]

This was from the rest api - but I guess it has the same problem.

[joejuzl]

Ok so the scope of this issue is purely to check for bugs in the training process?

[Ghostvv]

the scope of this issue to check that rasa train core fails gracefully and that rasa train correctly retrains various models depending on the changed data (nlu data now influence core model and core data influence nlu model)

[Ghostvv]

also, during loading for prediction core should be able to check that loaded nlu model is the same as during training

[joejuzl]

Findings so far

1. > rasa train was called, nlu data didn't change but e2e stories did and added e2e examples -> need to retrain nlu model

Seems to work.

1. > rasa train core was called, but core stories contain e2e examples -> the command should raise an error that rasa train with appropriate nlu pipeline should be called

Throws this error:

> rasa train core                                                                                   (/Users/josephjuzl/.kube/config doesn't exist)
Training Core model...
2020-11-20 12:46:44 INFO     rasa.core.policies.ensemble  - MappingPolicy not included in policy ensemble. Default intents 'restart and back will not trigger actions 'action_restart' and 'action_back'.
Processed story blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 2392.64it/s, # trackers=1]
Processed story blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1111.96it/s, # trackers=3]
Processed story blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 307.63it/s, # trackers=12]
Processed story blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 74.00it/s, # trackers=39]
Processed trackers: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 120/120 [00:00<00:00, 255.81it/s, # actions=601]
Traceback (most recent call last):
  File "/Users/josephjuzl/.pyenv/versions/rasa-py3.7/bin/rasa", line 5, in <module>
    main()
  File "/Users/josephjuzl/code/rasa/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/josephjuzl/code/rasa/rasa/cli/train.py", line 124, in train_core
    additional_arguments=additional_arguments,
  File "/Users/josephjuzl/code/rasa/rasa/train.py", line 317, in train_core
    additional_arguments=additional_arguments,
  File "/Users/josephjuzl/code/rasa/rasa/utils/common.py", line 308, in run_in_loop
    result = loop.run_until_complete(f)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/Users/josephjuzl/code/rasa/rasa/train.py", line 373, in train_core_async
    additional_arguments=additional_arguments,
  File "/Users/josephjuzl/code/rasa/rasa/train.py", line 409, in _train_core_with_validated_data
    interpreter=interpreter,
  File "/Users/josephjuzl/code/rasa/rasa/core/train.py", line 67, in train
    agent.train(training_data, **additional_arguments)
  File "/Users/josephjuzl/code/rasa/rasa/core/agent.py", line 721, in train
    training_trackers, self.domain, interpreter=self.interpreter, **kwargs
  File "/Users/josephjuzl/code/rasa/rasa/core/policies/ensemble.py", line 194, in train
    trackers_to_train, domain, interpreter=interpreter, **kwargs
  File "/Users/josephjuzl/code/rasa/rasa/core/policies/ted_policy.py", line 461, in train
    training_trackers, domain, interpreter, **kwargs
  File "/Users/josephjuzl/code/rasa/rasa/core/policies/policy.py", line 172, in featurize_for_training
    training_trackers, domain, interpreter
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/tracker_featurizers.py", line 192, in featurize_trackers
    entity_tags = self._create_entity_tags(trackers_as_entities, interpreter)
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/tracker_featurizers.py", line 104, in _create_entity_tags
    for trackers_entities in trackers_as_entities
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/tracker_featurizers.py", line 104, in <listcomp>
    for trackers_entities in trackers_as_entities
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/tracker_featurizers.py", line 102, in <listcomp>
    for entity_data in trackers_entities
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/single_state_featurizer.py", line 274, in encode_entities
    for token in parsed_text.get(TOKENS_NAMES[TEXT]):
AttributeError: 'NoneType' object has no attribute 'get'

1. > when nlu config changes, rasa train should retrain core as well

Does not retrain core:

Core stories/configuration did not change. No need to retrain Core model.

1. > do not allow both intent: and user: in training data

Currently does allow it

1. > rasa train was called, but no nlu data are present. However, your stories contain e2e examples. -> We need to train an nlu model.

Throws this error:

Training Core model...
2020-11-20 14:34:15 INFO     rasa.core.policies.ensemble  - MappingPolicy not included in policy ensemble. Default intents 'restart and back will not trigger actions 'action_restart' and 'action_back'.
Processed story blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1837.99it/s, # trackers=1]
Processed story blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2242.94it/s, # trackers=1]
Processed story blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1276.80it/s, # trackers=2]
Processed story blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 854.41it/s, # trackers=3]
Processed trackers: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 393.59it/s, # actions=25]
Traceback (most recent call last):
  File "/Users/josephjuzl/.pyenv/versions/rasa-py3.7/bin/rasa", line 5, in <module>
    main()
  File "/Users/josephjuzl/code/rasa/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/josephjuzl/code/rasa/rasa/cli/train.py", line 90, in train
    nlu_additional_arguments=extract_nlu_additional_arguments(args),
  File "/Users/josephjuzl/code/rasa/rasa/train.py", line 55, in train
    loop,
  File "/Users/josephjuzl/code/rasa/rasa/utils/common.py", line 308, in run_in_loop
    result = loop.run_until_complete(f)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/Users/josephjuzl/code/rasa/rasa/train.py", line 110, in train_async
    nlu_additional_arguments=nlu_additional_arguments,
  File "/Users/josephjuzl/code/rasa/rasa/train.py", line 183, in _train_async_internal
    additional_arguments=core_additional_arguments,
  File "/Users/josephjuzl/code/rasa/rasa/train.py", line 409, in _train_core_with_validated_data
    interpreter=interpreter,
  File "/Users/josephjuzl/code/rasa/rasa/core/train.py", line 67, in train
    agent.train(training_data, **additional_arguments)
  File "/Users/josephjuzl/code/rasa/rasa/core/agent.py", line 721, in train
    training_trackers, self.domain, interpreter=self.interpreter, **kwargs
  File "/Users/josephjuzl/code/rasa/rasa/core/policies/ensemble.py", line 194, in train
    trackers_to_train, domain, interpreter=interpreter, **kwargs
  File "/Users/josephjuzl/code/rasa/rasa/core/policies/ted_policy.py", line 461, in train
    training_trackers, domain, interpreter, **kwargs
  File "/Users/josephjuzl/code/rasa/rasa/core/policies/policy.py", line 172, in featurize_for_training
    training_trackers, domain, interpreter
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/tracker_featurizers.py", line 192, in featurize_trackers
    entity_tags = self._create_entity_tags(trackers_as_entities, interpreter)
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/tracker_featurizers.py", line 104, in _create_entity_tags
    for trackers_entities in trackers_as_entities
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/tracker_featurizers.py", line 104, in <listcomp>
    for trackers_entities in trackers_as_entities
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/tracker_featurizers.py", line 102, in <listcomp>
    for entity_data in trackers_entities
  File "/Users/josephjuzl/code/rasa/rasa/core/featurizers/single_state_featurizer.py", line 274, in encode_entities
    for token in parsed_text.get(TOKENS_NAMES[TEXT]):
AttributeError: 'NoneType' object has no attribute 'get'

1. > during loading for prediction core should be able to check that loaded nlu model is the same as during training

I'm not sure what this means exactly. Is this a scenario when the nlu model is manually replaced? Or via a training command?

[Ghostvv]

in point 2. it should fail gracefully, by giving a user helpful error message
in point 6.

Is this a scenario when the nlu model is manually replaced?

yes

Or via a training command?

if we fix training commands, theoretically, it shouldn't happen

[wochinge]

@Ghostvv

do not allow both intent: and user: in training data

For clarification: You mean that one step can't contain both, user and intent key, don't you? Because mixing e2e stories with labeled stories should be fine.

rasa train was called, but no nlu data are present. However, your stories contain e2e examples. -> We need to train an nlu model.

I don't see the problem here. E2EImporter will automatically add NLU examples from the stories to the NLU training data which means we will automatically end up with the necessary NLU data (I'd argue that it's probably too less data to train a proper model, but I guess that's up to the user).

[joejuzl]

I don't see the problem here. E2EImporter will automatically add NLU examples from the stories to the NLU training data which means we will automatically end up with the necessary NLU data (I'd argue that it's probably too less data to train a proper model, but I guess that's up to the user).

This currently doesn't work, it throws the error in my reply above.

[wochinge]

Mhm, interesting that it fails for the entity encoding 🤔

[Ghostvv]

I'd argue that it's probably too less data to train a proper model, but I guess that's up to the user

in this case you wouldn't train DIET, but we still need to call Rasa NLU training to train all the featurizers

[Ghostvv]

@joejuzl can we close this issue?