Rasa: Mapping Policy Ignores NLU Thresholds--And is there any way to prioritize one policy over another?

[lgrinberg]

Rasa version: 0.14.4

Python version: 3.6

Operating system (windows, osx, ...): Redhat

Issue:
I have NLU Threshold configured with default 0.3 value. “greet” intent is mapped to “action_greet”. When I get incoming user intent classified as “greet” with 0.24 confidence, I would expect the classification to fail and use the fallback policy. Instead the mapping policy is invoked. So you get the following:

2019-05-21 11:30:25 DEBUG rasa_core.processor - Received user message ‘attending training tomorrow’ with intent ‘{‘name’: ‘greet’, ‘confidence’: 0.24704762503877342}’ and entities ‘[{‘start’: 19, ‘end’: 27, ‘text’: ‘tomorrow’, ‘value’: ‘2019-05-22T00:00:00.000-07:00’, ‘confidence’: 1.0, ‘additional_info’: {‘values’: [{‘value’: ‘2019-05-22T00:00:00.000-07:00’, ‘grain’: ‘day’, ‘type’: ‘value’}], ‘value’: ‘2019-05-22T00:00:00.000-07:00’, ‘grain’: ‘day’, ‘type’: ‘value’}, ‘entity’: ‘time’, ‘extractor’: ‘DucklingHTTPExtractor’}]’

2019-05-21 11:30:25 DEBUG rasa_core.policies.fallback - NLU confidence 0.24704762503877342 is lower than NLU threshold 0.3.

2019-05-21 11:30:25 DEBUG rasa_core.policies.memoization - There is a memorised next action ‘35’ 2019-05-21 11:30:25 DEBUG rasa_core.policies.form_policy - There is an active form ‘student_loan_form’ 2019-05-21 11:30:25 DEBUG rasa_core.policies.ensemble - Predicted next action using policy_4_MappingPolicy

Is there any way for the MappingPolicy to only work when the NLU confidence is greater than threshold?

Content of configuration file (config.yml):

  - name: KerasPolicy
    epochs: 270
    max_history: 3
  - name: FallbackPolicy
    fallback_action_name: 'action_custom_fallback'
  - name: MemoizationPolicy
    max_history: 5
  - name: FormPolicy
  - name: MappingPolicy

Content of domain file (domain.yml) (if used & relevant):

intents:
  # add your intents
  - wait: {triggers: action_waiting}
  - ready: {triggers: action_ready}
  - initialize
  - greet: {triggers: action_greet}
  - affirm
  - inform_none
  - deny
  - repeat: {triggers: action_repeat}
  - more_info: {triggers: action_utter_verify_info}
  - not_sure: {triggers: action_not_sure}
  - inform_loan
  - inform_loan_exceeds
  - inform_loan_under
  - automated_message_hold
  - automated_message_terminate
  - out_of_scope: {triggers: action_utter_back_on_track}
  - ambiguous_affirm_deny: {triggers: action_utter_ambiguous_response}
  - ask_is_machine: {triggers: utter_i_am_machine}
  - want_agent: {triggers: utter_will_get_agent}

[akelad]

@lgrinberg yeah the fallback policy should definitely always override everything else. we'll look into it -- @erohmensing can you?
As for prioritizing one over the other, you can add a priority parameter to your policy config

[erohmensing]

Yeah @akelad i'm thinking fallback should get higher priority than form and mapping policies, which it currently doesn't. if it's under the NLU fallback then it will predict fallback with confidence 1, so it'd override mapping/form, whereas if it's over the nlu fallback, then it will predict with core fallback <1, so it won't override mapping/form, which sounds like ideal behaviour to me .

@lgrinberg I can fix this, but in the meantime as @akelad said, just add - priority: 6 to the FallbackPolicy in your configuration and it should work as desired (your feedback for if this is ideal would be great too).

[lgrinberg]

Yes, one of our developers found the "priority" parameter by looking through the code (it's undocumented).

The defaults are:
Mapping: 5
Forms: 4
Fallback: 3
Memoization: 2
Keras: 1

We are going to change and test with this:

policies:

• name: FormPolicy
  priority: 7
• name: FallbackPolicy
  fallback_action_name: 'action_custom_fallback'
  priority: 6
• name: MappingPolicy
  priority: 3
• name: MemoizationPolicy
  max_history: 5
  priority: 2
• name: KerasPolicy
  epochs: 270
  max_history: 3
  priority: 1

[tahamr83]

There's is another bug. The priority for Mapping Policy doesn't change if even you try to change it. If you set

policies:

• name: FormPolicy
  priority: 5
• name: FallbackPolicy
  fallback_action_name: 'action_custom_fallback'
  priority: 4
• name: MappingPolicy
  priority: 3
• name: MemoizationPolicy
  max_history: 5
  priority: 2
• name: KerasPolicy
  epochs: 270
  max_history: 3
  priority: 1

During training if you try to debug and print the priorities it's fine. But when you start the core server you will get a same priority warning. Which shows that Mapping Policy and Form policy have the same priority i.e 5. It remains constant at 5. This is why we had to bump up the priorities by 1 to get the desired result.

[erohmensing]

@lgrinberg, great, let me know how those policy priorities go for you. You're right that they're undocumented, this is because we want to encourage the default prioirities (but to do this we have to get them right!). We're still trying to decide what should get priority in general.

@tahamr83 thank you for catching this bug! Seems to happen here https://github.com/RasaHQ/rasa/blob/master/rasa/core/policies/mapping_policy.py#L97-L106 -- the MappingPolicy doesn't persist and load the priority, a result of the priority and mapping policy PRs being worked on around the same time. Will add to the fix.

[tahamr83]

Ah. I get it. I did go to model directory to find no folder for mapping policy and suspected that could be the bug. Nice