Rasa: Rules without conditions are ignored when featurized slot is set in the conversation

[nbeuchat]

Rasa version: 2.0.5

Rasa SDK version: 2.0.0

Python version: 3.7.9

Operating system: Ubuntu

Issue:
When a featurized slot is set within a conversation and some rules have a condition to check for this slot (in this case, the boolean slot preferences_filled), rules that do not have a condition to check for this slot are not executed.

To illustrate this, I have a simple rule which resets the conversation states. When called at the beginning of the conversation, it works fine. I then have an intent that triggers a form and a couple of follow-up actions if the slot preferences_filled is false. The important part is that after submitting the form, the slot preferences_filled is set to true.

The problem is that when I try to use the simple rule for resetting the bot, it does not work anymore, The RulePolicy fails to quick-in.

I do not understand what I am expected to set in the condition part of each rules. Why would the Reset rule be affected by the slot preferences_filled?

I have added the rules I use below as well as a test to show the issue.

rules.yml

version: "2.0"
rules:
- rule: Reset (expected to behave as the old triggers
  steps:
    - intent: reset
    - action: action_reset_full

- rule: Activate preferences form
  condition:
  - slot_was_set:
    - preferences_filled: false
  steps:
  - intent: ask_search_apartments
  - action: action_introduce_form_preferences
  - action: form_preferences
  - active_loop: form_preferences

- rule: Submit preferences form
  condition:
    - active_loop: form_preferences
  steps:
    # Form is deactivated
    - action: form_preferences
    - slot_was_set:
        - requested_slot: null
    - active_loop: null
    - action: action_submit_form_preferences
    - slot_was_set:
        - preferences_filled: true  # <-- THAT ONE IS CAUSING ISSUE DOWN THE LINE
    - action: action_wait
    - action: action_get_apts_search
  wait_for_user_input: false

- rule: Search with results
  condition:
  - slot_was_set:
    - preferences_filled: true  # WITH THIS IT WORKS, IT DOESN'T OTHERWISE
  steps:
  - action: action_get_apts_search
  - action: action_display_apt_selector

Tests
I have setup the following test which shows the unexpected behavior. I first try a reset which correctly does the action_reset_full. I then go through the flow that sets the featurized slot preferences_filled to true and then start a search. So far, so good. When I used again a reset, the RulePolicy does not quick-in although it should have.

test_rules.yml

version: "2.0"
stories:
- story: search apartment
  steps:
    - user: |
        reset
      intent: reset
    - action: action_reset_full
    - user: |
        search apartments
      intent: ask_search_apartments
    - action: action_introduce_form_preferences
    - action: form_preferences
    - active_loop: form_preferences
    - slot_was_set:
        - region: 68773
    - slot_was_set:
        - requested_slot: null
    - active_loop: null
    - action: action_submit_form_preferences
    - slot_was_set:
        - preferences_filled: true
    - action: action_wait
    - action: action_get_apts_search
    - slot_was_set:
        - matches_found: 123
    - action: action_display_apt_selector
    - user: |
        reset
      intent: reset
    - action: action_reset_full

Test results

version: "2.0"
stories:
- story: search apartment (/tmp/tmpjpohyalo/cc6566db51884d09bca1ac66f7a2aec4_test_rules.yml)
  steps:
  - intent: reset
  - action: action_reset_full
  - intent: ask_search_apartments
  - action: action_introduce_form_preferences
  - action: form_preferences
  - active_loop: form_preferences
  - slot_was_set:
    - region: 68773
  - slot_was_set:
    - requested_slot: null
  - active_loop: null
  - action: action_submit_form_preferences
  - slot_was_set:
    - preferences_filled: true
  - action: action_wait
  - action: action_get_apts_search
  - slot_was_set:
    - matches_found: 123
  - action: action_display_apt_selector
  - intent: reset
  - action: action_reset_full  # predicted: action_default_fallback
  - action: action_listen  # predicted: action_default_fallback

Error (including full traceback):
action: action_reset_full predicted as action_default_fallback

Command or request that led to error:

 rasa test core -s tests/test_rules.yml

Content of configuration file (config.yml) (if relevant):

policies:
  - name: RulePolicy
    check_for_contradictions: true

Content of domain file (domain.yml) (if relevant):

slots:
  ...
  preferences_filled:
    type: bool
    initial_value: false
    influence_conversation: true

[sara-tagger]

Thanks for raising this issue, @tttthomasssss will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[nbeuchat]

The only way I managed to make it work is to add two rules such that both conditions preferences_filled: false and preferences_filled: true are covered:

version: "2.0"
rules:
- rule: Reset
  condition:
    - slot_was_set:
      - preferences_filled: false
  steps:
    - intent: reset
    - action: action_reset_full

- rule: Reset
  condition:
    - slot_was_set:
      - preferences_filled: true
  steps:
    - intent: reset
    - action: action_reset_full

Obviously, that will not work as I have a few slots that I will use in the conditions so I cannot make all combinations to ensure a rule is applied. My expectation is that if a slot is not in a condition, the rule is applied whatever the value of that slot is (even if it is not set).

[nbeuchat]

@koernerfelicia I would love your insights on this one! Am I misunderstanding how conditions should be set? Thanks a lot for your help!

[samsucik]

Hi @nbeuchat, this seems rather complex, let's try to unpack it :-) First of all, I wanted to ask about the problematic preferences_filled: true line in this rule:

- rule: Submit preferences form
  condition:
    - active_loop: form_preferences
  steps:
    # Form is deactivated
    - action: form_preferences
    - slot_was_set:
        - requested_slot: null
    - active_loop: null
    - action: action_submit_form_preferences
    - slot_was_set:
        - preferences_filled: true  # <-- THAT ONE IS CAUSING ISSUE DOWN THE LINE
    - action: action_wait
    - action: action_get_apts_search
  wait_for_user_input: false

First, let's make sure we both understand it the same way: Rules describe which action(s) should happen next, but they cannot trigger any SlotSet events. So, the problematic line in the rule above _describes_ that the preceding action (action_submit_form_preferences) has set the slot. However, then the role of the line isn't clear -- if it doesn't actually set the slot, then it could be a condition? Well, currently, I think, it behaves like a condition, but this is not the recommended (and reliable) way of using conditions -- use condition at the beginning of a rule instead.

All in all, I think the first step could be to remove that line altogether and check if you're still getting the weird behaviour. What do you say @nbeuchat ?

[samsucik]

@nbeuchat I now realised I was wrong in my previous comment. Sorry for that! I'm looking into it again.

[nbeuchat]

Hi @samsucik ! Thanks for your messages and for taking the time to look into it!

I like your approach of making it more explicit with the slot_was_set in the condition instead of within the steps. In the example above, I could easily break it into two rules. It might not always be possible to split the rules like that but I like it as a general approach.

However, I still have the issue that other stories that do not depend on that specific slot value are not triggered at all as they do not specify it in the condition

By the way, regarding your question in the forum, what I meant by "featurized slot" was a slot with influence_conversation: true

Thanks a lot for your help 🙂

[samsucik]

@nbeuchat I have some good news and some bad news 😉

Good news: I found out why the rule that seemingly doesn't condition on any slots still gets triggered depending on specific slot values. Bad news: It's quite a peculiar issue that leads us back to the basics of what's the purpose of initial_value. Let's see how exactly it will be fixed.

As soon as you define initial_value for some slot in your domain, the simple rule will condition on that slot value. Why? Because when RulePolicy is trained, it sees the set slot value present everywhere and it thinks that that's the "normal/default" case that it should always see. It kind of makes sense, but it's not intuitive and the user has no way to know about this implicit conditioning. Hence, we're considering a change that would ignore during training all slots set using initial_value. This way, only rules containing explicit condition or slot_was_set would be trained to condition on some slots. What do you say to this kind of change?

Anyway, for the time being, you might get rid of initial_value in your domain and things should work alright. The entire concept of initial_value is quite interesting as most of the times you shouldn't need to use it. Please, do let me know if you can think of a use case where initial_value really should be used, I'm curious to hear any thoughts!

[nbeuchat]

@samsucik That's great news!! I removed all initial_value for slots that influence the conversation and it works fine now 🥳 Thanks so much!

To be honest, the initial_value is not really necessary for our use cases. I was using it because I was using boolean slots and I didn't really care about the null case. My logic for a slot like preferences_filled was that it can either be True or False as when you start the conversation, the preferences are not filled (except if the user had them filled from another system but that doesn't matter for the point here). I can really just use null instead of False and everything will work fine.

The other use case I have for initial values is to keep track of some counts. For example, we could track how many viewings were requested and if it's many, we don't need that many explanations for example. In that case, I would intuitively put a 0 as the initial value but as for the preferences_filled above, a null would work just as well I suppose.

[samsucik]

Alright, @nbeuchat it's great to hear that things are working now! And thanks for describing the use cases; looks like initial_value isn't really needed here 🙂