Rasa: Next Action is predicted using KerasPolicy with Probability=1(randomly and not consistent) with Boolean Slots

[thepurpleowl]

Rasa Core version:
rasa-core (0.10.4)
rasa-nlu (not used)

Python version:
3.6.0

Operating system (windows, osx, ...):
Linux

Issue:
My stories are trained with both Memoization and Keras policy. When the slot values of my parse/continue requests match with the story, memoization policy is used to predict the next action as expected. However when my boolean slot has false in the request but my story has true, then Keras policy is being used in predicting the next action with confidence=1. I am not sure if getting nextaction with confidence value=1 using Keras Policy. Please help with more details on the same.

No changes have been done to the standard policies that are availble in the GIT repository. I am using it as is.

The issue is recreatable with all the below hyperparameter values.
no checkpoint in stories
epochs tested : 10,20,50,100
augmentation factor : 0, 20,50
MaxHistory : 1,3
Number of stories : 7
SlotType : Bool

Content of domain file (if used & relevant):
Stories format

## story_1
* intent_A
- action_A
- slot{key1 :val1, key2: val2}
* inform({key1 :val1, key2: val2})
- action_B
- slot{key3 :val3, key4: val4}
* inform({key3 :val3, key4: val4})
- action_C
- slot{key4 :val4, key5: val5}
* inform({key4 :val4, key5: val5})
- action_D
- slot{key6 :val6, key7: val7}
* inform({key6 :val6, key7: val7})
- action_E

In domain file, all the slots are of type 'bool'

[akelad]

This seems more like a usage question to me, please post these kinds of questions in the forum instead

[psds01]

Honestly, nobody answers questions in the forum. I have put many of my queries there, but still no response.

@thepurpleowl you need to remove slot from stories, don't set them manually, rasa does that for you:

## story_1
* intent_A
  - action_A
* inform({key1 :val1, key2: val2})
  - action_B
* inform({key3 :val3, key4: val4})
  - action_C
* inform({key4 :val4, key5: val5})
  - action_D
* inform({key6 :val6, key7: val7})
  - action_E

Enjoy.

[thepurpleowl]

This seems more like a usage question to me, please post these kinds of questions in the forum instead

@akelad I already had asked the question in forum, only after not getting any response I reached here. Completely agree with @psds01 with this; many others will also do.

[akelad]

Could you link your forum questions? we're working on answering more questions on there

[psds01]

https://forum.rasa.com/u/psds01

[akelad]

looks like they've pretty much all been answered now :)

[thepurpleowl]

@akela Forum question link - for this question

[thepurpleowl]

@psds01 Removing slots doesn't solve the problem.

[akelad]

let's keep the discussion on the forum

[thepurpleowl]

I found the root cause. This was happening because of a custom code written which was borrowed from line 90 in rasa_core-0.10.4\rasa_core\policies\ensemble.py.

Instead of having
      return max_index
in the next line(line no 93), I had changed it to
      return {"max_index":max_index,"score":"{:.2f}".format(probability_scores[max_index])}.

Because of the decision precision of 2 decimal points, decimals like 0.9995 where getting rounded off to 1.0.

As the logs should be in sync with result, so I checked the latest code for this log; to suggest change. Happy to see this formatting is not in the latest code.