Rasa: Does an end-to-end test mechanism have limit of the number instructions? (predicted: circuit breaker tripped)

Created on 20 May 2020  路  5Comments  路  Source: RasaHQ/rasa

Rasa version: 1.9.7
Rasa SDK version: 1.9.0
Python version: 3.7

Software:

  • ubuntu 20.04
  • docker version 18.09.9
  • docker compose version 1.25.5. Rasa backend container + mongoDB container for logs

Issue:
I鈥檓 working on the questionnaire (in italian). When I鈥檓 using the bot in a production (telegram as frontend) everything working fine, but when I trying to use end-to-end tests it fails. I checked all values it鈥檚 correct.

Error (including full traceback):

## Shortest path
* start: /start
    - utter_compliance
* agree_compliance: Accetto
    - slot{"age": 22}
    - slot{"gender": "m"}
    - slot{"school_degree": "laurea specialistica"}
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - slot{"physical_activity_per_day": "10 o pi霉"}
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"physical_activity_per_week": "150 o pi霉"}
    - physical_activity_per_week_form
    - form{"name": "physical_activity_per_week_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"red_meat": "meno di 2"}
    - red_meat_form
    - form{"name": "red_meat_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"cured_meat": "meno di 1"}
    - cured_meat_form
    - form{"name": "cured_meat_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"fish_products": "2 o pi霉"}
    - fish_products_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "fish_products_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"fruit_portions_per_day": "3 o pi霉"}
    - fruit_portions_per_day_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "fruit_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"vegetable_portions_per_day": "2 o pi霉"}
    - vegetable_portions_per_day_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "vegetable_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"sweets": "meno di 2"}
    - sweets_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "sweets_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"milk_products": "1 o pi霉"}
    - milk_products_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "milk_products_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"olive_oil": "si"}
    - olive_oil_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "olive_oil_form"}
    - form{"name": null}
    - action_finish_questionare   <!-- predicted: circuit breaker tripped -->
    - utter_thanks_for_answers   <!-- predicted: circuit breaker tripped -->
    - action_listen   <!-- predicted: circuit breaker tripped -->

Command or request that led to error:

sudo docker exec -ti $(sudo docker ps -f "name=rasa-backend" -q) rasa test core -s tests/test.md --e2e

Content of configuration file (config.yml):

language: it

pipeline:
  - name: "WhitespaceTokenizer"
  - name: "CRFEntityExtractor"
  - name: "EntitySynonymMapper"
  - name: "CountVectorsFeaturizer"
  - name: "EmbeddingIntentClassifier"

policies:
  - name: FormPolicy
  - name: FallbackPolicy
    nlu_threshold: 0.8
    ambiguity_threshold: 0.8
    core_threshold: 0.8
    fallback_action_name: action_default_fallback
  - name: MemoizationPolicy
  - name: MappingPolicy
  - name: TEDPolicy
    epochs: 100

Content of stories file (stories.md):

## standard workflow
* start
    - utter_compliance
* agree_compliance
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_week_form
    - form{"name": "physical_activity_per_week_form"}
    - form{"name": null}
    - action_delay_response
    - red_meat_form
    - form{"name": "red_meat_form"}
    - form{"name": null}
    - action_delay_response
    - cured_meat_form
    - form{"name": "cured_meat_form"}
    - form{"name": null}
    - action_delay_response
    - fish_products_form
    - form{"name": "fish_products_form"}
    - form{"name": null}
    - action_delay_response
    - fruit_portions_per_day_form
    - form{"name": "fruit_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - vegetable_portions_per_day_form
    - form{"name": "vegetable_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - sweets_form
    - form{"name": "sweets_form"}
    - form{"name": null}
    - action_delay_response
    - milk_products_form
    - form{"name": "milk_products_form"}
    - form{"name": null}
    - action_delay_response
    - olive_oil_form
    - form{"name": "olive_oil_form"}
    - form{"name": null}
    - action_finish_questionare
    - utter_thanks_for_answers

## non standard workflow
* start
    - utter_compliance
* not_now
    - utter_accept_compliance_to_start

Content of test file (test.md) :

## Accept rules later
* start: /start
    - utter_compliance
* not_now: Non ora
    - utter_accept_compliance_to_start

## Shortest path
* start: /start
    - utter_compliance
* agree_compliance: Accetto
    - slot{"age": 22}
    - slot{"gender": "m"}
    - slot{"school_degree": "laurea specialistica"}
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - slot{"physical_activity_per_day": "10 o pi霉"}
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"physical_activity_per_week": "150 o pi霉"}
    - physical_activity_per_week_form
    - form{"name": "physical_activity_per_week_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"red_meat": "meno di 2"}
    - red_meat_form
    - form{"name": "red_meat_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"cured_meat": "meno di 1"}
    - cured_meat_form
    - form{"name": "cured_meat_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"fish_products": "2 o pi霉"}
    - fish_products_form
    - form{"name": "fish_products_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"fruit_portions_per_day": "3 o pi霉"}
    - fruit_portions_per_day_form
    - form{"name": "fruit_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"vegetable_portions_per_day": "2 o pi霉"}
    - vegetable_portions_per_day_form
    - form{"name": "vegetable_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"sweets": "meno di 2"}
    - sweets_form
    - form{"name": "sweets_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"milk_products": "1 o pi霉"}
    - milk_products_form
    - form{"name": "milk_products_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"olive_oil": "si"}
    - olive_oil_form
    - form{"name": "olive_oil_form"}
    - form{"name": null}
    - action_finish_questionare
    - utter_thanks_for_answers

requirements.txt:

tensorflow==2.1.0
tensorflow-text==2.1.1
tensorflow-addons==0.9.1
rasa_sdk==1.9.0
rasa==1.9.7

What I've tried:

  1. Shuffle an order of forms (each form contains group of questions). The same error.
  2. Instead of
* agree_compliance: Accetto
    - slot{"age": 22}
...
  - personal_information_form

use

* agree_compliance: Accetto
    - personal_information_form
...
* form: [22](age)

The same error.

  1. Remove a few forms (in stories.md and tests.md). The same error.
  2. Use only two forms
## standard workflow
* start
    - utter_compliance
* agree_compliance
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}

This one is working fine. It seems like an end-to-end has limit of the number instructions. Probably the rasa in not the best choice for questionnaire, but any suggestions are welcomed. The whole questionnaire can be splitted on the group of questions with indentical logic, that鈥檚 why I used forms for it.

area type
Source
picture duketemon

Most helpful comment

Hi @duketemon, the maximum number of predictions before the circuitbreaker trips is controlled by the environmental variable MAX_NUMBER_OF_PREDICTIONS. The default number is 10
. See this issue for when and why the parameter was added to E2E testing https://github.com/RasaHQ/rasa/issues/5489.

Since you're using forms, it looks like in practice the user would be interrupting the story with input, meaning the bot doesn't actually predict the whole long list of events right off the bat.

For example here:

* agree_compliance: Accetto
    - slot{"age": 22}
    - slot{"gender": "m"}
    - slot{"school_degree": "laurea specialistica"}
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - slot{"physical_activity_per_day": "10 o pi霉"}
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}

Is the user inputting age, gender, and school degree?

picture melindaloubser1 on 26 May 2020
👍2

All 5 comments

Thanks for the issue, @melindaloubser1 will get back to you about it soon!

You may find help in the docs and the forum, too 馃
sara-tagger picture sara-tagger on 25 May 2020

Hi @duketemon, the maximum number of predictions before the circuitbreaker trips is controlled by the environmental variable MAX_NUMBER_OF_PREDICTIONS. The default number is 10
. See this issue for when and why the parameter was added to E2E testing https://github.com/RasaHQ/rasa/issues/5489.

Since you're using forms, it looks like in practice the user would be interrupting the story with input, meaning the bot doesn't actually predict the whole long list of events right off the bat.

For example here:

* agree_compliance: Accetto
    - slot{"age": 22}
    - slot{"gender": "m"}
    - slot{"school_degree": "laurea specialistica"}
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - slot{"physical_activity_per_day": "10 o pi霉"}
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}

Is the user inputting age, gender, and school degree?

melindaloubser1 picture melindaloubser1 on 26 May 2020
👍2

What I can see being an issue here, is if you don't get an error message during validation/training that too many events follow an input (i.e. it exceeds max_number_of_predictions). If that is the case, that might be a bug.

melindaloubser1 picture melindaloubser1 on 26 May 2020

Hi @melindaloubser1,
That's right. Every instruction slot{"slot_name": "value"} is an imitation of the user input.
Before run a test (rasa test core -s tests/test.md --e2e) I changed the MAX_NUMBER_OF_PREDICTIONS value and now it's working correctly.
Thank you

duketemon picture duketemon on 27 May 2020

Ok great!

melindaloubser1 picture melindaloubser1 on 27 May 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mit4dev picture mit4dev  路  4Comments
igormiranda001 picture igormiranda001  路  3Comments
rayush7 picture rayush7  路  3Comments
nikhilcss97 picture nikhilcss97  路  3Comments
N-Olbert picture N-Olbert  路  3Comments