Rasa: Fallback Policy always falls back

[DomAmato]

Rasa version: 1.8.0

Rasa SDK version (if used & relevant): not used

Rasa X version (if used & relevant): not used

Python version: 3.6.10

Operating system (windows, osx, ...): Docker

Issue:
Rasa is always using the fallback action no matter the confidence level in the predicted intent. In the logs below you can see it predicts the greet intent with a 0.9965264201164246 confidence but still falls back.

Here is the greet story:

## greet
* greet
   - utter_greet
   - utter_pose_query

Error (including full traceback):

2020-03-02 18:21:21.240376: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-02 18:21:21.241199: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-02 18:21:21.241271: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-03-02 18:21:23 DEBUG    rasa.cli.utils  - Parameter 'endpoints' not set. Using default location 'endpoints.yml' instead.
2020-03-02 18:21:23 DEBUG    rasa.cli.utils  - Parameter 'credentials' not set. Using default location 'credentials.yml' instead.
2020-03-02 18:21:23 DEBUG    rasa.cli.utils  - Parameter 'actions' not set. Using default location 'actions' instead.
2020-03-02 18:21:24 DEBUG    rasa.model  - Extracted model to '/tmp/tmpqjof7g9i'.
2020-03-02 18:21:25 DEBUG    sanic.root  - CORS: Configuring CORS with resources: {'/*': {'origins': [''], 'methods': 'DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT', 'allow_headers': ['.*'], 'expose_headers': None, 'supports_credentials': True, 'max_age': None, 'send_wildcard': False, 'automatic_options': True, 'vary_header': True, 'resources': {'/*': {'origins': ''}}, 'intercept_exceptions': True, 'always_send': True}}
2020-03-02 18:21:25 DEBUG    rasa.core.utils  - Available web server routes: 
/webhooks/rasa                                     GET                            custom_webhook_RasaChatInput.health
/webhooks/rasa/webhook                             POST                           custom_webhook_RasaChatInput.receive
/webhooks/rest                                     GET                            custom_webhook_RestInput.health
/webhooks/rest/webhook                             POST                           custom_webhook_RestInput.receive
/                                                  GET                            hello
/webhooks/slack                                    GET                            slack_webhook.health
/webhooks/slack/webhook                            POST                           slack_webhook.webhook
2020-03-02 18:21:25 INFO     root  - Starting Rasa server on http://localhost:5005
2020-03-02 18:21:25 DEBUG    rasa.core.utils  - Using the default number of Sanic workers (1).
2020-03-02 18:21:25 INFO     root  - Enabling coroutine debugging. Loop id 94616407104440.
2020-03-02 18:21:26 DEBUG    rasa.model  - Extracted model to '/tmp/tmp6y5fzs1b'.
2020-03-02 18:21:27 INFO     absl  - Using /tmp/tfhub_modules to cache modules.
2020-03-02 18:21:27.966728: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-03-02 18:21:53 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-03-02 18:21:55 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-03-02 18:21:55 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-03-02 18:22:03 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-03-02 18:22:03 DEBUG    rasa.nlu.classifiers.diet_classifier  - Failed to load model. Maybe the path '/tmp/tmp6y5fzs1b/nlu' doesn't exist?
2020-03-02 18:22:03 INFO     rasa.nlu.selectors.response_selector  - Retrieval intent parameter was left to its default value. This response selector will be trained on training examples combining all retrieval intents.
2020-03-02 18:22:03 DEBUG    rasa.core.tracker_store  - Connected to InMemoryTrackerStore.
2020-03-02 18:22:03 DEBUG    rasa.core.lock_store  - Connected to lock store 'InMemoryLockStore'.
2020-03-02 18:22:03 DEBUG    rasa.model  - Extracted model to '/tmp/tmptd5u5hu3'.
2020-03-02 18:22:03 DEBUG    pykwalify.compat  - Using yaml library: /usr/local/lib/python3.6/site-packages/ruamel/yaml/__init__.py
2020-03-02 18:22:04 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-03-02 18:22:04 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-03-02 18:22:04 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-03-02 18:22:05 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-03-02 18:22:05 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
2020-03-02 18:22:14 DEBUG    rasa.core.tracker_store  - Creating a new tracker for id 'dom'.
2020-03-02 18:22:14 DEBUG    rasa.core.processor  - Starting a new session for conversation ID 'dom'.
2020-03-02 18:22:14 DEBUG    rasa.core.processor  - Action 'action_session_start' ended with events '[<rasa.core.events.SessionStarted object at 0x7f47e73d9358>, <rasa.core.events.ActionExecuted object at 0x7f47e73d9320>]'.
2020-03-02 18:22:14 DEBUG    rasa.core.processor  - Current slot values: 
    detail: None
    environment: None
    id: None
    resource: None
2020-03-02 18:22:18 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model: component is either not trained or didn't receive enough training data.
2020-03-02 18:22:18 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
2020-03-02 18:22:18 DEBUG    rasa.core.processor  - Received user message 'Hello' with intent '{'name': 'greet', 'confidence': 0.9965264201164246}' and entities '[]'
2020-03-02 18:22:18 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2020-03-02 18:22:18 DEBUG    rasa.core.policies.memoization  - Current tracker state [{}, {'intent_greet': 1.0, 'prev_action_listen': 1.0}]
2020-03-02 18:22:18 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '17'
2020-03-02 18:22:18 DEBUG    rasa.core.policies.ensemble  - Action 'action_listen' was predicted after a user message using policy_0_FallbackPolicy. Predicting fallback action: action_listen
2020-03-02 18:22:18 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_FallbackPolicy
2020-03-02 18:22:18 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-03-02 18:22:18 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2020-03-02 18:22:18 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'dom'.

Content of configuration file (config.yml) (if relevant):

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en
pipeline:
  - name: ConveRTTokenizer
  - name: ConveRTFeaturizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
  - name: EntitySynonymMapper
  - name: ResponseSelector

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: "FallbackPolicy"
    nlu_threshold: 0.3
    ambiguity_threshold: 0.1
    core_threshold: 0.3
    fallback_action_name: "action_listen"
  - name: MemoizationPolicy
    max_history: 2
  - name: TEDPolicy
  - name: MappingPolicy

domain.yml

intents:
  - greet
  - query_hologram_organization
  - query_hologram_device
  - query_hologram_plan
  - query_hologram_pool
  - query_aws
  - affirm
  - deny
  - thank

slots:
  resource:
    type: text
  id:
    type: float
    max_value: 1000000000
    min_value: 1
  detail:
    type: text
  environment:
    type: text

entities:
  - resource
  - id
  - detail
  - environment

actions:
  - utter_greet
  - utter_did_that_help
  - utter_ask_more
  - utter_pose_query
  - utter_yay
  - utter_sorry
  - utter_goodbye

responses:
  utter_greet:
    - text: "Hi"
    - text: "Howdy"
    - text: "Hey"
    - text: "Hello"

  utter_pose_query:
    - text: "What can I help with?"
    - text: "What do you need to know?"
    - text: "What are you looking for?"

  utter_ask_more:
    - text: "Need anything else?"
    - text: "Is there anything else I can help with?"

  utter_did_that_help:
    - text: "Did that help you?"
    - text: "Is this what you were looking for?"

  utter_yay:
    - text: "Yay"
    - text: "Woo"
    - text: "Aww yea"

  utter_sorry:
    - text: "Sorry I will try harder next time"
    - text: "My bad"
    - text: "Aww :("

  utter_goodbye:
    - text: "Bye"
    - text: "So long"
    - text: "Aidos"
    - text: "Later"

session_config:
  session_expiration_time: 5 # value in minutes, 0 means infinitely long
  carry_over_slots_to_new_session: false # set to false to forget slots between sessions

[sara-tagger]

Thanks for the issue, @tabergma will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[Ghostvv]

Thank you for spotting it. I think, it happens because your fallback action is action_listen. FallbackPolicy predicts action_listen after the fallback action here: https://github.com/RasaHQ/rasa/blob/449f8c6ba13ceeca6661ad5b6b16c2e7ce435f2c/rasa/core/policies/fallback.py#L147 Which results in predicting action_listen after action_listen which is caught by ensemble to predict fallback action...

[Ghostvv]

I think this check should be fixed with not action_listen and also debug statement should be added there. Would you like to create a PR?

[DomAmato]

yeah I can do that, seems like an easy fix. Thanks for pointing me in the right direction :)

[chkoss]

I think this check should be fixed with not action_listen and also debug statement should be added there. Would you like to create a PR?

@Ghostvv What do you mean by debug statement? (I'm reviewing the PR, it fixes the check but doesn't add anything else)

[Ghostvv]

inside this if there is no debug statement why FallbackPolicy chose to predict itself. E.g. like it does it here: https://github.com/RasaHQ/rasa/blob/1c627fe2dbf92d41cfa3a685b0b66e5d1b4367dc/rasa/core/policies/fallback.py#L109

[chkoss]

It doesn't predict itself anymore though, or does it?
Inside this if it chooses to predict ACTION_LISTEN_NAME, but with the fixed check in the PR it only does this if self.fallback_action_name!= ACTION_LISTEN_NAME. So it will never predict the fallback action inside of this.

[Ghostvv]

it will predict action_listen if fallback_action_name is not action_listen. Then, it'd be good to have a debug statement, smth like "Predicted 'action_listen' after fallback action '{fallback_action_name}'"

[muthukumars]

I am facing the issue. can any help ?

[muthukumars]

Is there any specific env variable or config.yml should be set to solve this error.

[muthukumars]

My feedback.py is still as below

if tracker.latest_action_name == self.fallback_action_name:
result = self._default_predictions(domain)
idx = domain.index_for_action(ACTION_LISTEN_NAME)
result[idx] = 1.0

    elif self.should_nlu_fallback(nlu_data, tracker.latest_action_name):
        result = self.fallback_scores(domain)

    else:
        # NLU confidence threshold is met, so
        # predict fallback action with confidence `core_threshold`
        # if this is the highest confidence in the ensemble,
        # the fallback action will be executed.
        logger.debug(
            "NLU confidence threshold met, confidence of "
            "fallback action set to core threshold ({}).".format(
                self.core_threshold
            )
        )
        result = self.fallback_scores(domain, self.core_threshold)

    return result

this does not have latest changes happened 20 days ago. Please suggest.

[DomAmato]

did you pull the latest changes? How do you have Rasa installed? you would have to pull from master as this probably hasnt made its way to release yet

[DomAmato]

looking at release notes it should be in version 1.9.+ of rasa

[muthukumars]

I did pip install rasa. this will not pull from latest and I see my version is :

(Rasa-Virtual-Env) m.srinivasan@ATTACHCREDITTONE Rasa-Virtual-Env % rasa --version
Rasa 1.9.3
(Rasa-Virtual-Env) m.srinivasan@ATTACHCREDITTONE Rasa-Virtual-Env %

is this right version ?

[muthukumars]

DomAmato

I have built another virtual environment and pulled from source.

(venv) m.srinivasan@ATTACHCREDITTONE venv % rasa --version
Rasa 1.9.4
(venv) m.srinivasan@ATTACHCREDITTONE venv %

master branch.

I am still facing this issue.

[muthukumars]

any help

[DomAmato]

Sounds like something isnt right with the package uploaded to pypi. You might want to create a new issue for this as even though you are experiencing the same issue its likely for a different reason.

[muthukumars]

Sir, I have spent several hours to make this work.

Built a separate virtual instance (Ubuntu 18.x) in Vmware fusion. the issue still persists

I have created a bug

https://github.com/RasaHQ/rasa/issues/5547

[ClaudioVictorSpinelli]

I have the same issue, not always, specifically when after the first use of the complete form follows same chichat for example. No news for this?

[chkoss]

@ClaudioVictorSpinelli The issues that posters here were facing have been resolved.
Have you checked the other linked issue (https://github.com/RasaHQ/rasa/issues/5547) to see if it's the same thing for you? If it's different, please open a new issue.