Rasa: RASA Execution fails while running "rasa run -m models --enable-api --log-file out.log --endpoints endpoints.yml", after "rasa train"'s successful execution

[NikhilSrihari]

Rasa version: 1.10.14

Rasa SDK version (if used & relevant): 1.10.3

Rasa X version (if used & relevant): NA

Python version: 3.6.9

Operating system (windows, osx, ...): Ubuntu 18.04.2 LTS

Issue:
RASA Execution fails while running "rasa run -m models --enable-api --log-file out.log --endpoints endpoints.yml", after "rasa train"'s successful execution
"rasa train" created the following output:
models/20201120-191143.tar.gz

Error (including full traceback):

(rasa) root@e1694f49d39f:/aa/rasa-weatherbot# rasa run -m models --enable-api --log-file out.log --endpoints endpoints.yml
2020-11-20 19:24:19 INFO     root  - Starting Rasa server on http://localhost:5005
2020-11-20 19:24:32 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en'.
/pythonenvs/rasa/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
2020-11-20 19:24:32 ERROR    rasa.core.agent  - Could not load model due to 'str' object has no attribute 'decode'.
[2020-11-20 19:24:32 +0000] [145] [ERROR] Experienced exception while trying to serve
Traceback (most recent call last):
  File "/pythonenvs/rasa/lib/python3.6/site-packages/sanic/app.py", line 1192, in run
    serve(**server_settings)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/sanic/server.py", line 912, in serve
    trigger_events(before_start, loop)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/sanic/server.py", line 668, in trigger_events
    loop.run_until_complete(result)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/run.py", line 257, in load_agent_on_start
    action_endpoint=endpoints.action,
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/agent.py", line 306, in load_agent
    remote_storage=remote_storage,
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/agent.py", line 973, in load_local_model
    path_to_model_archive=model_archive,
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/agent.py", line 429, in load
    ensemble = PolicyEnsemble.load(core_model) if core_model else None
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/policies/ensemble.py", line 256, in load
    policy = policy_cls.load(policy_path)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/policies/keras_policy.py", line 259, in load
    model = load_model(model_file)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 146, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 166, in load_model_from_hdf5
    model_config = json.loads(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
Traceback (most recent call last):
  File "/pythonenvs/rasa/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/__main__.py", line 92, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/cli/run.py", line 83, in run
    rasa.run(**vars(args))
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/run.py", line 56, in run
    **kwargs,
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/run.py", line 216, in serve_application
    endpoints.lock_store if endpoints else None
  File "/pythonenvs/rasa/lib/python3.6/site-packages/sanic/app.py", line 1192, in run
    serve(**server_settings)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/sanic/server.py", line 912, in serve
    trigger_events(before_start, loop)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/sanic/server.py", line 668, in trigger_events
    loop.run_until_complete(result)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/run.py", line 257, in load_agent_on_start
    action_endpoint=endpoints.action,
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/agent.py", line 306, in load_agent
    remote_storage=remote_storage,
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/agent.py", line 973, in load_local_model
    path_to_model_archive=model_archive,
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/agent.py", line 429, in load
    ensemble = PolicyEnsemble.load(core_model) if core_model else None
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/policies/ensemble.py", line 256, in load
    policy = policy_cls.load(policy_path)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/rasa/core/policies/keras_policy.py", line 259, in load
    model = load_model(model_file)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 146, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
  File "/pythonenvs/rasa/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 166, in load_model_from_hdf5
    model_config = json.loads(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
sys:1: RuntimeWarning: coroutine 'Loop.create_server' was never awaited

Command or request that led to error:

rasa train #Which was successful. And then:
rasa run -m models --enable-api --log-file out.log --endpoints endpoints.yml #This one threw the error

Content of configuration file (config.yml) (if relevant):

language: "en"
pipeline: "pretrained_embeddings_spacy"


policies:
  - name: MemoizationPolicy
  - name: KerasPolicy
    epochs: 300
  - name: "FallbackPolicy"
    nlu_threshold: 0.35
    core_threshold: 0.3
    fallback_action_name: "action_default_fallback"

Content of domain file (domain.yml) (if relevant): Part of the domain.yml file

slots:
  ...

intents:
  - ...

entities:
  - ...

responses:
  utter_ask_action:
  - text: ...

actions:
  - ....

session_config:
  session_expiration_time: 30
  carry_over_slots_to_new_session: False

[sara-tagger]

Thanks for the issue, @rctatman will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[NikhilSrihari]

Thanks for the issue, @rctatman will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

@rctatman - Any updates on this?

[buzzf]

Thanks for the issue, @rctatman will get back to you about it soon!

You may find help in the docs and the forum, too hugs

@rctatman - Any updates on this?

I just got this issue, and then I changed my h5py version from 3.1.0 to 2.8.0, fixed it.

[luis-f-lins]

Thanks for the issue, @rctatman will get back to you about it soon!

You may find help in the docs and the forum, too hugs

@rctatman - Any updates on this?

I just got this issue, and then I changed my h5py version from 3.1.0 to 2.8.0, fixed it.

Thank you, @buzzf! This also worked for me.