Rasa: RasaX: Maximum recursion level reached
Rasa version: 1.9.5
Rasa SDK version (if used & relevant):
Rasa X version (if used & relevant): 0.27.4
Python version: 3.6
Operating system (windows, osx, ...): Ubuntu
Issue:
Started RasaX containers (RasaX, rasa-production, rasa-worker). Starting conversation with bot.
I migrated them to ElasticBeanstalk configuration (Dockerrun.aws.json).
Which authorization/token/username is it missing here?
Error (including full traceback):
rasax_1 | ERROR:asyncio:Task exception was never retrieved
rasax_1 | future: <Task finished coro=<run_operation_in_single_sanic_worker.<locals>.execute()
done, defined at /usr/local/lib/python3.6/site-packages/rasax/community/utils.py:760> exception=ServerDis
connectedError()>
rasax_1 | Traceback (most recent call last):
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasax/community/utils.py", line 768, i
n execute
rasax_1 | await f()
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service
.py", line 753, in discover_models
rasax_1 | await model_service.discover_models()
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service
.py", line 110, in discover_models
rasax_1 | await _run_loop(max_retries, sleep_in_seconds)
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service
.py", line 96, in _run_loop
rasax_1 | await self._discover_models()
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service
.py", line 152, in _discover_models
rasax_1 | minimum_version = await self._retry_fetching_minimum_compatible_version()
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service
.py", line 143, in _retry_fetching_minimum_compatible_version
rasax_1 | minimum_version = await self.minimum_compatible_version()
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service
.py", line 129, in minimum_compatible_version
rasax_1 | info = await stack_service.status()
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasax/community/services/stack_service
.py", line 52, in status
rasax_1 | return await self.stack_endpoint.request(subpath="/version", method="get")
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasa/utils/endpoints.py", line 146, in
request
rasax_1 | **kwargs,
rasax_1 | File "/usr/local/lib/python3.6/site-packages/aiohttp/client.py", line 1012, in __ae
nter__
rasax_1 | self._resp = await self._coro
rasax_1 | File "/usr/local/lib/python3.6/site-packages/aiohttp/client.py", line 504, in _requ
est
rasax_1 | await resp.start(conn)
rasax_1 | File "/usr/local/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 847, i
n start
rasax_1 | message, payload = await self._protocol.read() # type: ignore # noqa
rasax_1 | File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 591, in read
rasax_1 | await self._waiter
rasax_1 | aiohttp.client_exceptions.ServerDisconnectedError
[...]
rasax_1 | Exception occurred while handling uri: 'http://localhost:5002/api/version'
rasax_1 | Traceback (most recent call last):
rasax_1 | File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
rasax_1 | response = await response
rasax_1 | File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
rasax_1 | return response.json(result)
rasax_1 | File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
rasax_1 | dumps(body, **kwargs),
rasax_1 | OverflowError: Maximum recursion level reached
Command or request that led to error:
Content of configuration file (config.yml) (if relevant):
Content of domain file (domain.yml) (if relevant):
cristianmtr
All 27 comments
Thanks for raising this issue, @dakshvar22 will get back to you about it soon✨
Please also check out the docs and the forum in case your issue was raised there too 🤗
sara-tagger
on 25 May 2020
Hey @cristianmtr, we're not super familiar with deployment on ElasticBeanstalk. From what I can see here though, Rasa X is trying to call the rasa-production container on 'http://localhost:5002/, which I think is actually the URL for your rasa-x container. And so it seems like the container is calling itself.
Is it possible that you set the RASA_PRODUCTION_HOST (or RASA_WORKER_HOST) variable incorrectly somewhere? It may be worth exec-ing into the container and checking what those variables are set to.
akelad
on 27 May 2020
Hey @akelad
This is the environment of the container running rasa x.
rasa@6173079fd6d9:~$ env
[...]
RASA_PRODUCTION_HOST=http://192.168.178.30:5005
[...]
rasa@6173079fd6d9:~$ curl $RASA_PRODUCTION_HOST
Hello from Rasa: 1.9.5
It seems the env var is correctly set up. I fear rasax is using another hardcoded path somewhere
cristianmtr
on 27 May 2020
Also the /api/version is a RasaX API.
Rasa's API is only /version.
So it seems like Rasa is calling RasaX to get a version number?
Doing it myself, from the RasaX container, I can see the same error:
$ curl localhost:5002/api/version
{"version":"0.27.4","status":"failure","message":"See the server logs for more information.","reason":"The server encountered an internal error and cannot complete the request.","details":{},"help":null,"code":500}
So it seems there's something wrong with this endpoint in RasaX, perhaps.
cristianmtr
on 27 May 2020
if the rasa x container is crashing, then that response would be correct from a local curl. And I know that the /api/version refers to Rasa X usually, but the code at that point is referring to the stack_endpoint, which is the Rasa Open Source instance.
Are the logs you posted in the original issue actually from the rasa x container or a different one? I only ask because the service is usually called rasa-x rather than rasax. Maybe you can send the logs prior to this error as well, from both rasa-x and rasa-production.
akelad
on 29 May 2020
I renamed it to rasax during the conversion from docker-compose.yml to the EB equivalent Dockerrun.aws.json.
RasaX log:
INFO:rasax.community.services.event_service:Waiting until database migrations have been executed...
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.
INFO:rasax.community.services.event_service:Check for database migrations completed.
INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasax' on pika host 'amqp://ivhpdymg:[email protected]/ivhpdymg'.
[92mStarting Rasa X server... 🚀[0m
INFO:rasax.community.services.integrated_version_control.git_service:Cloning git repository from URL '[email protected]:alpaca-technology/chatbot.git'.
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<run_operation_in_single_sanic_worker.<locals>.execute() done, defined at /usr/local/lib/python3.6/site-packages/rasax/community/utils.py:760> exception=ServerDisconnectedError()>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/rasax/community/utils.py", line 768, in execute
await f()
File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 753, in discover_models
await model_service.discover_models()
File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 110, in discover_models
await _run_loop(max_retries, sleep_in_seconds)
File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 96, in _run_loop
await self._discover_models()
File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 152, in _discover_models
minimum_version = await self._retry_fetching_minimum_compatible_version()
File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 143, in _retry_fetching_minimum_compatible_version
minimum_version = await self.minimum_compatible_version()
File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 129, in minimum_compatible_version
info = await stack_service.status()
File "/usr/local/lib/python3.6/site-packages/rasax/community/services/stack_service.py", line 52, in status
return await self.stack_endpoint.request(subpath="/version", method="get")
File "/usr/local/lib/python3.6/site-packages/rasa/utils/endpoints.py", line 146, in request
**kwargs,
File "/usr/local/lib/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
self._resp = await self._coro
File "/usr/local/lib/python3.6/site-packages/aiohttp/client.py", line 504, in _request
await resp.start(conn)
File "/usr/local/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 847, in start
message, payload = await self._protocol.read() # type: ignore # noqa
File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 591, in read
await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError
ERROR:rasax.community.services.test_service:Unable to get tests from /app/git/1/tests/conversation_tests.md:
File '/app/git/1/tests/conversation_tests.md' does not exist.
[93m/usr/local/lib/python3.6/site-packages/rasa/core/domain.py:133: FutureWarning: Your domain file contains the key: 'templates'. This has been deprecated and renamed to 'responses'. The 'templates' key will no longer work in future versions of Rasa. Please replace 'templates' with 'responses'
return cls.from_dict(data)
[0m[93m/usr/local/lib/python3.6/site-packages/rasa/core/domain.py:151: FutureWarning: No tracker session configuration was found in the loaded domain. Domains without a session config will automatically receive a session expiration time of 60 minutes in Rasa version 2.0 if not configured otherwise.
session_config = cls._get_session_config(data.get(SESSION_CONFIG_KEY, {}))
[0mExecution of job "GitService.run_background_synchronization (trigger: cron[minute='*'], next run at: 2020-06-02 09:35:00 UTC)" skipped: maximum number of running instances reached (1)
Exception occurred while handling uri: 'http://localhost:5002/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
Exception occurred while handling uri: 'http://localhost:5002/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
Exception occurred while handling uri: 'http://localhost/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
Exception occurred while handling uri: 'http://localhost/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
Exception occurred while handling uri: 'http://localhost/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
Exception occurred while handling uri: 'http://localhost/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
Execution of job "GitService.run_background_synchronization (trigger: cron[minute='*'], next run at: 2020-06-02 09:36:00 UTC)" skipped: maximum number of running instances reached (1)
Exception occurred while handling uri: 'http://172.17.0.1:5002/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
Exception occurred while handling uri: 'http://172.17.0.1:5002/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
ERROR:rasax.community.api.blueprints.stack:Failed to send message to Rasa Chat webhook. Error: 401, Unauthorized, body='b'Error: Unauthorized''
Rasa prod logs:
Chatbot entrypoint.sh
rasa x --no-prompt --production --config-endpoint http://172.17.0.1:5002/api/config?token=i7v6989bsLK0ckz --port 5005 --jwt-method HS256 --jwt-secret Q1jpRXPUgi4hjVEVDCfWKw2ds --auth-token q3Uhf17EOAaRCL4 --debug --enable-api --cors *
2020-06-02 09:33:56 DEBUG rasa.cli.x - Failed to connect to server. Retrying.
2020-06-02 09:34:01 DEBUG rasa.cli.x - Failed to connect to server. Retrying. [Errno 104] Connection reset by peer
2020-06-02 09:34:07 DEBUG sanic_jwt.configuration - validating provided secret
2020-06-02 09:34:07 DEBUG sanic_jwt.configuration - validating keys (if needed)
2020-06-02 09:34:07 DEBUG sanic_jwt.configuration - loading secret and/or keys (if needed)
2020-06-02 09:34:07 DEBUG rasa.core.utils - Available web server routes:
/conversations/<conversation_id>/messages POST add_message
/conversations/<conversation_id>/tracker/events POST append_events
/auth OPTIONS auth_bp.AuthenticateEndpoint
/auth/me OPTIONS auth_bp.RetrieveUserEndpoint
/auth/verify OPTIONS auth_bp.VerifyEndpoint
/webhooks/rasa GET custom_webhook_RasaChatInput.health
/webhooks/rasa/webhook POST custom_webhook_RasaChatInput.receive
/model/test/intents POST evaluate_intents
/model/test/stories POST evaluate_stories
/conversations/<conversation_id>/execute POST execute_action
/domain GET get_domain
/ GET hello
/model PUT load_model
/model/parse POST parse
/conversations/<conversation_id>/predict POST predict
/conversations/<conversation_id>/tracker/events PUT replace_events
/conversations/<conversation_id>/story GET retrieve_story
/conversations/<conversation_id>/tracker GET retrieve_tracker
/status GET status
/model/predict POST tracker_predict
/model/train POST train
/conversations/<conversation_id>/trigger_intent POST trigger_intent
/model DELETE unload_model
/version GET version
2020-06-02 09:34:07 INFO root - Starting Rasa server on http://localhost:5005
2020-06-02 09:34:07 DEBUG rasa.core.utils - Using the default number of Sanic workers (1).
2020-06-02 09:34:07 INFO root - Enabling coroutine debugging. Loop id 94470717195752.
2020-06-02 09:34:07 DEBUG root - Could not load interpreter from 'None'.
2020-06-02 09:34:07 DEBUG rasa.core.brokers.broker - Instantiated event broker to 'PikaEventBroker'.
2020-06-02 09:34:07 DEBUG rasa.core.tracker_store - Attempting to connect to database via 'postgresql://postgres:[email protected]:5432/postgres'.
2020-06-02 09:34:07 DEBUG rasa.core.brokers.pika - RabbitMQ connection to 'amqp://ivhpdymg:[email protected]/ivhpdymg' was established.
2020-06-02 09:34:07 DEBUG rasa.core.brokers.pika - RabbitMQ channel was opened. Declaring fanout exchange.
2020-06-02 09:34:12 DEBUG rasa.core.tracker_store - Connection to SQL database 'tracker' successful.
2020-06-02 09:34:12 DEBUG rasa.core.tracker_store - Connected to SQLTrackerStore.
2020-06-02 09:34:12 DEBUG rasa.core.lock_store - Connected to lock store 'RedisLockStore'.
2020-06-02 09:34:12 DEBUG rasa.core.nlg.generator - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
2020-06-02 09:34:12 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:34:12 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:34:12 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:34:22 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:34:22 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:34:22 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:34:32 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:34:32 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:34:32 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:34:42 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:34:42 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:34:42 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:34:52 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:34:52 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:34:52 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:35:02 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:35:02 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:35:02 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:35:12 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:35:12 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:35:12 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:35:22 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:35:22 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:35:22 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:35:32 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:35:32 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:35:32 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:35:42 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:35:42 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:35:42 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:35:49 DEBUG rasa.core.tracker_store - Can't retrieve tracker matching sender id '97793766fe2c4d8e9bea9578df998a28' from SQL storage. Returning `None` instead.
2020-06-02 09:35:49 DEBUG rasa.core.tracker_store - Can't retrieve tracker matching sender id '97793766fe2c4d8e9bea9578df998a28' from SQL storage. Returning `None` instead.
2020-06-02 09:35:50 DEBUG rasa.core.tracker_store - Tracker with sender_id '97793766fe2c4d8e9bea9578df998a28' stored to database
2020-06-02 09:35:50 DEBUG rasa.core.processor - Starting a new session for conversation ID '97793766fe2c4d8e9bea9578df998a28'.
2020-06-02 09:35:50 DEBUG rasa.core.processor - Action 'action_session_start' ended with events '[<rasa.core.events.SessionStarted object at 0x7f56e66d1e80>, <rasa.core.events.ActionExecuted object at 0x7f56e66d1550>]'.
2020-06-02 09:35:51 DEBUG rasa.core.tracker_store - Can't retrieve tracker matching sender id '97793766fe2c4d8e9bea9578df998a28' from SQL storage. Returning `None` instead.
2020-06-02 09:35:52 DEBUG rasa.core.tracker_store - Can't retrieve tracker matching sender id '97793766fe2c4d8e9bea9578df998a28' from SQL storage. Returning `None` instead.
2020-06-02 09:35:53 DEBUG rasa.core.tracker_store - Tracker with sender_id '97793766fe2c4d8e9bea9578df998a28' stored to database
2020-06-02 09:35:53 DEBUG rasa.core.tracker_store - Can't retrieve tracker matching sender id '97793766fe2c4d8e9bea9578df998a28' from SQL storage. Returning `None` instead.
2020-06-02 09:35:53 DEBUG rasa.core.brokers.pika - Published Pika events to exchange 'rasa-exchange' on host 'amqp://ivhpdymg:[email protected]/ivhpdymg':
{"sender_id": "97793766fe2c4d8e9bea9578df998a28", "event": "action", "timestamp": 1591090550.07452, "name": "action_session_start", "policy": null, "confidence": null}
2020-06-02 09:35:53 DEBUG rasa.core.brokers.pika - Published Pika events to exchange 'rasa-exchange' on host 'amqp://ivhpdymg:[email protected]/ivhpdymg':
{"sender_id": "97793766fe2c4d8e9bea9578df998a28", "event": "session_started", "timestamp": 1591090550.074552}
2020-06-02 09:35:53 DEBUG rasa.core.brokers.pika - Published Pika events to exchange 'rasa-exchange' on host 'amqp://ivhpdymg:[email protected]/ivhpdymg':
{"sender_id": "97793766fe2c4d8e9bea9578df998a28", "event": "action", "timestamp": 1591090550.0746439, "name": "action_listen", "policy": null, "confidence": null}
2020-06-02 09:35:54 DEBUG rasa.core.tracker_store - Tracker with sender_id '97793766fe2c4d8e9bea9578df998a28' stored to database
2020-06-02 09:35:54 DEBUG rasa.core.lock_store - Deleted lock for conversation '97793766fe2c4d8e9bea9578df998a28'.
2020-06-02 09:36:02 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:36:02 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:36:02 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:36:07 ERROR rasa.core.channels.rasa_chat - Failed to fetch JWT public key from URL 'http://172.17.0.1:5002/api/version' with status code 500: {"version":"0.27.4","status":"failure","message":"See the server logs for more information.","reason":"The server encountered an internal error and cannot complete the request.","details":{},"help":null,"code":500}
2020-06-02 09:36:07 ERROR rasa.core.channels.rasa_chat - Failed to decode bearer token.
Traceback (most recent call last):
File "/opt/venv/lib/python3.6/site-packages/rasa/core/channels/rasa_chat.py", line 86, in _decode_bearer_token
return self._decode_jwt(bearer_token)
File "/opt/venv/lib/python3.6/site-packages/rasa/core/channels/rasa_chat.py", line 77, in _decode_jwt
authorization_header_value, self.jwt_key, algorithms=self.jwt_algorithm
File "/opt/venv/lib/python3.6/site-packages/jwt/api_jwt.py", line 92, in decode
jwt, key=key, algorithms=algorithms, options=options, **kwargs
File "/opt/venv/lib/python3.6/site-packages/jwt/api_jws.py", line 156, in decode
key, algorithms)
File "/opt/venv/lib/python3.6/site-packages/jwt/api_jws.py", line 220, in _verify_signature
key = alg_obj.prepare_key(key)
File "/opt/venv/lib/python3.6/site-packages/jwt/algorithms.py", line 209, in prepare_key
raise TypeError('Expecting a PEM-formatted key.')
TypeError: Expecting a PEM-formatted key.
2020-06-02 09:36:08 ERROR rasa.core.channels.rasa_chat - Failed to fetch JWT public key from URL 'http://172.17.0.1:5002/api/version' with status code 500: {"version":"0.27.4","status":"failure","message":"See the server logs for more information.","reason":"The server encountered an internal error and cannot complete the request.","details":{},"help":null,"code":500}
2020-06-02 09:36:08 ERROR rasa.core.channels.rasa_chat - Failed to decode bearer token.
Traceback (most recent call last):
File "/opt/venv/lib/python3.6/site-packages/rasa/core/channels/rasa_chat.py", line 86, in _decode_bearer_token
return self._decode_jwt(bearer_token)
File "/opt/venv/lib/python3.6/site-packages/rasa/core/channels/rasa_chat.py", line 73, in _decode_jwt
authorization_header_value = bearer_token.replace(
AttributeError: 'NoneType' object has no attribute 'replace'
[92mStarting Rasa X in production mode... 🚀[0m
[2020-06-02 09:36:08 +0000] [8] [ERROR] Exception occurred while handling uri: 'http://172.17.0.1:5005/webhooks/rasa/webhook'
Traceback (most recent call last):
File "/opt/venv/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
return self.gen.send(None)
File "/opt/venv/lib/python3.6/site-packages/rasa/core/channels/channel.py", line 462, in receive
sender_id = await self._extract_sender(request)
File "/usr/local/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
return self.gen.send(None)
File "/opt/venv/lib/python3.6/site-packages/rasa/core/channels/rasa_chat.py", line 105, in _extract_sender
abort(401)
File "/opt/venv/lib/python3.6/site-packages/sanic/exceptions.py", line 303, in abort
raise sanic_exception(message=message, status_code=status_code)
sanic.exceptions.Unauthorized: Unauthorized
2020-06-02 09:36:12 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:36:12 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:36:12 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:36:22 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:36:22 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:36:22 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:36:32 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:36:32 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:36:32 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:36:42 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:36:42 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:36:42 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:36:52 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:36:52 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:36:52 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:37:02 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:37:02 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:37:02 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:37:12 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:37:12 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:37:12 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:37:22 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:37:22 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:37:22 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:37:32 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:37:32 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:37:32 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:37:42 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:37:42 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:37:42 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:37:52 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:37:52 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:37:52 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:38:02 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:38:02 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:38:02 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:38:12 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:38:12 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:38:12 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:38:22 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:38:22 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:38:22 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:38:32 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:38:32 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:38:32 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:38:42 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:38:42 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:38:42 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:38:52 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:38:52 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:38:52 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:39:02 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:39:02 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:39:02 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:39:12 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:39:12 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:39:12 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:39:22 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:39:22 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:39:22 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:39:32 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:39:32 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:39:32 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:39:42 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:39:42 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:39:42 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:39:52 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:39:52 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:39:52 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
2020-06-02 09:40:02 DEBUG rasa.core.agent - Requesting model from server http://172.17.0.1:5002/api/projects/default/models/tags/production...
2020-06-02 09:40:02 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://172.17.0.1:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.
2020-06-02 09:40:02 DEBUG rasa.core.agent - No new model found at URL http://172.17.0.1:5002/api/projects/default/models/tags/production
Seems like rasa prod is trying to retrieve a token from RasaX, which crashes.
2020-06-02 09:36:07 ERROR rasa.core.channels.rasa_chat - Failed to fetch JWT public key from URL 'http://172.17.0.1:5002/api/version' with status code 500: {"version":"0.27.4","status":"failure","message":"See the server logs for more information.","reason":"The server encountered an internal error and cannot complete the request.","details":{},"help":null,"code":500}
2020-06-02 09:36:07 ERROR rasa.core.channels.rasa_chat - Failed to decode bearer token.
Traceback (most recent call last):
File "/opt/venv/lib/python3.6/site-packages/rasa/core/channels/rasa_chat.py", line 86, in _decode_bearer_token
return self._decode_jwt(bearer_token)
File "/opt/venv/lib/python3.6/site-packages/rasa/core/channels/rasa_chat.py", line 77, in _decode_jwt
authorization_header_value, self.jwt_key, algorithms=self.jwt_algorithm
File "/opt/venv/lib/python3.6/site-packages/jwt/api_jwt.py", line 92, in decode
jwt, key=key, algorithms=algorithms, options=options, **kwargs
File "/opt/venv/lib/python3.6/site-packages/jwt/api_jws.py", line 156, in decode
key, algorithms)
File "/opt/venv/lib/python3.6/site-packages/jwt/api_jws.py", line 220, in _verify_signature
key = alg_obj.prepare_key(key)
File "/opt/venv/lib/python3.6/site-packages/jwt/algorithms.py", line 209, in prepare_key
raise TypeError('Expecting a PEM-formatted key.')
TypeError: Expecting a PEM-formatted key.
@cristianmtr thanks for the additional logs, that definitely gives better context for the issue. We're looking into what may be going wrong
akelad
on 4 Jun 2020
👍1
Hey @cristianmtr! Are you still having problems with this issue? If so, would you mind sharing your environments.yml file?
federicotdn
on 17 Jun 2020
Hey @federicotdn
Yes, I am still experiencing the problem.
Here's the environments.yml file:
rasa:
production:
url: ${RASA_PRODUCTION_HOST}
token: ${RASA_TOKEN}
worker:
url: ${RASA_WORKER_HOST}
token: ${RASA_TOKEN}
To me it seems like Rasa X is encountering problems when trying to retrieve the version of either Rasa Open Source node (it retrieves the versions when the /api/version endpoint is called).
You did a test where, from the Rasa X container, you used curl $RASA_PRODUCTION_HOST to test the connectivity to that node. Could you try doing the same for RASA_WORKER_HOST as well?
federicotdn
on 19 Jun 2020
Hey again
It seems they are available
(from within the rasax container)
root@950537bf5fe5:~# curl $RASA_WORKER_HOST
Hello from Rasa: 1.9.5
root@950537bf5fe5:~# curl $RASA_PRODUCTION_HOST
Hello from Rasa: 1.9.5
Does the /version endpoint work correctly for both as well? That would be curl $RASA_XYZ_HOST/version for both
federicotdn
on 19 Jun 2020
You could also test a Rasa Open Source endpoint that requires authentication, e.g. /conversations/<sender_id>/tracker.
Maybe: curl $RASA_WORKER_HOST/conversations/foo/tracker?token=$RASA_TOKEN. If the token is correctly configured that should return 200.
federicotdn
on 19 Jun 2020
Yep, they seem to all work
root@950537bf5fe5:~# curl $RASA_WORKER_HOST/version
{"version":"1.9.5","minimum_compatible_version":"1.9.0"}
root@950537bf5fe5:~# curl $RASA_PRODUCTION_HOST/version
{"version":"1.9.5","minimum_compatible_version":"1.9.0"}
root@950537bf5fe5:~# curl $RASA_WORKER_HOST/conversations/foo/tracker?token=q3Uhf17EOAaRCL4
{"sender_id":"foo","slots":{},"latest_message":{"intent":{},"entities":[],"text":null,"message_id":null,"metadata":{}},"latest_event_time":1592580536.5613121986,"followup_action":null,"paused":false,"events":[{"event":"action","timestamp":1592580536.5611925125,"name":"action_session_start","policy":null,"confidence":null},{"event":"session_started","timestamp":1592580536.5612330437},{"event":"action","timestamp":1592580536.5613121986,"name":"action_listen","policy":null,"confidence":null}],"latest_input_channel":null,"active_form":{},"latest_action_name":"action_listen"}
root@950537bf5fe5:~# curl $RASA_PRODUCTION_HOST/conversations/foo/tracker?token=q3Uhf17EOAaRCL4
{"sender_id":"foo","slots":{},"latest_message":{"intent":{},"entities":[],"text":null,"message_id":null,"metadata":{}},"latest_event_time":1592580577.7758102417,"followup_action":null,"paused":false,"events":[{"event":"action","timestamp":1592580577.7757072449,"name":"action_session_start","policy":null,"confidence":null},{"event":"session_started","timestamp":1592580577.7757356167},{"event":"action","timestamp":1592580577.7758102417,"name":"action_listen","policy":null,"confidence":null}],"latest_input_channel":null,"active_form":{},"latest_action_name":"action_listen"}
root@950537bf5fe5:~#
Ok, that's good. Akela has shared with me your Dockerrun.aws.json file. I've never tried EB personally but I noticed that under the extraHosts key for Rasa X you have only configured rasa-production and not rasa-worker - is this intentional?
federicotdn
on 19 Jun 2020
Hey,
That's an artifact of me debugging the EB setup. It's not used anywhere at the moment. I tried with having both of them there, and without. The problem is the same.
cristianmtr
on 22 Jun 2020
For me it seems like the main error is here:
TypeError: Expecting a PEM-formatted key.
What would be this key and how can I check it's correct?
cristianmtr
on 22 Jun 2020
Unfortunately that's the jwt complaining about the JWT key being None. If you check the implementation of RasaChatInput, the self.jwt_key property is set using a value read from the Rasa X /api/version endpoint. Because that endpoint is not working correctly (as seen on your Rasa X logs), then this property is being set to None, which breaks the jwt.decode function afterwards.
federicotdn
on 22 Jun 2020
I see.
Pff, I'm at a loss here : I have tested connectivity via curl from each container to the others. They all passed
cristianmtr
on 22 Jun 2020
I think there's one more test you could do:
From both the containers with Rasa Open Source (worker and production), try to run curl ${RASA_X_HOST}/api/version. In theory that should return 500. Then you could paste here only the Rasa X logs generated when handling that request, without previous logs, so that we can be sure that the logs we're reading correspond to this error.
federicotdn
on 22 Jun 2020
after request from production:
Exception occurred while handling uri: 'http://172.17.0.1:5002/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
after request from worker
Exception occurred while handling uri: 'http://172.17.0.1:5002/api/version'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/usr/local/lib/python3.6/site-packages/rasax/community/api/blueprints/project.py", line 110, in version
return response.json(result)
File "/usr/local/lib/python3.6/site-packages/sanic/response.py", line 234, in json
dumps(body, **kwargs),
OverflowError: Maximum recursion level reached
Seems to be the same :
cristianmtr
on 23 Jun 2020
@federicotdn if it's easier and helpful, we could schedule a google call some time this week
cristianmtr
on 23 Jun 2020
That's a good idea, I'll send you an email. :+1:
federicotdn
on 23 Jun 2020
Just an update: this issue was due to the RASA_PRODUCTION_HOST and RASA_WORKER_HOST URLs not having the prefix http://.
cristianmtr
on 26 Jun 2020
👍1
Still, the error message could be improved.
cristianmtr
on 26 Jun 2020
👍1
Glad we managed to resolve it @cristianmtr! am going to close this, @federicotdn will you create an issue to improve the error message?
akelad
on 26 Jun 2020
Done!
federicotdn
on 26 Jun 2020
🎉1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
alonsopg
·
3Comments
Poojan66
·
3Comments
nicolasfarina
·
3Comments
edouardlp
·
3Comments
nikhilcss97
·
3Comments