Python-slack-sdk: Getting 404 error from WS URL in RTMClient#start() with aiohttp 3.7.0 or lower versions

[mgtech-sydney]

RTMClient.start() failing as of 2021-01-09. Connection attempted with both slack_sdk and (legacy) slackclient.

The Slack SDK version

slack-sdk 3.1.1

(Also occuring on slackclient 2.9.3)

Python runtime version

Python 3.6.7

OS info

Microsoft Windows [Version 10.0.18362.1256]

Steps to reproduce:

> import slack_sdk.rtm as rtm

> rtm_client = rtm.RTMClient(token=token) # also attempted with ssl_context provided
> rtm_client.start()

Testing via https://api.slack.com/methods/rtm.connect/test is successful with the same token as used when running code sample above.

Actual result:

  File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 473, in run_until_complete
    return future.result()
  File "D:\Repos\analysis-tools\slack-bot\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 360, in _connect_and_read
    proxy=self.proxy,
  File "D:\Repos\analysis-tools\slack-bot\src\env\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "D:\Repos\analysis-tools\slack-bot\src\env\lib\site-packages\aiohttp\client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')

[seratch]

@mgtech-sydney I guess the default SSL version in Python 3.6.7 on Windows OS may be a bit old. Can you try the following code out?

import logging
logging.basicConfig(level=logging.DEBUG)

import ssl
from slack_sdk.rtm import RTMClient

# Use this specific SSL version
ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
token = "xoxb-***"
RTMClient(token=token, ssl=ssl_context).start()

[mgtech-sydney]

Hi @seratch - output from running the above below, no change in the error.

I also ran this on python:3.7-alpine docker image and got the same result.

DEBUG:slack_sdk.rtm:Retrieving websocket info.
DEBUG:slack_sdk.web.legacy_base_client:Sending a request - url: https://www.slack.com/api/rtm.connect, query_params: {}, body_params: {}, files: {}, json_body: None, headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': '(redacted)', 'User-Agent': 'Python/3.6.7 slackclient/3.1.1 Windows/10'}
DEBUG:slack_sdk.web.legacy_slack_response:Received the following response - status: 200, headers: {'date': 'Mon, 11 Jan 2021 03:39:22 GMT', 'server': 'Apache', 'x-xss-protection': '0', 'pragma': 'no-cache', 'cache-control': 
'private, no-cache, no-store, must-revalidate', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'x-slack-req-id': 'f3ae8d4f615cb9bbf668bf34d1505b08', 'x-content-type-options': 'nosniff', 'referrer-policy': 'no-referrer', 'access-control-expose-headers': 'x-slack-req-id, 
retry-after', 'x-slack-backend': 'r', 'x-oauth-scopes': 'identify,bot:basic', 'x-accepted-oauth-scopes': 'rtm:stream,client', 'expires': 'Mon, 26 Jul 1997 05:00:00 GMT', 'vary': 'Accept-Encoding', 'access-control-allow-headers': 'slack-route, x-slack-version-ts, x-b3-(continues...)
Traceback (most recent call last):
  File "main.py", line 23, in <module>
    rtm.start()
...\__init__.py", line 52, in start
    rtm_client.start()
  ...\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 209, in start
    return self._event_loop.run_until_complete(future)
  File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 473, in run_until_complete
    return future.result()
  ...\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 360, in _connect_and_read
    proxy=self.proxy,
  ...\src\env\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  ...src\env\lib\site-packages\aiohttp\client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')

[seratch]

Can you check the OpenSSL version you are using? You can check the version by printing ssl.OPENSSL_VERSION in Python. If you are not using the latest (1.1.1), upgrading the version and enabling it for your Python interpreter may help.

[jvperrin]

I'm getting this too, from an ubuntu bionic container:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/code/marley/main.py", line 297, in <module>
    main()
  File "/code/marley/main.py", line 293, in main
    marley.start()
  File "/code/marley/main.py", line 190, in start
    self.rtm.start()
  File "/code/virtualenv_run/lib/python3.7/site-packages/slack_sdk/rtm/__init__.py", line 209, in start
    return self._event_loop.run_until_complete(future)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/code/virtualenv_run/lib/python3.7/site-packages/slack_sdk/rtm/__init__.py", line 360, in _connect_and_read
    proxy=self.proxy,
  File "/code/virtualenv_run/lib/python3.7/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/code/virtualenv_run/lib/python3.7/site-packages/aiohttp/client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')

I am using Python 3.7.5 and OpenSSL 1.1.1:

$ /code/virtualenv_run/bin/python
Python 3.7.5 (default, Nov  7 2019, 10:50:52)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.1  11 Sep 2018'

Adding in ssl.SSLContext(ssl.PROTOCOL_SSLv23) and RTMClient(token=token, ssl=ssl_context).start() seems to have no effect either, unfortunately.

[Harpsichord1207]

I got the same error since last night(01/11), without modifying any code. Maybe it's a slack server error?

[KIM-DONGWON]

I am using Python 3.7 and docker environment and the same error occurs but I cannot find the cause. Is it an error on the slack server?

[seratch]

We are sorry for the disruption.

I have been trying to reproduce the situation on my end. However, I haven't managed to see the error with my classic apps yet.

We would like to figure out if this is specific to this SDK or Python runtime first. It would be greatly appreciated if anyone could help us identify the conditions. What happens if you try to connect using the same classic bot token with @slack/rtm-api package in the Node SDK? Does the same situation arise?

If the issue occurs with Node SDK too, we will communicate to the server-side teams an closely work with them. If this happens only with this Python SDK, I will continue looking at the steps to reproduce.

[ongjp]

I am having the same problem here as everyone using the RTMClient. I am using Python 3.7 and my RTM chatbot was working up until about 1 hour ago.
Unfortunately I am not familiar with Node, but I can confirm on Python it is not working for me either.

[undral23]

I got the same error since about 3-4 hours ago.

[redliu312]

me too
OpenSSL OpenSSL 1.1.1d
Python 3.8.3

[jusdino]

Same problem. Python 3.6, OpenSSL 1.1.1.

Looks like there may be a lot of us. Perhaps we should switch to :+1: on the post if we have no new info to add?

[Harpsichord1207]

@seratch I'm not familiar with node, i try to do the following steps:

• npm install @slack/rtm-api
• create a test.js with content:
  ```
  const { RTMClient } = require('@slack/rtm-api');

// Read a token from the environment variables
const token = 'xoxb-xxxxxxxxxxxxxxx';

// Initialize
const rtm = new RTMClient(token);

(async () => {
  // Connect to Slack
  const { self, team } = await rtm.start();
})();
```

• run node test.js

and there‘s no output or error with correct token. If I set a wrong token, it raise an UnhandledPromiseRejectionWarning: Error: An API error occurred: invalid_auth, so I guess Node SDK works fine.

[sujan-paidipala]

I see the same problem since couple of hours.
+1

[Yannnnn001]

I see the same problem since couple of hours.
+1

[edribeirojr]

I see the same problem since couple of hours.
+1

[brendan-mz]

My bot started experiencing the same issue at 6:45pm PST and has not been able to re-connect since.

Python 3.6.8
OpenSSL 1.0.2s

[aoberoi]

For those of you experiencing this issue, we'd like to know if you are (👍) or are not (👎) using the proxy option to connect via a proxy. Would you mind reacting to this comment?

[aoberoi]

@jvperrin and @mgtech-sydney did you both redact the URL on the last line of this stack trace, or is this the exact URL you saw?

if you redacted it, given that those URLs expire within 10 mins, they would no longer be dangerous to share. and just being cautious, if you'd like to email me (ankur @ slack-corp dot com), seeing those URLs might be helpful to our investigation.

DEBUG:slack_sdk.rtm:Retrieving websocket info.
DEBUG:slack_sdk.web.legacy_base_client:Sending a request - url: https://www.slack.com/api/rtm.connect, query_params: {}, body_params: {}, files: {}, json_body: None, headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': '(redacted)', 'User-Agent': 'Python/3.6.7 slackclient/3.1.1 Windows/10'}
DEBUG:slack_sdk.web.legacy_slack_response:Received the following response - status: 200, headers: {'date': 'Mon, 11 Jan 2021 03:39:22 GMT', 'server': 'Apache', 'x-xss-protection': '0', 'pragma': 'no-cache', 'cache-control': 
'private, no-cache, no-store, must-revalidate', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'x-slack-req-id': 'f3ae8d4f615cb9bbf668bf34d1505b08', 'x-content-type-options': 'nosniff', 'referrer-policy': 'no-referrer', 'access-control-expose-headers': 'x-slack-req-id, 
retry-after', 'x-slack-backend': 'r', 'x-oauth-scopes': 'identify,bot:basic', 'x-accepted-oauth-scopes': 'rtm:stream,client', 'expires': 'Mon, 26 Jul 1997 05:00:00 GMT', 'vary': 'Accept-Encoding', 'access-control-allow-headers': 'slack-route, x-slack-version-ts, x-b3-(continues...)
Traceback (most recent call last):
  File "main.py", line 23, in <module>
    rtm.start()
...\__init__.py", line 52, in start
    rtm_client.start()
  ...\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 209, in start
    return self._event_loop.run_until_complete(future)
  File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 473, in run_until_complete
    return future.result()
  ...\src\env\lib\site-packages\slack_sdk\rtm\__init__.py", line 360, in _connect_and_read
    proxy=self.proxy,
  ...\src\env\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  ...src\env\lib\site-packages\aiohttp\client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')

[mgtech-sydney]

Hi @aoberoi

The below is a URL from our production environment (Alpine:Python3.7) which saw the error.

wss://cerberus-xxxx.lb.slack-msgs.com/websocket/6_EWYtaXs4UzN7m2JH0huysec3Di2Kgi5cF_77o8hJA_AtEfGxnKC9OI2lAJDAHPixU0ZwDo7y7UNfhspz4KHdj35qcIAhznqkuARS75wAA=

I can generate the error / URL locally in a Windows environment if that would help you as well.

[jhofeditz]

@aoberoi we are seeing the same thing just started in the last 3 hours. No proxy that I am aware of.

Connecting to Slack using RTM api.
Connection to slack failed: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/FGtDpGKyogzoJ3k3qzT2S5cfipIxjI4AnBZhRmp2AYk4h6FaxgNPtcV_nnnNMujbd19vwtmvWi0J-uflBvJGzGHbXcGujUh_J4TIizcuDkE=')

[ongjp]

I was facing the same issue. It started working after doing the following:

1. Upgrading aiohttp - pip install -U aiohttp
2. from slack import RTMClient -> from slack_sdk.rtm import RTMClient

this worked for me - thanks very much

[mgtech-sydney]

For me also. Thanks.

[brendan-mz]

Confirming the aiohttp upgrade works. I was on 3.6.2, upgraded to 3.7.3 and everything started working again.

[jhofeditz]

we just deployed a new release upgrading slackclient from v2.7.3 to v2.9.3 and it seems to be working now.

[seratch]

To: all in this thread
Thank you very much for sharing the workaround.

I was on 3.6.2, upgraded to 3.7.3 and everything started working again.

@brendan-mz Thanks a lot for this helpful information. I managed to reproduce the issue with aiohttp==3.6.2 too.

While our server-side teams are actively working on this and the issue will be resolved soon, as a workaround at this point, please upgrade your aiohttp version to the 3.7.1 or higher if you're using an older version.

Update: using aiohttp==3.7.3 (at least 3.7.1+) is recommended. It seems 3.7.0 does not work due to another issue.

[fitz-xue]

Confirming the aiohttp upgrade works. I was on 3.6.2, upgraded to 3.7.3 and everything started working again.

works for me. :flower:

[sujan-paidipala]

Working for me too after aiohttp upgrade. Thanks for the help!

[JeckPikachan]

Upgrading aiohttp worked for me too. Thanks a lot!

[makdee88]

Upgrading to aiohttp-3.7.3 and slackclient-2.9.3 fixed the issue. Thanks a ton.

[slackbot@xxxxslackbot]$ pip3 install -U aiohttp --proxy http://x.x.x:8888 --user slackbot
...
Successfully installed aiohttp-3.7.3 attrs-20.3.0 certifi-2020.12.5 idna-3.1 multidict-5.1.0 requests-2.25.1 urllib3-1.26.2 yarl-1.6.3

[slackbot@xxxx slackbot]$
[slackbot@xxxx slackbot]$ pip3 install -U slackclient --proxy http://x.x.x:8888 --user slackbot
...
Successfully installed idna-2.10 slackclient-2.9.3
[slackbot@xxxx slackbot]$

[andrey-tikhov]

Saving me. SAVING ME!!!! aiohttp-3.7.3

[hkonala]

@seratch I am still getting the same error even after upgrading aiohttp and slackclient. Could someone help?

[ec2-user@xxxxxxxxxxxxx skills]$ pip freeze | grep -E 'slackclient|aiohttp|opsdroid'
aiohttp==3.7.3
opsdroid==0.19.0
opsdroid-get-image-size==0.2.2
slackclient==2.9.3

Traceback (most recent call last):
  File "/usr/local/bin/opsdroid", line 11, in <module>
    load_entry_point('opsdroid==0.19.0', 'console_scripts', 'opsdroid')()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/opsdroid/cli/start.py", line 43, in start
    opsdroid.run()
  File "/usr/local/lib/python3.6/site-packages/opsdroid/core.py", line 169, in run
    self.eventloop.run_until_complete(asyncio.gather(*pending))
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/slack/rtm/client.py", line 357, in _connect_and_read
    proxy=self.proxy,
  File "/usr/local/lib64/python3.6/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib64/python3.6/site-packages/aiohttp/client.py", line 738, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status', url=URL('wss://cerberus-xxxx.lb.slack-msgs.com/websocket/fH8ZRCPmqUOTp3W58-XT7cylC8tGT5m1n_xr1U3ji9Ea2WEJQIaER14w9dNJ9qeDFwrLliVkW3ONyfE7Qs8mOp8S6e4XBcHIuDrh4US2KSA=')
ERROR: Unhandled exception in opsdroid, exiting...

[seratch]

File "/usr/local/lib64/python3.6/site-packages/aiohttp/client.py", line 738, in _ws_connect
headers=resp.headers)

@hkonala The stacktrace indicates that your app still loads aiohttp v3.6

• https://github.com/aio-libs/aiohttp/blob/v3.6.3/aiohttp/client.py#L738
• https://github.com/aio-libs/aiohttp/blob/v3.7.3/aiohttp/client.py#L738

[hkonala]

@seratch it is saying opsdroid 0.19.0 requires slackclient==2.6.2 and aiohttp==3.6.2

[ec2-user@ip-xxxxxxxx site-packages]$ opsdroid start
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (slackclient 2.9.3 (/home/ec2-user/.local/lib/python3.6/site-packages), Requirement.parse('slackclient==2.6.2'), {'opsdroid'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/opsdroid", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3261, in <module>
    @_call_aside
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3245, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3274, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 586, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'aiohttp==3.6.2' distribution was not found and is required by opsdroid

[seratch]

@hkonala I am sorry to say that, in your case, there is no workaround until the issue is fixed on the server-side. I cannot tell the exact timing of the deployment, but they will resolve this early tomorrow, Pacific Time. Again, we are sorry for the disruption.

[hkonala]

@seratch Thanks for the info!! will wait for the fix :)

[biboc]

Upgraded to aiohttp==3.7.3 and it works
But there is big delay between question from user and bot receiving the question now

[tgwaste]

What is the status of the fix?

[fndiaz]

Install _aiohttp_, work for me too!

[tgwaste]

A freaking heads up from slack sure would had been nice. They sure have interrupted business a lot lately.

[spladug]

For anyone else running into this that isn't using aiohttp, it appears this was the change to aiohttp that made the difference.

If your client implementation sends any capitalization other than Upgrade: websocket (e.g. Upgrade: WebSocket) you'll get the 404.

In my case, I patched autobahn here: https://github.com/crossbario/autobahn-python/blob/master/autobahn/websocket/protocol.py#L3576

[seratch]

According to the server-side teams, no apps should be experiencing this issue now. We apologize for the inconvenience and appreciate your patience.

[hkonala]

Yes @seratch .. Things seem to be working fine now.