Python-slack-sdk: TypeError - decode() argument 'encoding' must be str, not None

Created on 5 Oct 2020 · 7Comments · Source: slackapi/python-slack-sdk

It seems slack just had a blip and returned some kind of error responses that did not feature the charset header. This broke in the SDK with:

TypeError - decode() argument 'encoding' must be str, not None

On this line:

https://github.com/slackapi/python-slackclient/blob/5a2c80eb78fa82c2805251a60bc3e0eccb0b1269/slack/web/base_client.py#L489

From the Sentry local-caputring I can see charset was None - in typing terms it's Optional[str] being used as a str.

Reproducible in:

Requires slack returning these unexpected responses, or simulating them

The Slack SDK version

2.9.1

Python runtime version

3.8.5

OS info

Linux

Steps to reproduce:

As above

Expected result:

Some kind of fallback decoding?

Actual result:

TypeError raised

Requirements

For general questions/issues about Slack API platform or its server-side, could you submit questions at https://my.slack.com/help/requests/new instead. :bow:

Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

2x bug web-client

Source

adamchainz

All 7 comments

Seeing same error with slackclient version 2.8.2, Python version 3.7.3

jalanb on 5 Oct 2020

Thanks @adamchainz and @jalanb! It looks like Slack is having an intermitted issue right now.

I'm going to try to get a copy of the response headers and payload that is causing this issue in our SDK. If either of you have it on hand (@adamchainz maybe your Sentry issue has it logged) then I'd appreciate a copy (please scrub any confidential data). 🙏🏻

We can use the headers/payload to improve the error handling of the Python and Node SDKs.

mwbrooks on 5 Oct 2020

We were trying to message Slack from deployment scripts, which use stackprint. and end of that traceback follows:

File "/opt/wwts/venvs/deployers/lib/python3.7/site-packages/slack/web/base_client.py", line 352, in _urllib_api_call
    266  def _urllib_api_call(
    267      self,
    268      *,
    269      token: str = None,
    270      url: str,
    271      query_params: Dict[str, str] = {},
    272      json_body: Dict = {},
    273      body_params: Dict[str, str] = {},
    274      files: Dict[str, io.BytesIO] = {},
    275      additional_headers: Dict[str, str] = {},
    276  ) -> SlackResponse:
 (...)
    348          if query_params:
    349              q = urlencode(query_params)
    350              url = f"{url}&{q}" if "?" in url else f"{url}?{q}"
    351
--> 352          response = self._perform_urllib_http_request(url=url, args=request_args)
    353          if response.get("body", None):
    ..................................................
     self = <slack.web.client.WebClient object at 0x7f624882c358>
     token = None
     url = 'https://www.slack.com/api/chat.postMessage'
     query_params = None
     Dict = typing.Dict
     json_body = {'text': '```$ __main__ soso --servers=sosotest```',
                  'username': 'build_bot',
                  'icon_emoji': ':computer:',
                  'channel': '********'}
     body_params = None
     files = None
     io.BytesIO = <class '_io.BytesIO'>
     additional_headers = {'Content-Type': 'application/json;charset=utf-8',
                           'Authorization': '********',
                           'User-Agent': 'Python/3.7.3 slackclient/2.8.2 Linux/3.10.0-
                           957.12.2.el7.x86_64'}
     SlackResponse = <class 'slack.web.slack_response.SlackResponse'>
     self._perform_urllib_http_request = <method 'BaseClient._perform_urllib_http_request' of <slack.
                                          web.client.WebClient object at 0x7f624882c358> base_client.p
                                          y:384>
     request_args = {'headers': {'Content-Type': 'application/json;charset=utf-8
                     ',
                                 'User-Agent': 'Python/3.7.3 slackclient/2.8.2 L
                     inux/3.10.0-957.12.2.el7.x86_64',
                                 'Authorization': '********'},
                     'data': {},
                     'params': None,
                     'files': None,
                     'json': {'text': '```$ __main__ soso --servers=sosotest```'
                     ,
                              'username': 'build_bot',
                              'icon_emoji': ':computer:',
                              'channel': '*******'}}
    ..................................................

File "/opt/wwts/venvs/deployers/lib/python3.7/site-packages/slack/web/base_client.py", line 483, in _perform_urllib_http_request
    384  def _perform_urllib_http_request(
    385      self, *, url: str, args: Dict[str, Dict[str, any]]
    386  ) -> Dict[str, any]:
 (...)
    479              # for compatibility with aiohttp
    480              resp["headers"]["Retry-After"] = resp["headers"]["retry-after"]
    481
    482          charset = e.headers.get_content_charset()
--> 483          body: str = e.read().decode(charset)  # read the response body here
    484          resp["body"] = body
    ..................................................
     self = <slack.web.client.WebClient object at 0x7f624882c358>
     url = 'https://www.slack.com/api/chat.postMessage'
     args = {'headers': {'Content-Type': 'application/json;charset=utf-8
             ',
                         'User-Agent': 'Python/3.7.3 slackclient/2.8.2 L
             inux/3.10.0-957.12.2.el7.x86_64',
                         'Authorization': '********},
             'data': {},
             'params': None,
             'files': None,
             'json': {'text': '```$ __main__ soso --servers=sosotest```'
             ,
                      'username': 'build_bot',
                      'icon_emoji': ':computer:',
                      'channel': '********'}}
     Dict = typing.Dict
     resp = {'status': 503,
             'headers': <http.client.HTTPMessage object at 0x7f6244152c1
             8>}
     charset = None
     body = b'{"text": "```$ __main__ soso --servers=sosotest```", "user
             name": "build_bot", "icon_emoji": ":computer:", "channel": "
             ********"}'
    ..................................................

---- (full traceback above) ----
File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
File "/opt/wwts/clones/gitlab/bots/deployers/bots/apps/deploy/__main__.py", line 7, in <module>
    app.run()
File "/opt/wwts/clones/gitlab/bots/deployers/bots/apps/deploy_app.py", line 24, in run
    self._parse_args(inspect.currentframe().f_back)
File "/opt/wwts/clones/gitlab/bots/deployers/bots/apps/deploy_app.py", line 21, in _parse_args
    argv.parse_deploy_args(parser)
File "/opt/wwts/clones/gitlab/bots/deployers/tools/argv.py", line 110, in parse_deploy_args
    bots.build_bot.info(f'```$ {command_line()}```')
File "/opt/wwts/clones/gitlab/bots/deployers/tools/messaging/bots.py", line 95, in info
    self.log(template, *args)
File "/opt/wwts/clones/gitlab/bots/deployers/tools/messaging/bots.py", line 160, in log
    self.send(template, *args)
File "/opt/wwts/clones/gitlab/bots/deployers/tools/messaging/bots.py", line 63, in send
    self.api_send(message)
File "/opt/wwts/clones/gitlab/bots/deployers/tools/messaging/bots.py", line 74, in api_send
    icon_emoji=self.icon)
File "/opt/wwts/venvs/deployers/lib/python3.7/site-packages/slack/web/client.py", line 1041, in chat_postMessage
    return self.api_call("chat.postMessage", json=kwargs)
File "/opt/wwts/venvs/deployers/lib/python3.7/site-packages/slack/web/base_client.py", line 150, in api_call
    return self._sync_send(api_url=api_url, req_args=req_args)
File "/opt/wwts/venvs/deployers/lib/python3.7/site-packages/slack/web/base_client.py", line 248, in _sync_send
    additional_headers=headers,
File "/opt/wwts/venvs/deployers/lib/python3.7/site-packages/slack/web/base_client.py", line 352, in _urllib_api_call
    response = self._perform_urllib_http_request(url=url, args=request_args)
File "/opt/wwts/venvs/deployers/lib/python3.7/site-packages/slack/web/base_client.py", line 483, in _perform_urllib_http_request
    body: str = e.read().decode(charset)  # read the response body here

TypeError: decode() argument 1 must be str, not None

jalanb on 5 Oct 2020

👀1 👍1

I'm going to try to get a copy of the response headers and payload that is causing this issue in our SDK. If either of you have it on hand...

I'm afraid I don't have any more info than I presented. But it's any response without the Content-Type header - the function being called is EmailMessage.get_content_charset (yes EmailMessage, Python reuses the logic for HTTP headers since they're the same) - https://docs.python.org/3.8/library/email.message.html#email.message.EmailMessage.get_content_charset - this returns None for no such header.

adamchainz on 6 Oct 2020

👀1

Btw it looks like the "happy path" would also have this problem, since it has identical lines for handling the charset + body:

https://github.com/slackapi/python-slackclient/blob/5a2c80eb78fa82c2805251a60bc3e0eccb0b1269/slack/web/base_client.py#L478-L479

adamchainz on 6 Oct 2020

👀1

Thanks for the details everyone! It seems like the unexpected response is:

Status Code 503
Undefined Content-Type header

@seratch do you feel that we need more information to improve the error handling for v2 and v3?

mwbrooks on 6 Oct 2020

Thank you very much for reporting this issue (and we're sorry for the disruption due to the outage). The assumption where get_content_charset always returns a string seems to be obviously wrong.

@seratch do you feel that we need more information to improve the error handling for v2 and v3?

@mwbrooks We should resolve this issue in a new patch version for v2, plus in the latest beta for v3. I will check this issue later and work on the fix very soon.