Python-slack-sdk: Expose rate limited data from response headers

[Arjunkansagara]

• [ ] I've read and understood the Contributing guidelines and have done my best effort to follow them.
• [ ] I've read and agree to the Code of Conduct.
• [ ] I've searched for any related issues and avoided creating a duplicate issue.

Description

e.g. When we fetch user list from SlackClient API then we received following error from API.

Here is code,
sc = SlackClient(token)
api_c = sc.api_call("users.list")

File "SlackPost.py", line 24, in postSlackMessage
api_c = sc.api_call("users.list")
File "/app/.heroku/python/lib/python2.7/site-packages/slackclient/_client.py", line 77, in api_call
result = json.loads(self.server.api_call(method, **kwargs))
File "/app/.heroku/python/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/app/.heroku/python/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/app/.heroku/python/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Reproducible in:

• [ ] This is reproducible in the sample project.
  python-slackclient version: 1.0.1
  Python Version: 2.7
  OS Version: window

Steps to reproduce:

1. 
   
   
   2.
   
3.

Expected result:

e.g. What you expected to happen

Actual result:

e.g. What actually happened

Attachments:

e.g. Logs, screenshots, screencast, sample project, funny gif, etc.

[Arjunkansagara]

Just want to know, When will you fix this issue.

[neotheicebird]

Using Py3.5.1 I occasionally get

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

when executing

SlackClient.api_call("users.list")

======

   users_list = slack_client.api_call("users.list")
  File "C:\Program Files\Miniconda3\lib\site-packages\slackclient\_client.py", line 77, in api_call
    result = json.loads(self.server.api_call(method, **kwargs))
  File "C:\Program Files\Miniconda3\lib\json\__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\Miniconda3\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\Miniconda3\lib\json\decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[thiagogalesi]

Was there a reason why the Slack API was returning an undecodable object? 5XX errors? Throttling? Is the operation supposed to be retried if 'ok' is False?

[neotheicebird]

I understand that after #174 is merged into master this issue might go off, but as of now I still get this error.

If more than one request to users.list happens within a minute or 2, I still get the JSON Decode error. Is this actually a throttling issue?

Added info: The web api tester for the method here doesn't have such a restrictive throttling, as in if I manually try more than 10 calls a minute, it works each time.

[neotheicebird]

Sorry, I used an environment without the latest version. Will check again and get back

[SupermanIsEvious]

Based on my logging, the JSON decode error is a red herring for the real problem... rate-limiting.

2017-04-26 15:26:21,062 - requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): slack.com 2017-04-26 15:26:21,243 - requests.packages.urllib3.connectionpool - DEBUG - "POST /api/users.list HTTP/1.1" 429 48 2017-04-26 15:26:21,248 - src.utils.api.slack - ERROR - No JSON object could be decoded 2017-04-26 15:26:21,284 - src.utils.api.slack - DEBUG - Calling method 'users.list' with kwargs {}. Attempt 2 of 5 2017-04-26 15:26:21,285 - requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): slack.com 2017-04-26 15:26:21,362 - requests.packages.urllib3.connectionpool - DEBUG - "POST /api/users.list HTTP/1.1" 429 48 2017-04-26 15:26:21,366 - src.utils.api.slack - ERROR - No JSON object could be decoded 2017-04-26 15:26:21,366 - src.utils.api.slack - DEBUG - Calling method 'users.list' with kwargs {}. Attempt 3 of 5 2017-04-26 15:26:21,368 - requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): slack.com 2017-04-26 15:26:21,474 - requests.packages.urllib3.connectionpool - DEBUG - "POST /api/users.list HTTP/1.1" 429 48 2017-04-26 15:26:21,477 - src.utils.api.slack - ERROR - No JSON object could be decoded 2017-04-26 15:26:21,478 - src.utils.api.slack - DEBUG - Calling method 'users.list' with kwargs {}. Attempt 4 of 5 2017-04-26 15:26:21,479 - requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): slack.com 2017-04-26 15:26:21,554 - requests.packages.urllib3.connectionpool - DEBUG - "POST /api/users.list HTTP/1.1" 429 48 2017-04-26 15:26:21,559 - src.utils.api.slack - ERROR - No JSON object could be decoded
There's no method provided to actually handle rate limit errors via the api_call (SlackRequest.do()) method, since you don't store the full requests object:

https://github.com/slackapi/python-slackclient/blob/master/slackclient/_slackrequest.py#L48

Providing access to headers/status_code in your client would be the ideal solution here, as we could properly handle rate limits.

[Roach]

I agree that it would be great if we added more transparency into the response payload, but the way we're doing it now means this might be a breaking change, so we'll need to think a bit about how we want to make this happen.

[seandstewart]

If you wish to maintain the current workflow (i.e., return a Python dict object decoded via requests.json()) so as not to be a breaking change, you could add a status_code and header to your SlackRequest namespace. In that way, we could access this information without a breaking change, since it's adding functionality rather than changing output-type

[cicuz]

Is there any work being done on this? I'm experiencing a total lack of feedback when the rate limit is reached.. Actually, it seems like the post call doesn't return anything, which confuses me a little.

[aoberoi]

@cicuz we haven't prioritized development of this change yet, but if anyone would be willing to help with a contribution, i'd be happy to review and get it merged.

[cicuz]

I'm not sure what the best approach would be here: as I see it, either the response text gets json.parsed in the api-call method and augmented with status code and headers (for retry-after), but that sounds like ingerence by the server, or the api-call method returns a dictionary object containing the new properties as well as the actual response text, but that would end up in requiring some unnecessary and undesirable additional processing by the client.. Any suggestion as to how to structure the code?

[aoberoi]

i just tried to reproduce the conditions of this issue and i'm finding that rate limited responses are parsed as JSON just fine. i repeatedly called users.list (which has a pretty low rate limit of 1 call per 10 seconds but 10 calls in 1 second at burst) and got the following dict returned: {u'ok': False, u'error': u'ratelimited'}

which api method were you able to see the JSON parse error with?

note: i still think we should expose the status code and headers, i just want to understand how i can get to the error.

[cicuz]

Yes I saw the correct behaviour yesterday as well, but as far as I remember that was the first time (was calling chat.postMessage btw).

The issue remains about passing back the Retry-After field tho, as it is inconveniently stored outside of the error message.

[Roach]

Update: We're going to try to roll this into version 2 of the library, along with some other tweaks to how the payload is surfaced to provide more visibility into the response. 👍

[ClementViricel]

Sometimes (quite randomly) when I use slackclient.api_call with chat.postMessage from my bot to post a message to a user (using the user ID), it raises that same error. can someone help me with this ?

Here's the code:
slack_client.api_call("chat.postMessage", text=some_text, attachments=some_attachmennts, channel=user, as_user=True)
Here the error :
File "/usr/local/lib/python3.4/dist-packages/slackclient/client.py", line 184, in api_call
self.token, request=method, timeout=timeout, **kwargs
File "/usr/local/lib/python3.4/dist-packages/slackclient/server.py", line 350, in api_call
response_json = json.loads(response.text)
File "/usr/lib/python3.4/json/__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.4/json/decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.4/json/decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

[RodneyU215]

With respect to exposing the Retry-After header. It looks like this was done already. See #261.

@ClementViricel for the issue you're experiencing I believe I've fixed it here: https://github.com/slackapi/python-slackclient/pull/395.