Requests: TypeError: getresponse() got an unexpected keyword argument 'buffering'
Firstly, please note I have already read issues #1289 and #1915.
Secondly, please note that I am fully aware that the problem is that the underlying URL fetch failed due to a problem that is outside Requests' control, i.e. a network problem or failure of the remote web server.
My problem is not that an exception is raised - my problem is that the _wrong_ exception is raised. According to the Requests documentation, in this circumstance a ReadTimeout exception should be raised, and in any event the exception should be a subclass of RequestException. Instead, a TypeError is being raised. This is fatal for my application, which needs to know _why_ the request failed and to be sure that it was indeed a network failure and not a bug in the application or Requests itself. As far as I can see, this is indeed a bug in Requests.
I am using Requests 2.5.3 on Python 3.4.0 on Ubuntu 14.04.2.
Traceback (most recent call last):
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 374, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.4/http/client.py", line 1147, in getresponse
response.begin()
File "/usr/lib/python3.4/http/client.py", line 351, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.4/http/client.py", line 313, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.4/socket.py", line 371, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/packages/urllib3/util/retry.py", line 245, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/packages/urllib3/packages/six.py", line 310, in reraise
raise value
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 304, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='www.libraryofbirmingham.com', port=80): Read timed out. (read timeout=30)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./bitatawa-client.py", line 231, in check
timeout=settings.getint("check-timeout", 30))
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/api.py", line 65, in get
return request('get', url, **kwargs)
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/api.py", line 49, in request
response = session.request(method=method, url=url, **kwargs)
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/jribbens/src/bitatawa/lib/python3.4/site-packages/requests/adapters.py", line 433, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='www.libraryofbirmingham.com', port=80): Read timed out. (read timeout=30)
jribbens
All 10 comments
Sorry @jribbens, but you didn't read #1289 or #1915 clearly enough. I will reproduce my statements from those issues again, in the hopes that this will be the last time I have to write this down. =)
This is an unforeseen problem to do with how exception tracebacks are being reported in Python 3. PEP 3134 introduced this 'chaining exceptions' reporting that you can see in the traceback. The purpose of this error reporting is to highlight that some exceptions occur in except blocks, and to work out what chain of exceptions was hit. This is potentially very useful: for instance, you can hit an exception after destroying a resource and then attempt to use that resource in the except block, which hits another exception. It's helpful to be able to see both exceptions at once.
The key is that the TypeError raised as the first exception is unrelated to the subsequent ones. In fact, that's the standard control flow in urllib3. This means that the real exception that's being raised here is the request.exceptions.ReadTimeout exception that wraps the urllib3.exceptions.ReadTimeoutError exception being raised in urllib3.
The exception that actually bubbled up to the user code is the last one, not the first one. The way you read that traceback is as follows:
- I hit a
TypeErrorin this method. - In the
exceptblock that caught theTypeError, I raised asocket.timeout(in practice, this is well down the call stack, in a totally standard control flow) - In the
exceptblock that caught thesocket.timeout, I raised arequests.packages.urllib3.exceptions.ReadTimeoutError - In the
exceptblock that caught therequests.packages.urllib3.exceptions.ReadTimeoutError, I raised arequests.exceptions.ReadTimeout. - This exception was never caught.
To be clear, I have stated quite publicly that I think Python 3 screwed users on this 'feature'. I think it's a misfeature that discourages the Easier to Ask Forgiveness than Permission convention in the Python community because it leads to repeated bug reports like this one, even by people who honestly believe that they have read and understood the previous bug reports on this topic.
I promise you, the ReadTimeout is exactly what was raised here.
Lukasa
on 21 Mar 2015
Ah, I see. Thank you for your response. I had wrapped the requests call in a try: except requests.RequestException which is why I thought it was not raising ReadTimeout, but yet another exception in that block was confusing the issue even further.
Python 3's stack backtrace here does seem to be completely insane. I don't see the point of tracking back past a raise unless it was raise ... from .... But this is, of course, not your fault ;-)
jribbens
on 21 Mar 2015
Python 3's stack backtrace here does seem to be completely insane. I don't see the point of tracking back past a raise unless it was raise ... from .... But this is, of course, not your fault ;-)
I'm entirely agreed with you here, this has cost the requests team more time than I'd like. Such is life. =)
Lukasa
on 21 Mar 2015
Could you not just break the chain with raise <E> from None?
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> try:
... r = 5/0
... except:
... raise FileNotFoundError
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ZeroDivisionError: division by zero
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
FileNotFoundError
>>> try:
... r = 5/0
... except:
... raise FileNotFoundError from None
...
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
FileNotFoundError
Grazfather
on 6 Nov 2015
Nope: we have a common codebase for Python 2 and Python 3, and that syntax isn't valid in Python 2.
Lukasa
on 6 Nov 2015
Surely a test for legacy python would be worth avoiding all these bug reports?
Grazfather
on 6 Nov 2015
@Grazfather No, it's a syntax error: Python 2 cannot even parse the file with that in it
Lukasa
on 6 Nov 2015
A shame. Thanks.
Grazfather
on 7 Nov 2015
httplib_response = conn.getresponse(buffering=True) # This is almost certainly not the error you are looking for!
kevinburke
on 7 Nov 2015
(sorry)
kevinburke
on 7 Nov 2015
Related issues
Gonzalliz
路
3Comments
mitar
路
4Comments
xsren
路
3Comments
jake491
路
3Comments
justlurking
路
3Comments
Most helpful comment
Sorry @jribbens, but you didn't read #1289 or #1915 clearly enough. I will reproduce my statements from those issues again, in the hopes that this will be the last time I have to write this down. =)
This is an unforeseen problem to do with how exception tracebacks are being reported in Python 3. PEP 3134 introduced this 'chaining exceptions' reporting that you can see in the traceback. The purpose of this error reporting is to highlight that some exceptions occur in except blocks, and to work out what chain of exceptions was hit. This is potentially very useful: for instance, you can hit an exception after destroying a resource and then attempt to use that resource in the except block, which hits another exception. It's helpful to be able to see both exceptions at once.
The key is that the
TypeErrorraised as the first exception is unrelated to the subsequent ones. In fact, that's the standard control flow in urllib3. This means that the real exception that's being raised here is the request.exceptions.ReadTimeout exception that wraps the urllib3.exceptions.ReadTimeoutError exception being raised in urllib3.The exception that actually bubbled up to the user code is the last one, not the first one. The way you read that traceback is as follows:
TypeErrorin this method.exceptblock that caught theTypeError, I raised asocket.timeout(in practice, this is well down the call stack, in a totally standard control flow)exceptblock that caught thesocket.timeout, I raised arequests.packages.urllib3.exceptions.ReadTimeoutErrorexceptblock that caught therequests.packages.urllib3.exceptions.ReadTimeoutError, I raised arequests.exceptions.ReadTimeout.To be clear, I have stated quite publicly that I think Python 3 screwed users on this 'feature'. I think it's a misfeature that discourages the Easier to Ask Forgiveness than Permission convention in the Python community because it leads to repeated bug reports like this one, even by people who honestly believe that they have read and understood the previous bug reports on this topic.
I promise you, the
ReadTimeoutis exactly what was raised here.