Botocore: Python 3.6: failing to make requests

[mattbriancon]

Versions

• boto3: 1.4.1
• botocore: 1.4.71
• python: 3.6.0b2 (side project; wanted to try out the f-strings!)
• os: Alpine Linux 3.4

I'm experiencing the following error when making any request via boto3. Using the S3 client as a minimal example:

Python 3.6.0b2 (default, Nov  1 2016, 00:18:55)
[GCC 5.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> s3 = boto3.client('s3')
>>> s3.list_buckets()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 251, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 526, in _make_api_call
    operation_model, request_dict)
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 141, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 170, in _send_request
    success_response, exception):
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 249, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 204, in _get_response
    proxies=self.proxies, timeout=self.timeout)
  File "/usr/local/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
TypeError: _send_request() takes 5 positional arguments but 6 were given

I traced the changed method signature to https://github.com/python/cpython/commit/a5289bebbf3997554e2cf5f386abcdba2391469d with more details on the bug tracker http://bugs.python.org/issue12319.

I was able to get around the issue by hacking together the following patch.

diff --git a/botocore/awsrequest.py b/botocore/awsrequest.py
index a667bb5..828fe32 100644
--- a/botocore/awsrequest.py
+++ b/botocore/awsrequest.py
@@ -118,7 +118,7 @@ class AWSHTTPConnection(HTTPConnection):
             if line in (b'\r\n', b'\n', b''):
                 break

-    def _send_request(self, method, url, body, headers):
+    def _send_request(self, method, url, body, headers, *py36_extra):
         self._response_received = False
         if headers.get('Expect', b'') == b'100-continue':
             self._expect_header_set = True
@@ -126,7 +126,7 @@ class AWSHTTPConnection(HTTPConnection):
             self._expect_header_set = False
             self.response_class = self._original_response_cls
         rval = HTTPConnection._send_request(
-            self, method, url, body, headers)
+            self, method, url, body, headers, *py36_extra)
         self._expect_header_set = False
         return rval

@@ -143,7 +143,7 @@ class AWSHTTPConnection(HTTPConnection):
         msg = b"\r\n".join(bytes_buffer)
         return msg

-    def _send_output(self, message_body=None):
+    def _send_output(self, message_body=None, **py36_extra):
         self._buffer.extend((b"", b""))
         msg = self._convert_to_bytes(self._buffer)
         del self._buffer[:]

Happy to provide more details!

[JordonPhillips]

I would rather just take *args, **kwargs in those methods since we don't know how they will be used. For _send_output we will need to look closer at what we want to do since the functionality is changing and we don't just wrap it.

[mattbriancon]

Yeah I didn't mean to propose my patch as a final solution, just another way to understand the issue.

[winston-zillow]

@JordonPhillips it seems the latest version is still 1.5.22 on PIP and still got this problem. Can this be published? thanks

[gamesbrainiac]

Hey folks! I am facing the same issue here. I am using boto3 1.9.10 and botocore 1.12.10. I am using Python 3.6.6.

[leleofg]

boto3 1.9.43 and same issue here.