Botocore: read_timeout in ~/.aws/config not working

[sean-zou]

I tried to put read_timeout in ~/.aws/config under default profile, and it didn't take the value.

Could it be read_timeout cannot be in ~/.aws/config?

It is similar to #1711

21:27:54 Traceback (most recent call last):
21:27:54 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
21:27:54 six.raise_from(e, None)
21:27:54 File "", line 3, in raise_from
21:27:54 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 416, in _make_request
21:27:54 httplib_response = conn.getresponse()
21:27:54 File "/usr/local/lib/python3.7/http/client.py", line 1344, in getresponse
21:27:54 response.begin()
21:27:54 File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
21:27:54 version, status, reason = self._read_status()
21:27:54 File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status
21:27:54 line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
21:27:54 File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
21:27:54 return self._sock.recv_into(b)
21:27:54 File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
21:27:54 return self.read(nbytes, buffer)
21:27:54 File "/usr/local/lib/python3.7/ssl.py", line 929, in read
21:27:54 return self._sslobj.read(len, buffer)
21:27:54 socket.timeout: The read operation timed out
21:27:54
21:27:54 During handling of the above exception, another exception occurred:
21:27:54
21:27:54 Traceback (most recent call last):
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
21:27:54 chunked=self._chunked(request.headers),
21:27:54 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
21:27:54 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
21:27:54 File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
21:27:54 raise six.reraise(type(error), error, _stacktrace)
21:27:54 File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
21:27:54 raise value
21:27:54 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
21:27:54 chunked=chunked,
21:27:54 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
21:27:54 self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
21:27:54 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
21:27:54 self, url, "Read timed out. (read timeout=%s)" % timeout_value
21:27:54 urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='ec2.us-east-1.amazonaws.com', port=443): Read timed out. (read timeout=60)
21:27:54
21:27:54 During handling of the above exception, another exception occurred:
21:27:54
21:27:54 Traceback (most recent call last):
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/commands.py", line 283, in run
21:27:54 policy()
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 1045, in call
21:27:54 resources = PullMode(self).run()
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 288, in run
21:27:54 resources = self.policy.resource_manager.resources()
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/resources/ec2.py", line 93, in resources
21:27:54 return super(EC2, self).resources(query=query)
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 460, in resources
21:27:54 resources = self.source.resources(query)
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 230, in resources
21:27:54 return self.query.filter(self.manager, query)
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 90, in filter
21:27:54 getattr(resource_manager, 'retry', None)) or []
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 69, in _invoke_client_enum
21:27:54 data = results.build_full_result()
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 449, in build_full_result
21:27:54 for response in self:
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter
21:27:54 response = self._make_request(current_kwargs)
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 686, in _make_request
21:27:54 return self.retry(self._method, *current_kwargs)
21:27:54 File "/usr/local/lib/python3.7/site-packages/c7n/utils.py", line 390, in _retry
21:27:54 return func(args, *kw)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 276, in _api_call
21:27:54 return self._make_api_call(operation_name, kwargs)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 573, in _make_api_call
21:27:54 operation_model, request_dict, request_context)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 592, in _make_request
21:27:54 return self._endpoint.make_request(operation_model, request_dict)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
21:27:54 return self._send_request(request_dict, operation_model)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
21:27:54 success_response, exception):
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
21:27:54 caught_exception=caught_exception, request_dict=request_dict)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
21:27:54 return self._emitter.emit(aliased_event_name, *kwargs)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
21:27:54 return self._emit(event_name, kwargs)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
21:27:54 response = handler(kwargs)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in call
21:27:54 if self._checker(attempts, response, caught_exception):
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in call
21:27:54 caught_exception)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
21:27:54 return self._checker(attempt_number, response, caught_exception)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in call
21:27:54 caught_exception)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in call
21:27:54 attempt_number, caught_exception)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
21:27:54 raise caught_exception
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
21:27:54 http_response = self._send(request)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
21:27:54 return self.http_session.send(request)
21:27:54 File "/usr/local/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
21:27:54 raise ReadTimeoutError(endpoint_url=request.url, error=e)
21:27:54 botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://ec2.us-east-1.amazonaws.com/"

[swetashre]

@sean-zou - Thank you for your post. According to the documentation you have to pass read_timeout in the Config section. Something like this:

import boto3
from botocore.config import Config

config = Config(read_timeout=1)
client = boto3.client('ec2',config=config)

I hope this helps.

[sean-zou]

@swetashre Thanks for the quick response!
Can we make this a feature request? Would be nice to have both read_timeout and connect_timeout configurable in .aws/config

[kapilt]

@swetashre this really should be in profile config imo, else every sdk using app has to reimplement this. ie in your example its being set on a config class which already sources several valuse from config files.. re every app reimplement, it looks like the aws cli already has to do this and introduce bespoke cli options for it (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-options.html).. as does every other app using boto, why not just make it available by default?

[swetashre]

@sean-zou @kapilt - yeah this seems reasonable to me. Marking this as feature request.