Botocore: boto2.num_retries equivalent in retryhandler.py or _retry.json

[taraslayshchuk]

Hi!

I have a problem with using logic to retry failed requests.

lsb_release:    Ubuntu 14.04.4 LTS

---
Name: botocore
Version: 1.4.11
Location: /usr/local/lib/python2.7/dist-packages
Requires: docutils, jmespath, python-dateutil

14:00:52 Traceback (most recent call last):
14:00:52   File "deploy_scripts/gen-sg.py", line 638, in <module>
14:00:52     main()
14:00:52   File "deploy_scripts/gen-sg.py", line 631, in main
14:00:52     gsga_results = _get_all_IPs()
14:00:52   File "deploy_scripts/gen-sg.py", line 420, in _get_all_IPs
14:00:52     security_group_addresses = _get_all_IPs_for_Stacks(stacks_security_groups)
14:00:52   File "deploy_scripts/gen-sg.py", line 407, in _get_all_IPs_for_Stacks
14:00:52     security_group_addresses.update(_get_all_IPs_for_Groups(client, sec_groups))
14:00:52   File "deploy_scripts/gen-sg.py", line 387, in _get_all_IPs_for_Groups
14:00:52     groups = client.desc_sg(GroupIds=sec_groups)
14:00:52   File "/usr/lib/python2.7/site-packages/botocore/client.py", line 228, in _api_call
14:00:52     return self._make_api_call(operation_name, kwargs)
14:00:52   File "/usr/lib/python2.7/site-packages/botocore/client.py", line 492, in _make_api_call
14:00:52     raise ClientError(parsed_response, operation_name)
14:00:52 botocore.exceptions.ClientError: An error occurred (RequestLimitExceeded) when calling the DescribeSecurityGroups operation: Request limit exceeded.

I had used script for creating bulk of security group and I met RequestLimitExceeded Exception.
I don't see any probably workaround for result this issue, because I don't have any built in function to change retry config. (Yes, I can change json, or in some way do something like that:

client = boto3.client('ec2', region_name='us-west-2', config=boto3_config)
client.meta.events._unique_id_handlers['retry-config-ec2']['handler']._checker.__dict__['_max_attempts'] = 20

, or maybe I can create retryhandler with correct retry_config and transfer this object in some constructor, but I did not find this solution). So, is it true that botocore does not provide any supported and built-in methods for change retries count variable or delay base value?

[kyleknap]

Yeah there is not really any public methods for changing the retry logic. As of now, you have to modify the json at runtime or you can also override the _retry.json file with your own file and logic by placing the file at ~/.aws/models/_retry.json.

But in general, a public way to configure retries is something we would like to add. Marking as a feature request.

[jonathanwcrane]

+1 for this feature request--just ran into a ratelimitexceeded error and AWS Support says the retries happened well within the window where they would just add to the problem rather than ease it. So I'm not even sure that the exponential backoff retry code is working as intended.

[taraslayshchuk]

​@jonathanwcrane, Pull request #891 for this feature has been already created. You could use this PR, if it critical for you, until @jamesls did not look at it.

[tbarbugli]

+1 this should be a parameter

[spstarr]

Can we get https://github.com/boto/botocore/pull/891 merged please?

[joguSD]

Related to: https://github.com/boto/botocore/issues/1205

[adubkov]

I'm using this way. Inspired by botocore unit tests code:

from botocore.config import Config

config = Config(
    retries = dict(
        max_attempts = 10
    )
)

ec2 = boto3.client('ec2', config=config)

[taraslayshchuk]

@adubkov, you can do it because PR #1260 Add ability to set max retry attempts was merged and this feature was added.