Cloud-custodian: ClientError: An error occurred (ThrottlingException) when calling the DescribeLogStreams operation (reached max retries: 4): Rate exceeded

[rracterr]

When running following policy:

CloudWatch Policies

- name: cloudwatch-delete-stale-log-group
  resource: log-group
  filters:
   - type: last-write
     days: 90
  actions:
   - delete

configuration:

using c7n_org, executing policies across 50 accounts.

Exception:

2017-11-20 17:45:50,684: c7n_org:INFO Ran account:entsys-qa region:us-west-2 policy:cloudformation-delete-failed-stacks matched:1 time:2.08
2017-11-20 17:45:55,970: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
File "/root/custodian/local/lib/python2.7/site-packages/c7n/policy.py", line 305, in run
resources = self.policy.resource_manager.resources()
File "/root/custodian/local/lib/python2.7/site-packages/c7n/query.py", line 397, in resources
return self.filter_resources(resources)
File "/root/custodian/local/lib/python2.7/site-packages/c7n/manager.py", line 88, in filter_resources
resources = f.process(resources, event)
File "/root/custodian/local/lib/python2.7/site-packages/c7n/resources/cw.py", line 185, in process
return super(LastWriteDays, self).process(resources)
File "/root/custodian/local/lib/python2.7/site-packages/c7n/filters/core.py", line 170, in process
return list(filter(self, resources))
File "/root/custodian/local/lib/python2.7/site-packages/c7n/resources/cw.py", line 194, in __call__
limit=3).get('logStreams')
File "/root/custodian/local/lib/python2.7/site-packages/botocore/client.py", line 312, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/root/custodian/local/lib/python2.7/site-packages/botocore/client.py", line 605, in _make_api_call
raise error_class(parsed_response, operation_name)
ClientError: An error occurred (ThrottlingException) when calling the DescribeLogStreams operation (reached max retries: 4): Rate exceeded

[thisisshi]

Rate exceeded errors are typically due to high API usage within each account/region. Check to see if there are other resources/actors making AWS API calls through automation or otherwise.

[rracterr]

The assumption that any large organization has only one application running against an environment at a given time is questionable. Rate limit exceeded errors should be handled by exponential backoff and coding should assume other actors in an environment.

[kapilt]

we have utilities for the exponential backoff, and we can add one here, thanks for the filing the issue, the reality is exponential backoff makes things worse for the other apps in the account as well, also looking at continuing some rate limiting work started at the reinvent sprint.

[kapilt]

found one service that actually describes their rate limits, http://docs.aws.amazon.com/step-functions/latest/dg/limits.html

[kapilt]

there's now retries for all resources when paginating for resource collection.

[kapilt]

the retry mechanism on resource fetching was enhanced to work per page in pagination instead of across the entire pagination last year as well. closing this out. in general we will eventually give up on a retry after backoff if the account is tipping over on throttle errors.