Scoutsuite: Feature Request - handle API rate limiting

[museadmin]

Hi

AWS has a weakness in that it rate limits API calls in a blanket fashion. So if I run Scout Suite against our account while the platform is busy, it can break production.

I'm specifically hitting issues with autoscaling failing because AWS draws no distinction between the scanner making API calls on the CLI and their own autoscaling feature making API calls. Because the latter then fails to bring up the instance, the environment ends up in an indeterminate mess and the platform team have to scramble.

It would help if you had an option to run in single threaded mode to limit the amount of API calls being made. Perhaps go even further and add an ability to rate limit your calls from within the application, so that AWS didn't kick in and do so itself.

[x4v13r64]

Adding single threaded mode should be a quick fix.

Scout does handle rate limiting, but that won't stop the situation you describe from happening.

[x4v13r64]

The https://github.com/nccgroup/ScoutSuite/blob/master/ScoutSuite/providers/base/configs/threads.py file configures the amount of threads created when using the --thread-config option (1 to 5, default at 4). As you can see, setting --thread-config 1 will spin 2 threads, 1 for listing resources and another one for parsing them (parsing typically does not interact with the API).

Have you tried using this option to limit impact on API usage?

[museadmin]

Thanks for the feedback Xavier,

No I wasn't aware of that and will certainly make use of it. I only run the suite once every two weeks (a sprint) so will try it next time.

When you say that the suite handles rate limiting...

Most people handle this by implementing an incremental backoff, which if that is the case here, would not help as the problem is really within the way they are implementing autoscaling.

I guess the real answer is persuading the platform guys to implement something to handle the failure. The fact that they struggle to fix the broken instances suggests to me that they could do better with the way the system is architected.

Once again, thanks for the thread advice.

Brad

[x4v13r64]

Having another look at the code in https://github.com/nccgroup/ScoutSuite/blob/master/ScoutSuite/utils.py#L75 and how the is_throttled function is being used, it seems that calls that fail due to rate limiting are either 1) put back in the queue to be re-requested later or 2) simply raised as an exception.

This implementation doesn't help handle rate limiting (as no backoff is actually implemented), nor in all cases manage to make the request.

In addition, this is currently only implemented for AWS. was implemented but not in the refactoring.

TODO:

• Actually handle the APIs rate limiting Scout
• Do this for AWS/GCP/Azure

[museadmin]

Thanks for looking at this guys. Can I add these comments to this ticket because I think there is a case for going further with a remediation:

E.g. I see this error even when running with --max-workers 1

2019-06-27 09:12:18 MAC2802.local scout[5467] ERROR ec2.py L127: Failed to describe EC2 snapshot attributes: An error occurred (RequestLimitExceeded) when calling the DescribeSnapshotAttribute operation (reached max retries: 4): Request limit exceeded.

AWS have blighted pretty much all of the scanning tools I’ve evaluated, yours being one of the best, because of a number of reasons:

1. The primary issue is that the rate limiting, as applied by AWS, is dumb. It draws no distinction between scout and an instance that is using the API’s because it is autoscaling for example.
   This means that the production platform can, and does, break when running scout. This is particularly a problem for my security conscious client who has implemented a legion of objects making API calls that are constantly monitoring the integrity of the platform.
2. AWS exacerbate the issue further by recommending that we implement a retry mechanism of some kind in order to ram the API call through anyway, thereby making the situation far worse for the struggling autoscaling instance mentioned above!
3. I think without exception, all of the tutorials and examples on using the AWS CLI prioritise network bandwidth over API rate limiting. Which is crazy, because nobody maintains a cloud platform via a dial up modem…

If I can give you an example, I created two different versions of a script to check all of the volumes in my account to ensure that they are encrypted.

The first script took a conventional approach I found in Cloud Security Suite:

1. Call describe-instances with a filter to return just the volume ids. (Cuts down the size of the data returned to save network bandwidth).
2. Iterate over the list of volume ids and use it to parameterise a query of describe-volumes to see if the Encrypted field was set to True for each volume.

The script took 8.5 minutes to run and made hundreds of API calls to describe-volumes.

The second script took an approach that was optimised for rate limiting, rather than network bandwidth:

1. Call describe-instances with no filter
2. Call describe-volumes with no filter
3. Iterate over the data returned to perform the same check for Encrypted=True locally on my laptop.
   This script ran in 37 seconds and made only two API calls…

Please understand, that avoiding rate limiting in this fashion is not just a nice to have. It is the difference between your scanner being usable or not on platforms that are making heavy use of the APIs already.

This seems to be a problem across the board, not just for scout. I think this is in some measure attributable to AWS providing so much in the way of generous filtering options and also because they recommend implementing a retry mechanism when tools hit rate limiting.

So, this feature request is for you to also optimise your code for API rate limiting as described above and removing the reliance on a retry mechanism to avoid traumatising the user’s platform!

Hope that all makes sense.

Bradley

[x4v13r64]

Thanks for this @museadmin. We've indeed been seeing this happen much more these last few weeks so we should put this higher up the stack.

The primary issue is that the rate limiting, as applied by AWS, is dumb.

Indeed, the API call quota is per-account (not per-session, per-service, etc.) so if you're running Scout against an account with a lot of ops going on it doesn't play nice.

If I can give you an example, I created two different versions of a script to check all of the volumes in my account to ensure that they are encrypted.

Agreed that for many API calls we can likely "optimize" API call count (without caring of network bandwidth).

Will have to figure out a methodology & how to test though. If you have access to an account that triggers this and could try out a dev branch against it would be much appreciated.

[museadmin]

Hi J4v,

Yes, I have an account that is consistently hitting this issue. Probably best to take this offline if you want me to test the changes. You can contact me here - bradley.atkins (at) bjss.com

[thommor]

We are also running into this issue alot with accounts that are not that large. It would be useful for this to be looked at asap. @j4v Let me know if you need us to help out with this

[x4v13r64]

Closing this issue as the exponential backoff has been merged to develop in https://github.com/nccgroup/ScoutSuite/pull/461. This is currently AWS-specific, will add handling for other providers if we encounter this issue too.

The second element discussed here (rate limiting) is tracked in https://github.com/nccgroup/ScoutSuite/issues/492. @museadmin is working on implementing this natively in boto3 and we'll look into supporting this cross-provider.

This approach (rate limiting + exponential backoff) should hopefully allow handling most cases.