Scoutsuite: Recent S3 region selection change does not account for deactivated regions

[chris-codaio]

Recent changes (https://github.com/nccgroup/ScoutSuite/pull/896) to the AWS S3 region selection code do not account for regions that are deactivated in a given account (e.g., ap-east-1 is an optional region), nor does it account for the --exclude-regions flag.

• Console output using the --debug argument.

  $ pipenv run python ./tools/scout.py aws --profile root --exceptions ./exceptions.json --ruleset ./config/scout2-rulesets/default.json --report-dir ./scout-reports --force --debug
  2020-11-03 11:30:28 Christopher-Eck-MacBook scout[30050] INFO Launching Scout
  2020-11-03 11:30:28 Christopher-Eck-MacBook scout[30050] INFO Authenticating to cloud provider
  2020-11-03 11:30:30 Christopher-Eck-MacBook scout[30050] INFO Gathering data from APIs
  2020-11-03 11:30:30 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the ACM service
  2020-11-03 11:30:30 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the Lambda service
  2020-11-03 11:30:31 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the CloudFormation service
  2020-11-03 11:30:31 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the CloudTrail service
  2020-11-03 11:30:31 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the CloudWatch service
  2020-11-03 11:30:32 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the Config service
  2020-11-03 11:30:32 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the Direct Connect service
  2020-11-03 11:30:33 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the DynamoDB service
  2020-11-03 11:30:33 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the EC2 service
  2020-11-03 11:30:34 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the EFS service
  2020-11-03 11:30:34 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the ElastiCache service
  2020-11-03 11:30:34 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the ELB service
  2020-11-03 11:30:35 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the ELBv2 service
  2020-11-03 11:30:35 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the EMR service
  2020-11-03 11:30:36 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the IAM service
  2020-11-03 11:30:36 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the KMS service
  2020-11-03 11:30:36 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the RDS service
  2020-11-03 11:30:36 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the RedShift service
  2020-11-03 11:30:37 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the Route53 service
  2020-11-03 11:30:37 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the S3 service
  2020-11-03 11:30:38 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the SES service
  2020-11-03 11:30:39 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the SNS service
  2020-11-03 11:30:39 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the SQS service
  2020-11-03 11:30:40 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the VPC service
  2020-11-03 11:30:40 Christopher-Eck-MacBook scout[30050] INFO Fetching resources for the Secrets Manager service
  2020-11-03 11:31:23 Christopher-Eck-MacBook scout[30050] INFO Hitting API rate limiting, will retry in 15s
  2020-11-03 11:31:31 Christopher-Eck-MacBook scout[30050] ERROR Failed to list buckets: An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.
  2020-11-03 11:31:46 Christopher-Eck-MacBook scout[30050] INFO Hitting API rate limiting, will retry in 15s
  2020-11-03 11:32:02 Christopher-Eck-MacBook scout[30050] INFO Running pre-processing engine
  2020-11-03 11:32:02 Christopher-Eck-MacBook scout[30050] INFO Running rule engine
  ...

• Errors json

  [
  {
  "additional_details": null,
  "exception": "Failed to list buckets: An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.",
  "file": null,
  "line": null,
  "traceback": "None"
  }
  ]

[x4v13r64]

Thanks, will review ASAP.

[x4v13r64]

@chrisleck can you provide more details around your setup / the issue? I'm not able to reproduce.

do not account for regions that are deactivated in a given account (e.g., ap-east-1 is an optional region)

I'm running this against an account with the optional regions deactivated and am not running into any issues.

nor does it account for the --exclude-regions flag

How so? I've played around running with --exclude-regions without any issues.

@mohit614 you're running into this too?

[chris-codaio]

Regarding the --exclude-regions flag, this code makes it appear as if we're just going to blindly use the set of regions reported by S3, without applying the filter? https://github.com/nccgroup/ScoutSuite/blob/master/ScoutSuite/providers/aws/facade/s3.py#L20

[chris-codaio]

I've tried narrowing down the error case, and can't quite figure out what is causing this. We are using STS tokens with our identity provider (Okta), but that shouldn't cause errors for only some regions.

The AWS CLI works fine with all of the regions ScoutSuite is complaining about - for example AWS_PROFILE=<profile name> aws s3api list-buckets --region ap-southeast-1. So this definitely seems like something specific to the way ScoutSuite is using boto3 in a way different from the AWS CLI, but it isn't clear what that might be.

[chris-codaio]

Looking at the code from above, it's really unclear to me why a list-buckets failure in a single region is counted as a failure for the tool run as a whole (non-zero exit code). Seems like we're just looking for any region that can report back a list and we're happy if we have to swallow a few errors along the way?

[chris-codaio]

Ah - I see at least one bug in the code.

if not buckets:
                if exception:
                    print_exception(f'F1 Failed to list buckets: {exception}')
                return []

This is assuming that if we got some errors, then a success, that the success must carry a non-empty array. In my case, I have no buckets in the accounts that are causing issues, so the bucket list is correctly [] and we do have an exception from one of the disabled regions.

[x4v13r64]

Regarding the --exclude-regions flag, this code makes it appear as if we're just going to blindly use the set of regions reported by S3, without applying the filter?

Not exactly, it will take the regions specified by --regions if any, otherwise it will ask S3 for the available/enabled regions. S3 is a bit of a special case in terms of excluded regions, since you can poll any region for the same results.

--excluded-regions isn't being taken into account here as it would be tricky to implement.

So this definitely seems like something specific to the way ScoutSuite is using boto3 in a way different from the AWS CLI, but it isn't clear what that might be.

Scout should be making the same calls as the CLI. You are getting an InvalidToken response though, which doesn't seem related to the permissions / regions.

Looking at the code from above, it's really unclear to me why a list-buckets failure in a single region is counted as a failure for the tool run as a whole (non-zero exit code).

It's not...

Seems like we're just looking for any region that can report back a list and we're happy if we have to swallow a few errors along the way?

Yes, if you look at the comment and related issue it details the rationale for how it's implemented. Again, S3 behaves differently to other services as you can poll any region for the same config, and things get tricky when you have IAM policies denying requests to specific regions.

This is assuming that if we got some errors, then a success, that the success must carry a non-empty array. In my case, I have no buckets in the accounts that are causing issues, so the bucket list is correctly [] and we do have an exception from one of the disabled regions.

No - the error is only displayed if there was an exception raised. If there was a successful request, even if it returned an empty array, no error will be outputted.

[x4v13r64]

I'm not sure there's an actual bug being reported here.

Granted --exclude-regions is not being taken into account, but that's due to the "special case" that is S3 and the difficulty of implementing due to the architecture.

[chris-codaio]

Let me walk through the broken case to help clarify what I think is going on.

1. We retrieve a list of regions on line 20.
2. The first region we try on line 24 fails with an invalid token exception.
3. We capture that exception on line 26 and save it in the exception variable.
4. We loop around and try the next region and it is successful on line 24. The account has no buckets in it, so buckets is set to [].
5. We hit the break statement on line 28, leaving the loop.
6. The if not buckets check on line 29 is true as the array is empty. We go to line 30 and see that there is a value for exception so we execute print_exception.
7. print_exception stores the exception in global variable ERRORS_LIST.
8. Upon exiting the CLI, line 371 of __main__.py sees that the ERRORS_LIST array is not empty, so exits with a non-zero exit code of 200.

[chris-codaio]

One potential fix for this issue is to clear exception in the try/except block after a successful call to list_buckets().

[x4v13r64]

One potential fix for this issue is to clear exception in the try/except block after a successful call to list_buckets().

Sounds good to me, this is now in branch 5.10.2.

[chris-codaio]

What is the expected ship date for this fix? Is there a way to track upcoming releases?

[x4v13r64]

@chrisleck I've just released 5.10.2 (https://github.com/nccgroup/ScoutSuite/releases/tag/5.10.2). We don't tend to have hard deadlines for release dates, we usually push versions once we've completed a milestone (although we release x.y.z bug fix releases more often).

[chris-codaio]

Thanks @j4v ! Greatly appreciated.