Boto3: New region ap-east-1 causes issues with multiple services

[j4v]

Opening a new issue (formerly https://github.com/boto/boto3/issues/1943) as the new region has an impact on multiple services.

The following calls are failing:

• CloudFormation ListStacks
• CloudWatch DescribeAlarms
• CloutTrail DescribeTrails
• Config DescribeConfigRules
• Config DescribeConfigurationRecorderStatus
• Config DescribeConfigurationRecorders
• DynamoDB ListTables
• EC2 DescribeFlowLogs
• EC2 DescribeImages
• EC2 DescribeSnapshots
• EC2 DescribeVolumes
• EC2 DescribeVpcPeeringConnections
• EC2 DescribeVpcs
• ELB DescribeLoadBalancers
• EMR ListClusters
• ElastiCache DescribeCacheSecurityGroups
• Elasticache DescribeCacheParameterGroups
• KMS ListKeys
• Lambda ListFunctions
• RDS DBParameterGroups
• RDS DescribeDBSecurityGroups
• Redshift DescribeClusterParameterGroups
• Redshift DescribeClusterSecurityGroups
• SNS ListTopics
• SQS ListQueues

This is caused by the ap-east-1 region being returned by get_available_regions, even though it is not enabled.

There may be other instances, these are just the ones failing in https://github.com/nccgroup/ScoutSuite.

[kyleknap]

@j4v I agree we need to add a workaround for users that want to programmatically make API calls to all available regions but have not opted into every region (e.g. ap-east-1). I think it would make sense to have an opt-out flag in the get_available_regions method call that removes any opt-in regions from the returned list. So something like:
python import boto3.session # Does not list ap-east-1 because it is opt-in boto3.session.Session().get_available_regions('s3', allow_opt_in_regions=False) # However if you do not specify allow_opt_in_regions, you get all of the regions boto3.session.Session().get_available_regions('s3')
I think this direction makes sense because:

1) We cannot know if a specific account has opted into a region. The get_available_regions() does not make any HTTP request and there is not even an API call to list all of the regions that is available to a user. So we cannot add the functionality of excluding a region that is opt-in without a new parameter or new method on the session.

2) There is already a allow_non_regional parameter to exclude regions that are just special endpoints. So there is a precedence for having a filter parameter in the method.

3) We would make the allow_opt_in_regions default to True because it would be a breaking change for users that have already opted into ap-east-1 and expect the get_available_regions() calls to return it to start filtering it out.

The only issue is that we do not have the upstream metadata yet in our endpoints.json to programmatically account for each new opt-in region. So we will have to go get that added in order to programmatically account for any new opt-in region.

Let us know what you think.

[j4v]

Considering the limitations you're dealing with I think it's a good option.

Another thing that comes to mind is that the error responses we're getting when hitting this issue don't indicate what's actually happening (just states AuthFailure AWS was not able to validate the provided access credentials). If they did then it would be possible for us to filter these out.

[SpenGietz]

@kyleknap I like that idea a lot, it would be really helpful for my use case.

It's probably on the AWS side of development and not the boto side, but it would be nice if we could determine whether opt-in regions were enabled or not programmatically as well without the need for new APIs or permissions. Maybe something like STS GetCallerIdentity to a disabled opt-in region returns "region disabled" or something, rather than a somewhat-cryptic error message. I'm not sure the best route for that, more-so just throwing out ideas.

[j4v]

@kyleknap do you have an ETA for a fix?

[aph3rson]

FYI, it looks like there's been a recent (last month or so?) update to this. Per Managing AWS Regions:

Describing Your Regions Using the AWS CLI

Use the describe-regions command to describe the Regions available for your account, whether they are enabled or disabled.
aws ec2 describe-regions --all-regions

If the Region is enabled by default, the output includes the following:
"OptInStatus": "opt-in-not-required"

If the Region is not enabled, the output includes the following:
"OptInStatus": "not-opted-in"

After an opt-in Region is enabled, the output includes the following:
"OptInStatus": "opted-in"

Ergo, as the shape of DescribeRegions has changed, this should be something that can be detected when creating the boto session. Note that I'm here from nccgroup/ScoutSuite#381, so I'm not sure if it would be logic that would be in a boto client, or in boto/botocore.

[aph3rson]

@kyleknap Any updates on this?

[j4v]

Thanks @aph3rson, this allowed implementing a fix. This only works because all opt-in regions are supported in EC2 though, so we can leverage that endpoint to validate regions for other services.

[SpenGietz]

It's too bad that the ec2:DescribeRegions permission is required to find out if regions are enabled or not for the entire account

Also, ignoring how impossibly unlikely it is (I hope), what if a region comes out that doesn't support EC2?

I'd really like AWS to just implement real, consistent error messages that tell you if your access is denied because the region is disabled

[ViktorOrda]

@j4v @kyleknap
I've accidentally discovered that issue is not reproduced in older boto3 versions.
The next code

import boto3

session = boto3.Session()
regions = session.get_available_regions('rds')

returns disabled region 'ap-east-1' as well as all other regions when using 1.10.20 version.
But I've launched script before boto3 upgrade (1.7.71 version) and disabled regions were not returned by get_available_regions() function

[TheHumanGiraffe]

It's not optimal, but if you're looking for a workaround in the meantime and you know which service you want a client for, you can get a boto3 client and do

regions = client.describe_regions()['Regions']
regions = [region['RegionName'] for region in regions]

This doesn't return the opt in regions by default it seems.

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. Because it has been longer than one year since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure, or if the issue is already closed, please feel free to reopen it.

[j4v]

If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure

ping

[getsec]

I'm currently having this issue trying to run ec2:DescribeInstances on ap-east-1

for i in region_list:
         ec2 = boto3.client('ec2', region_name=i)
        data = ec2.describe_instances()

# bad_script_name.py
   > enumerating in ap-northeast-2
   > enumerating in ap-northeast-1
   > enumerating in sa-east-1
   > enumerating in ca-central-1
   > enumerating in ap-east-1
   > error with ap-east-1
   An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials


ClientError: An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials

Really would like some support with this please!

[aph3rson]

@getsec where are you getting your region list from? Botocore's get_available_regions method uses a list of regions baked-in to botocore.

I mentioned a little bit above, calling EC2's DescribeRegions method will give you a list of _enabled_ regions for your account instead (and calls the EC2 API endpoint to do so), assuming you have permissions to call DescribeRegions.

[getsec]

@aph3rson Ahh, I am using the describe_regions API call. Issues I'm having with this is we're running workloads in regions that I'm getting the auth errors in.

example:

def enum_regions():
    region_list = []

    ec2 = boto3.client('ec2')
    regions = ec2.describe_regions()
    for i in regions["Regions"]:
        region_list.append(i["RegionName"])
    return region_list