Boto3: New region ap-east-1 causes issues in describe_instances

[Timvissers]

We have ec2 instances in multiple regions. We have a scripts that fetches all our instances in all regions.

`
ec2client = boto3.client('ec2', region_name='eu-west-1')
response = ec2client.describe_regions()

for region in response['Regions']:
ec2client = boto3.client('ec2', region_name=region['RegionName'])
response = ec2client.describe_instances()
`

This fails since the new region in Hong Kong is open: https://aws.amazon.com/blogs/aws/now-open-aws-asia-pacific-hong-kong-region/

Note that we do not have the new region enabled. describe_regions includes it however. The documentation says 'regions available to you'.

Error message:

botocore.hooks: DEBUG: Event before-call.ec2.DescribeInstances: calling handler
botocore.endpoint: DEBUG: Making request for OperationModel(name=DescribeInstances) with params: {'body': {'Action': u'DescribeInstances', 'Version': u'2016-11-15'}, 'url': u'https://ec2.ap-east-1.amazonaws.com/', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.9.136 Python/2.7.12 Linux/4.4.0-146-generic Botocore/1.12.136'}, 'context': {'auth_type': None, 'client_region': 'ap-east-1', 'has_streaming_input': False, 'client_config': }, 'query_string': '', 'url_path': '/', 'method': u'POST'}
botocore.hooks: DEBUG: Event request-created.ec2.DescribeInstances: calling handler >
botocore.hooks: DEBUG: Event choose-signer.ec2.DescribeInstances: calling handler
botocore.auth: DEBUG: Calculating signature using v4 auth.
botocore.auth: DEBUG: CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:ec2.ap-east-1.amazonaws.com
x-amz-date:20190426T090746Z

content-type;host;x-amz-date
6171eb09865e32b0602af0f7957e26573a51f53caaedff02ff88883cb0275885
botocore.auth: DEBUG: StringToSign:
AWS4-HMAC-SHA256
20190426T090746Z
20190426/ap-east-1/ec2/aws4_request
4dba02c467b70e85d9856c14730f7e21bd85df96af3df7a495901cc429b79a8b
botocore.auth: DEBUG: Signature:
41370e62e65a0d3178d4e7990a89faedc26bbbbecc392204819aace2597f15ad
botocore.endpoint: DEBUG: Sending http request: urllib3.util.retry: DEBUG: Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): ec2.ap-east-1.amazonaws.com:443
urllib3.connectionpool: DEBUG: https://ec2.ap-east-1.amazonaws.com:443 "POST / HTTP/1.1" 401 None
botocore.parsers: DEBUG: Response headers: {'Transfer-Encoding': 'chunked', 'Date': 'Fri, 26 Apr 2019 09:07:47 GMT', 'Server': 'AmazonEC2'}
botocore.parsers: DEBUG: Response body:

AuthFailureAWS was not able to validate the provided access credentials9b981b0f-930f-48e7-baab-8914d0834889

[dgreene-r7]

I'm running into similar issues across all services where boto3.session.Session().get_available_regions returns all regions regardless of whether the region has been enabled.

Are there plans to expose a get_enabled_regions function or at least a param for get_available_regions so we can indicate that we only want enabled regions?

[swetashre]

@timv2 - Thank you for reporting the behavior. Now EC2 has rolled back the change that included ap-east-1 region in the output of DescribeRegions API call from all regions except ap-east-1. So now if you run the script you should not get the error.

But if you are still getting the error you can reach out to our service team on their AWS EC2 forums.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[SpenGietz]

Is there a potential fix for this when using boto3.session.Session().get_available_regions? I'm running into errors with services that don't have a describe_regions API call.

[j4v]

We're (https://github.com/nccgroup/ScoutSuite/issues/381) also getting this for a number of services/calls (CloudFormation, CloudTrail, Config, Lambda, DynamoDB, EC2, ELB, RDS, EMR, Redshift, SNS & SQS). Is this already on your radar?

Are there plans to expose a get_enabled_regions function or at least a param for get_available_regions so we can indicate that we only want enabled regions?

This would also be a good alternative...

@SpenGietz did you find a workaround?

[j4v]

For reference, the following are failing due to this issue:

• CloudFormation ListStacks
• CloudWatch DescribeAlarms
• CloutTrail DescribeTrails
• Config DescribeConfigRules
• Config DescribeConfigurationRecorderStatus
• Config DescribeConfigurationRecorders
• DynamoDB ListTables
• EC2 DescribeFlowLogs
• EC2 DescribeImages
• EC2 DescribeSnapshots
• EC2 DescribeVolumes
• EC2 DescribeVpcPeeringConnections
• EC2 DescribeVpcs
• ELB DescribeLoadBalancers
• EMR ListClusters
• ElastiCache DescribeCacheSecurityGroups
• Elasticache DescribeCacheParameterGroups
• KMS ListKeys
• Lambda ListFunctions
• RDS DBParameterGroups
• RDS DescribeDBSecurityGroups
• Redshift DescribeClusterParameterGroups
• Redshift DescribeClusterSecurityGroups
• SNS ListTopics
• SQS ListQueues

There may be other instances, these are just the ones failing in https://github.com/nccgroup/ScoutSuite

[SpenGietz]

I've not found a good workaround myself currently @j4v

[swetashre]

@SpenGietz - We have a tracking open issue for this problem. Please suggest your ideas on the GitHub issue #2022

[Timvissers]

My workaround was

ec2client = boto3.client('ec2', region_name='eu-west-1')
response = ec2client.describe_regions()
for region in response['Regions']:
    if region['RegionName'] == 'ap-east-1': https://github.com/boto/boto3/issues/1943
        continue
    ec2client = boto3.client('ec2', region_name=region['RegionName'])
    response = ec2client.describe_instances()

:)

[raajheshkannaa]

I'm doing this for 'me-south-1' now with my lambda function along with the 'ap-east-1'. What's the scalable way to get this accomplished.

[Akash021]

` session = boto3.Session(aws_access_key_id=accessKeyID, aws_secret_access_key=secretKeyID)

availableRegions = session.get_available_regions('ec2')
for region in availableRegions:
    client = session.client('ec2', region_name=region)
       try: 
           response = client.describe_instances()    
       except botocore.exceptions.ClientError as e:
            if e.response['Error']['Code'] == 'AuthFailure':
                   print("Region {region} seems to be disabled for this account, skipping")
                   continue
            else:
                   raise e    `

[infa-dgowran]

For reference, the following are failing due to this issue:

• CloudFormation ListStacks
• CloudWatch DescribeAlarms
• CloutTrail DescribeTrails
• Config DescribeConfigRules
• Config DescribeConfigurationRecorderStatus
• Config DescribeConfigurationRecorders
• DynamoDB ListTables
• EC2 DescribeFlowLogs
• EC2 DescribeImages
• EC2 DescribeSnapshots
• EC2 DescribeVolumes
• EC2 DescribeVpcPeeringConnections
• EC2 DescribeVpcs
• ELB DescribeLoadBalancers
• EMR ListClusters
• ElastiCache DescribeCacheSecurityGroups
• Elasticache DescribeCacheParameterGroups
• KMS ListKeys
• Lambda ListFunctions
• RDS DBParameterGroups
• RDS DescribeDBSecurityGroups
• Redshift DescribeClusterParameterGroups
• Redshift DescribeClusterSecurityGroups
• SNS ListTopics
• SQS ListQueues

There may be other instances, these are just the ones failing in https://github.com/nccgroup/ScoutSuite

plus EKS ListClusters as not availablein us-west-1

[thejazzid]

here is AWS solution https://aws.amazon.com/premiumsupport/knowledge-center/iam-validate-access-credentials/