Boto3: Boto3 not listing the right regions for gov-cloud

[moali87]

First I create the session.

def aws_session(profile_name='default'):
    try:
        session = boto3.session.Session(profile_name=profile_name)
        return session
    except botocore.exceptions.ProfileNotFound:
        print(
            'Unable to create session with AWS.  '
            'Please make sure your {0} profile is created in ~/.aws/(credentials, config)'.format(profile_name)
        )
        sys.exit(1)

Second I use my gov-cloud credentials profile from ~/.aws/credentials

session = aws_session(profile_name='gov-cloud')

Third describe regions for ELB (This is where the error is).

session.get_available_regions('elb')
['ap-northeast-1', 'ap-northeast-2', 'ap-south-1', 'ap-southeast-1', 'ap-southeast-2', 'ca-central-1', 'eu-central-1', 'eu-west-1', 'eu-west-2', 'sa-east-1', 'us-east-1', 'us-east-2', 'us-west-1', 'us-wes
t-2']

And last when attempting to describe load balancers, I get invalid token because of the incorrect region name.

>>> elb_client.describe_load_balancers()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    elb_client.describe_load_balancers()
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 543, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the DescribeLoadBalancers operation: The security token included in the request is invalid.

When I specify the correct name, it does work, I get my load balancers. The get_available_regions('elb') is returning the incorrect region list.

[dstufft]

Can you try using aws-gov-cloud instead of gov-cloud? I believe that will solve this for you.

[dstufft]

Scratch that, I misread a bit of code here. Can you show me what the region you have configured in your profile is as well as debug logs? You can turn them on with boto3.set_stream_logger('botocore').

[moali87]

This is my profile.

╭─mo.ali at pm-10423 in /Users/mo.ali
╰─λ cat ~/.aws/config                                                                                                                                                                           0 < 09:03:32
[default]
region = us-east-1
output = json

[gov-cloud]
output = json
region = us-gov-west-1

Here's the output with the logger enabled. NOTE: I specify us-east-1 because of get_available_regions('elb')

>>> import boto3
>>> def aws_session(profile_name='default'):
...     try:
...         session = boto3.session.Session(profile_name=profile_name)
...         return session
...     except botocore.exceptions.ProfileNotFound:
...         print(
...             'Unable to create session with AWS.  '
...             'Please make sure your {0} profile is created in ~/.aws/(credentials, config)'.format(profile_name)
...         )
...         sys.exit(1)
...
...
...
>>> session = aws_session()
>>> gov_session = aws_session(profile_name='gov-cloud')
>>> boto3.set_stream_logger('botocore')
>>> gov_region = gov_session.get_available_regions('elb')

2017-02-14 09:05:25,160 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.6/site-packages/botocore/data/endpoints.json2017-02-14 09:05:25,174 botocore.loaders [DEBUG] Loading JSON file: /
usr/local/lib/python3.6/site-packages/botocore/data/elb/2012-06-01/service-2.json2017-02-14 09:05:25,198 botocore.hooks [DEBUG] Event service-data-loaded.elb: calling handler <function register_retries_fo
r_service at 0x10828b0d0>2017-02-14 09:05:25,204 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.6/site-packages/botocore/data/_retry.json2017-02-14 09:05:25,212 botocore.handlers [DEBU
G] Registering retry handlers for service: elb
>>> gov_region
['ap-northeast-1', 'ap-northeast-2', 'ap-south-1', 'ap-southeast-1', 'ap-southeast-2', 'ca-central-1', 'eu-central-1', 'eu-west-1', 'eu-west-2', 'sa-east-1', 'us-east-1', 'us-east-2', 'us-west-1', 'us-wes
t-2']
>>>
>>> gov_client = gov_session.client('elb')

Traceback (most recent call last):
  File "<input>", line 1, in <module>
    gov_client = gov_session.client('elb')
  File "/usr/local/lib/python3.6/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 836, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 70, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 224, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.6/site-packages/botocore/args.py", line 45, in get_client_args
    endpoint_url, is_secure, scoped_config)
  File "/usr/local/lib/python3.6/site-packages/botocore/args.py", line 103, in compute_client_args
    service_name, region_name, endpoint_url, is_secure)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 297, in resolve
    service_name, region_name)
  File "/usr/local/lib/python3.6/site-packages/botocore/regions.py", line 122, in construct_endpoint
    partition, service_name, region_name)
  File "/usr/local/lib/python3.6/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.
2017-02-14 09:07:22,781 botocore.credentials [DEBUG] Skipping environment variable credential check because profile name was explicitly set.2017-02-14 09:07:22,789 botocore.credentials [DEBUG] Looking for
 credentials via: assume-role2017-02-14 09:07:22,795 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file2017-02-14 09:07:22,803 botocore.credentials [INFO] Found credentials
in shared credentials file: ~/.aws/credentials2017-02-14 09:07:22,810 botocore.client [DEBUG] Registering retry handlers for service: elb2017-02-14 09:07:22,818 botocore.hooks [DEBUG] Event creating-clien
t-class.elb: calling handler <function add_generate_presigned_url at 0x108211268>
>>>
>>> gov_client = gov_session.client('elb', region_name='us-east-1')




2017-02-14 09:12:43,459 botocore.client [DEBUG] Registering retry handlers for service: elb2017-02-14 09:12:43,469 botocore.hooks [DEBUG] Event creating-client-class.elb: calling handler <function add_gen
erate_presigned_url at 0x108211268>2017-02-14 09:12:43,478 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None2017-02-14 09:12:43,489 botocore.endpoint [DEBUG] Se
tting elasticloadbalancing timeout as (60, 60)
>>> gov_client.describe_load_balancers()




2017-02-14 09:12:56,624 botocore.auth [DEBUG] CanonicalRequest:
POST
/

host:elasticloadbalancing.us-east-1.amazonaws.com
x-amz-date:20170214T141256Z

host;x-amz-date

2017-02-14 09:12:56,633 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20170214T141256Z
20170214/us-east-1/elasticloadbalancing/aws4_request

2017-02-14 09:12:56,644 botocore.auth [DEBUG] Signature:





2017-02-14 09:12:57,080 botocore.parsers [DEBUG] Response body:



Traceback (most recent call last):
  File "<input>", line 1, in <module>
    gov_client.describe_load_balancers()
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 543, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the DescribeLoadBalancers operation: The security token included in the request is invalid.
2017-02-14 09:12:56,585 botocore.hooks [DEBUG] Event before-parameter-build.elasticloadbalancing.DescribeLoadBalancers: calling handler <function generate_idempotent_uuid at 0x1082889d8>2017-02-14 09:12:5
6,594 botocore.endpoint [DEBUG] Making request for OperationModel(name=DescribeLoadBalancers) (verify_ssl=True) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'User-Agent
': 'Boto3/1.4.4 Python/3.6.0 Darwin/16.4.0 Botocore/1.5.8'}, 'body': {'Action': 'DescribeLoadBalancers', 'Version': '2012-06-01'}, 'url': 'https://elasticloadbalancing.us-east-1.amazonaws.com/', 'context'
: {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x1084e1208>, 'has_streaming_input': False}}2017-02-14 09:12:56,603 botocore.hooks [DEBUG] Event request-created.elastic
loadbalancing.DescribeLoadBalancers: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1083724e0>>2017-02-14 09:12:56,613 botocore.auth [DEBUG] Calculating
 signature using v4 auth.236069f72bf74f0c7ddff0a34b0defa8a21d1d6a897e588764e1b2ff6319f94a8d7d936c346101283705eeb77797e101d3a3eb7b656b6193d0921373bd64114e0df64458f4170ca45fc5b1e791dba36a22d47a6d4b23ed038fd
9ca804ea12b882017-02-14 09:12:56,652 botocore.endpoint [DEBUG] Sending http request: <PreparedRequest [POST]>2017-02-14 09:12:56,660 botocore.vendored.requests.packages.urllib3.connectionpool [INFO] Start
ing new HTTPS connection (1): elasticloadbalancing.us-east-1.amazonaws.com2017-02-14 09:12:57,064 botocore.vendored.requests.packages.urllib3.connectionpool [DEBUG] "POST / HTTP/1.1" 403 3222017-02-14 09:
12:57,073 botocore.parsers [DEBUG] Response headers: {'x-amzn-requestid': 'af04c74c-f2bf-11e6-866b-f343ad8432e6', 'content-type': 'text/xml', 'content-length': '322', 'date': 'Tue, 14 Feb 2017 14:12:56 GM
T'}b'<ErrorResponse xmlns="http://elasticloadbalancing.amazonaws.com/doc/2012-06-01/">\n  <Error>\n    <Type>Sender</Type>\n    <Code>InvalidClientTokenId</Code>\n    <Message>The security token included
in the request is invalid.</Message>\n  </Error>\n  <RequestId>af04c74c-f2bf-11e6-866b-f343ad8432e6</RequestId>\n</ErrorResponse>\n'2017-02-14 09:12:57,089 botocore.hooks [DEBUG] Event needs-retry.elastic
loadbalancing.DescribeLoadBalancers: calling handler <botocore.retryhandler.RetryHandler object at 0x1079249e8>2017-02-14 09:12:57,095 botocore.retryhandler [DEBUG] No retry needed.
>>>

[eimajenthat]

Stumbled across this while researching GovCloud support in boto3. I know this ticket's old, and the requester has likely either figured out the solution or given up, but someone else might stumble on this and be helped.

At any rate, it looks like you have to specify partition_name='aws-us-gov' in get_available_regions to get the GovCloud regions. From the docs: http://boto3.readthedocs.io/en/latest/reference/core/session.html#boto3.session.Session.get_available_regions

Right above that in the docs, there is a function get_available_partitions which will return the names of the partitions.