Boto3: Strange behavior when trying to create an S3 bucket in us-east-1

[j3ffhubb]

Version info:
boto3 = 0.0.19 (from pip)
botocore = 1.0.0b1 (from pip)
Python = 2.7.9 (from Fedora 22)

I have no problem creating S3 buckets in us-west-1 or us-west-2, but specifying us-east-1 gives InvalidLocationConstraint

>>> conn = boto3.client("s3")
>>> conn.create_bucket(
    Bucket='testing123-blah-blah-blalalala', 
    CreateBucketConfiguration={'LocationConstraint': "us-east-1"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 200, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 255, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid

Also trying with a s3 client connected directly to us-east-1:

>>> conn = boto3.client("s3", region_name="us-east-1")
>>> conn.create_bucket(Bucket='testing123-blah-blah-blalalala', CreateBucketConfiguration={'LocationConstraint': "us-east-1"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 200, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 255, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid

When I do not specify a region, the bucket is created in us-east-1 (verified in the web console):

>>> conn.create_bucket(Bucket='testing123-blah-blah-blalalala')
{u'Location': '/testing123-blah-blah-blalalala', 'ResponseMetadata': {'HTTPStatusCode': 200, 'HostId': 'Qq2CqKPm4PhADUJ8X+ngxxEE3yRrsT3DOS4TefgzUpYBKzQO/62cQy20yPa1zs7l', 'RequestId': '06B36B1D8B1213C8'}}

...but the bucket returns None for LocationConstraint:

>>> conn.get_bucket_location(Bucket='testing123-blah-blah-blalalala')
{'LocationConstraint': None, 'ResponseMetadata': {'HTTPStatusCode': 200, 'HostId': 'nBGHNu30A/m/RymzuoHLiE2uWuzCsz3v1mcov324r2sMYX7ANq1jOIR0XphWiUIAxDwmxTOW8eA=', 'RequestId': '53A539CC4BCA08C4'}}

us-east-1 is listed as a valid region when I enumerate the regions:

>>> conn = boto3.client("ec2", region_name="us-east-1")
>>> [x["RegionName"] for x in conn.describe_regions()["Regions"]]
['eu-central-1', 'sa-east-1', 'ap-northeast-1', 'eu-west-1', 'us-east-1', 'us-west-1', 'us-west-2', 'ap-southeast-2', 'ap-southeast-1']

[jamesls]

This is an artifact of the underlying S3 API and the fact that it doesn't accept us-east-1. To create an S3 bucket in us-east-1 you can just not specify any CreateBucketConfiguration: s3.create_bucket(Bucket='foo').

Comparing our docs to the S3 API Reference docs (http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUT.html) it looks like we aren't documenting enum types, which I think would have helped here. We'll look into updating our docs.

[j3ffhubb]

I disagree that this shouldn't be fixed within Boto3/core, even if the REST API behaves this way. Having one special region that should be assumed default and raises a confusing Exception when explicitly named violates the Zen of Python

Explicit is better than implicit

Creating a boto3.client("ec2") will raise an exception for not specifying a region_name, so it's not as if the rest of Boto3 assumes us-east-1 as default.

Of course it's trivial for me just to add the below quirk now that I know of this behavior:

conn.create_bucket(
  BucketName='whatevs', 
  CreateBucketConfiguration=None if region == 'us-east-1' else {'LocationConstraint': region})

But why not add the quirk to Boto3's s3.create_bucket() instead of forcing every developer that ever creates S3 buckets to discover the behavior and re-implement the quirk in their own code?

I haven't looked at the Boto3/core code yet, but I would imagine that adding something like this would fix it and prevent numerous other people from running into the same issue:

if CreateBucketConfiguration is not None and \
CreateBucketConfiguration["LocationConstraint"] == "us-east-1":
    CreateBucketConfiguration = None

[jamesls]

Sorry about that. I wasn't suggesting we weren't open to any client side updates, just wanted to explain the current behavior. The other thing I forgot to mention is we don't actually have specific code for this method (or any method in general). We actually dynamically generate methods at runtime based on a shared JSON Description. So this essentially means that we surface the exact REST API in boto3 (and all the semantics of the underlying REST API). We can however add customizations if necessary.

My main hesitation here is that this isn't technically us-east-1, at least in the sense of how other AWS services refer to us-east-1. Our docs refer to this as "US Standard" and say that this can either be "facilities in Northern Virginia or the Pacific Northwest" (http://docs.aws.amazon.com/general/latest/gr/rande.html). This potentially false consistency could be confusing for customers as well.

We'll consider this though. Thanks for the feedback.

[devmage]

It's been a year. Have you figured out what to do about this problem? It still presents a poor developer experience, having to work around an API inconsistency in an interface (boto) that is ostensibly meant to hide API behaviors.

(he said, having spent a day debugging a problem caused by this bug)

[devmage]

On boto3 1.3.1 (and possibly boto but I haven't tested there), it's not as simple as passing CreateBucketConfiguration=None to the call to create_bucket, as that yields a validation error:

  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 262, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 528, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 581, in _convert_to_request_dict
    api_params, operation_model)
  File "/usr/local/lib/python2.7/dist-packages/botocore/validate.py", line 270, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
ParamValidationError: Parameter validation failed:
Invalid type for parameter CreateBucketConfiguration, value: None, type: <type 'NoneType'>, valid types: <type 'dict'>

Similarly, one cannot pass an empty dict CreateBucketConfiguration={}, as that raises:

ClientError: An error occurred (MalformedXML) when calling the CreateBucket operation: The XML you provided was not well-formed or did not validate against our published schema

It seems that you must omit the field. Similarly, if not using other CreateBucketConfiguration values, it seems that you must omit the entire parameter:

        if region == 'us-east-1':
            s3_conn.create_bucket(
                Bucket=bucket,
            )
        else:
            s3_conn.create_bucket(
                Bucket=bucket,
                CreateBucketConfiguration={'LocationConstraint': region},
            )

[edwardotis]

I sent amazon feedback to fix their documentation on the default region. I encourage others to do so. I can't believe this poor design decision on Amazon's part.
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUT.html

[tanzeelrana]

is this issue fixed?

[j3ffhubb]

@tanzeelrana:
I don't know if the client API still has this problem, and I don't have access to an AWS account anymore, but I wrote some code a few months ago that uses the higher level Bucket API, and it did not have this problem. I'm going to close the issue.

[ramyala]

why was this closed? This issue still exists.

[lvicentesanchez]

Yea, I can confirm this still happens. Why you close an issue when it has been fixed?

[alex-stout]

I've tried all of the suggestions above. Still having an issue.

[elveskevtar]

I have the same issue as well working on awslabs/cfncluster.

[symgryph]

I have the same issue. They need to update the docs to say 'if you are trying to create a bucket in us-1 tirefire-east-1 region, then you need to simply use no properties at all as it is a default

[stormm2138]

I am able to get all of these regions to work with the location constraint matching the region, except for us-east-1, which needs to use us-west-2 --

regions = ['ap-southeast-2', 'eu-central-1', 'eu-west-1' , 'us-east-1', 'us-east-2', 'us-west-1', 'us-west-2']
s = boto3.Session(profile_name)
c = s.client('s3', region)
if region == 'us-east-1':
    location_constraint = 'us-west-2'
else:
    location_constraint = region
c.create_bucket(Bucket=bucket_name, ACL='private', CreateBucketConfiguration={'LocationConstraint': location_constraint}) 

Using EU or defaulting to US standard as specified at: https://boto3.readthedocs.io/en/latest/reference/services/s3.html doesn't work

[arcivanov]

According to AWS support this will also work:

client = boto3.client('s3', 'us-east-1', config= Config(signature_version='s3v4'))
resp= client.create_bucket(Bucket="name")

Also, a bug has been logged with AWS and they will be fixing the documentation.

[AndriiIsiuk]

What works for me:

args = dict(Bucket = bucket_name)
if current_region != 'us-east-1':
args['CreateBucketConfiguration'] = dict(LocationConstraint = current_region)
bucket_response = s3_connection.create_bucket(**args)

[AdamShechter9]

Just experienced same error.
Upon reviewing boto3 documenation, I can see that 'us-east-1' is not a valid option for region.
very sneaky AWS. and bad user experience.

[AlexMapley]

Experienced this same issue with the golang aws-sdk and localstack, looks like it's a ubiquitous issue beyond just Boto. Doubt there's any workaround - on AWS to correct this.

[bingtimren]

Three and a half years passed and this strange, counter intuitive, confusing behavior is still there wasting so many people's life.

[RonWang]

Same bad experience here. This inconsistent behavior crossing different regions really needs an explanation.

[franchyze923]

Just encountered this myself. Thankful for this post..

[ianvonseggern1]

As many others have said, this inconsistency is a really terrible developer experience

[mikeSGman]

Like many before me, I'm certain, I ran into this (a year or so ago) and forgot about it after implementing a workaround. Unfortunately, I spent time tonight (on a different project) debugging the same issue - and then remembered: 'oh, yeah us-east-1 is a special s3 child!'.

[miron9]

Still the case in 2020 with boto3 1.11.11

[matheushent]

End of April 2020 and the issue still exists with boto3 1.12.44

Regrettable situation.

[andrewthetechie]

Just hit this issue right now and ended up having to research and come to this issue. This is not "well documented" or "an expected behavior". This needs to get fixed.

[sabelomaga]

Also experienced the same issue.

[adrianyorke]

Still happening July 2020, boto3 version 1.14.30 - good technical interview question.

[dannykopping]

Can confirm - still encountering this issue :(
FWIW, I'm using the aws cli and connecting to a Ceph RADOS cluster. Simply adding --region '' fixed this for me

[dazza-codes]

After about 5 years, this is still not even fixed in the documentation, e.g.

• https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-creating-buckets.html#create-an-amazon-s3-bucket
• boto3 1.16.44

heavy sigh - why is this issue closed?

The documentation fails to explain that us-east-1 will raise an exception if it is used explicitly. The sample code should include additional details of how to avoid that exception, thanks to snippets from this issue, e.g. it doesn't allow an empty dict nor a default None argument to the CreateBucketConfiguration parameter, it must be excluded entirely.

                if region_name == "us-east-1":
                    s3_client.create_bucket(Bucket=bucket_name)
                else:
                    location = {"LocationConstraint": region_name}
                    s3_client.create_bucket(
                        Bucket=bucket_name,
                        CreateBucketConfiguration=location
                    )