Aws-cli: s3 ls with path give redirect error

[bedge]

aws s3 ls
2014-05-30 07:42:30 me.dev.xx.xx

that's OK

aws s3 ls s3://me.dev.xx.xx
A client error (PermanentRedirect) occurred when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

not OK

However this works works s3cmd:

%> s3cmd ls s3://me.dev.xx.xx
DIR ....

Everything is on us-west-1.

[quiver]

aws s3 ls s3://me.dev.xx.xx
A client error (PermanentRedirect) occurred when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future > requests to this endpoint.

Everything is on us-west-1.

If a bucket name contains '.'(dots) and the bucket region is not us-east-1(Virginia, default region), PermanentRedirect occurs.
Specifying region(--region us-west-1)should circumvent this error ::

$ aws s3 ls s3://me.dev.xx.xx --region us-west-1

[bedge]

Is the --region parameter necessary even if I have this in my ~/.aws/config:
[default]
region = us-west-1

[jamesls]

Can you confirm the location of the bucket? You can run:

aws s3api get-bucket-location --bucket <bucket-name>

The output should say:

{
    "LocationConstraint": "us-west-1"
}

If you also run the command with the --debug option, the raw XML response will also have the region name you need to use.

[bedge]

Just a side note, I am obfuscating the actual bucket name.

%> aws s3api get-bucket-location --bucket me.dev.xx.xx
{
"LocationConstraint": null
}

%> aws s3 ls s3://me.dev.xx.xx --region us-west-1

A client error (PermanentRedirect) occurred when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

I do get this from the get-bucket-location command with --debug:
Starting new HTTPS connection (1): s3-us-west-1.amazonaws.com

[jamesls]

A LocationConstraint of null means that the bucket is in the standard region. When working with this bucket you'll need to use --region us-east-1 instead of us-west-1. Let me know if you're still having issues.

[bedge]

thanks for the help.

Adding --region us-east-1 worked.
However, that's pretty counter-intuitive.

Note that s3cmd doesn't have this problem:
%> s3cmd ls s3://ni.q4.wirl.com/incoming/
2013-07-19 16:11 0 s3://ni.q4.wirl.com/incoming/

%> aws s3 ls s3://ni.q4.wirl.com/incoming/ --profile ct
A client error (PermanentRedirect) occurred when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

Also, the AWS console states:
"S3 does not require region selection."
And it displays the buckets with no region specifier.

So the only tool that has an issue with this is the aws cli. Seems like a bug IMO. Going by the AWS console statement that S3 does not require a region, then requiring a specific region, one that's not really even relevant, for this command seems broken.

--debug with no --region shows:
2014-06-04 17:26:21,787 - botocore.hooks - DEBUG - Event before-auth.s3: calling handler
2014-06-04 17:26:21,787 - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3-us-west-1.amazonaws.com/ni.q4.wirl.com?delimiter=/&prefix=incoming/

Adding --region changes to:
2014-06-04 17:32:04,334 - botocore.hooks - DEBUG - Event before-auth.s3: calling handler
2014-06-04 17:32:04,334 - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/ni.q4.wirl.com?delimiter=/&prefix=incoming/
2014-06-04 17:32:04,334 - botocore.handlers - DEBUG - Not changing URI, bucket is not DNS compatible: ni.q4.wirl.com

So, IOW, adding --region us-east-1 in fact removes "-us-west-1' from the generated URL.

In case you want more info, here's the full --debug output with actual bucket names:
%> aws s3 ls s3://ni.q4.wirl.com/incoming/ --profile ct --debug
2014-06-04 17:26:21,706 - awscli.clidriver - DEBUG - CLI version: aws-cli/1.3.11 Python/2.7.5 Darwin/13.2.0, botocore version: 0.45.0
2014-06-04 17:26:21,707 - botocore.service - DEBUG - Creating service object for: s3
2014-06-04 17:26:21,723 - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler
2014-06-04 17:26:21,723 - botocore.hooks - DEBUG - Event service-created: calling handler
2014-06-04 17:26:21,723 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2014-06-04 17:26:21,723 - botocore.hooks - DEBUG - Event creating-endpoint.s3: calling handler
2014-06-04 17:26:21,723 - botocore.credentials - INFO - Found credentials in config file.
2014-06-04 17:26:21,782 - botocore.service - DEBUG - Creating operation objects for: Service(s3)
2014-06-04 17:26:21,786 - botocore.operation - DEBUG - Operation:ListObjects called with kwargs: {'prefix': u'incoming/', 'bucket': u'ni.q4.wirl.com', 'delimiter': '/'}
2014-06-04 17:26:21,786 - botocore.operation - DEBUG - Creating parameter objects for: Operation:ListObjects
2014-06-04 17:26:21,787 - botocore.endpoint - DEBUG - Making request for Operation:ListObjects (verify_ssl=True) with params: {'headers': {}, 'uri_params': {u'Delimiter': '/', u'Bucket': u'ni.q4.wirl.com', u'Prefix': u'incoming/'}, 'payload': }
2014-06-04 17:26:21,787 - botocore.endpoint - DEBUG - Building URI for rest endpoint.
2014-06-04 17:26:21,787 - botocore.endpoint - DEBUG - Templated URI path: /{Bucket}
2014-06-04 17:26:21,787 - botocore.endpoint - DEBUG - Templated URI query_params: delimiter={Delimiter}&marker={Marker}&max-keys={MaxKeys}&prefix={Prefix}&encoding-type={EncodingType}
2014-06-04 17:26:21,787 - botocore.endpoint - DEBUG - Rendered path: /ni.q4.wirl.com
2014-06-04 17:26:21,787 - botocore.endpoint - DEBUG - Rendered query_params: delimiter=/&prefix=incoming/
2014-06-04 17:26:21,787 - botocore.hooks - DEBUG - Event before-auth.s3: calling handler
2014-06-04 17:26:21,787 - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3-us-west-1.amazonaws.com/ni.q4.wirl.com?delimiter=/&prefix=incoming/
2014-06-04 17:26:21,787 - botocore.handlers - DEBUG - Not changing URI, bucket is not DNS compatible: ni.q4.wirl.com
2014-06-04 17:26:21,788 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2014-06-04 17:26:21,788 - botocore.auth - DEBUG - HTTP request method: GET
2014-06-04 17:26:21,788 - botocore.auth - DEBUG - StringToSign:
GET

Thu, 05 Jun 2014 00:26:21 GMT
/ni.q4.wirl.com
2014-06-04 17:26:21,792 - botocore.endpoint - DEBUG - Sending http request: 2014-06-04 17:26:21,792 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): s3-us-west-1.amazonaws.com
2014-06-04 17:26:22,383 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /ni.q4.wirl.com?delimiter=/&prefix=incoming/ HTTP/1.1" 301 None
2014-06-04 17:26:22,383 - botocore.response - DEBUG - Response Headers:
x-amz-id-2: E5ukdcjx69q5cYqrdn+IXiub+JW+8E2rhe9Ch5sLU9FGpBRagyyilWTfBFMNxLW+
server: AmazonS3
transfer-encoding: chunked
x-amz-request-id: 7ABD040697740AEA
date: Thu, 05 Jun 2014 00:26:21 GMT
content-type: application/xml
2014-06-04 17:26:22,383 - botocore.response - DEBUG - Response Body:

PermanentRedirectThe bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.7ABD040697740AEAni.q4.wirl.comE5ukdcjx69q5cYqrdn+IXiub+JW+8E2rhe9Ch5sLU9FGpBRagyyilWTfBFMNxLW+s3.amazonaws.com
2014-06-04 17:26:22,384 - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjects: calling handler
2014-06-04 17:26:22,384 - botocore.retryhandler - DEBUG - No retry needed.
2014-06-04 17:26:22,384 - botocore.hooks - DEBUG - Event after-call.s3.ListObjects: calling handler
2014-06-04 17:26:22,384 - awscli.errorhandler - DEBUG - HTTP Response Code: 301
2014-06-04 17:26:22,384 - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/usr/local/Cellar/awscli/1.3.11/lib/python2.7/site-packages/awscli/clidriver.py", line 188, in main
return command_tableparsed_args.command
File "/usr/local/Cellar/awscli/1.3.11/lib/python2.7/site-packages/awscli/customizations/s3/s3.py", line 151, in call
remaining, parsed_globals)
File "/usr/local/Cellar/awscli/1.3.11/lib/python2.7/site-packages/awscli/customizations/s3/s3.py", line 262, in call
return self._do_command(parsed_args, parsed_globals)
File "/usr/local/Cellar/awscli/1.3.11/lib/python2.7/site-packages/awscli/customizations/s3/s3.py", line 359, in _do_command
self._list_all_objects(bucket, key)
File "/usr/local/Cellar/awscli/1.3.11/lib/python2.7/site-packages/awscli/customizations/s3/s3.py", line 366, in _list_all_objects
for _, response_data in iterator:
File "/usr/local/Cellar/awscli/1.3.11/libexec/lib/python2.7/site-packages/botocore-0.45.0-py2.7.egg/botocore/paginate.py", line 150, in __iter__
current_kwargs)
File "/usr/local/Cellar/awscli/1.3.11/libexec/lib/python2.7/site-packages/botocore-0.45.0-py2.7.egg/botocore/operation.py", line 74, in call
parsed=response[1])
File "/usr/local/Cellar/awscli/1.3.11/libexec/lib/python2.7/site-packages/botocore-0.45.0-py2.7.egg/botocore/session.py", line 637, in emit
return self._events.emit(event_name, **kwargs)
File "/usr/local/Cellar/awscli/1.3.11/libexec/lib/python2.7/site-packages/botocore-0.45.0-py2.7.egg/botocore/hooks.py", line 150, in emit
response = handler(kwargs)
File "/usr/local/Cellar/awscli/1.3.11/lib/python2.7/site-packages/awscli/errorhandler.py", line 75, in call
http_status_code=http_response.status_code)
ClientError: A client error (PermanentRedirect) occurred when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
2014-06-04 17:26:22,390 - awscli.clidriver - DEBUG - Exiting with rc 255

[disappeared1]

@jamesls

I had to remove my dots in my bucket because I could't sync with them even with specifying both region and source region option. see below? Is this a bug?

I obfuscated my bucket name by putting xxxxx.

root@util:~/buckets# aws s3api get-bucket-location --bucket xxxxxx.com.local
{
    "LocationConstraint": "us-west-2"
}

root@util:~/buckets# aws s3api get-bucket-location --bucket xxxxxx.com.local.daily
{
    "LocationConstraint": "us-west-1"
}

root@util:~/buckets# aws s3 sync s3://xxxxxx.com.local s3://xxxxxx.com.local.daily --delete --source-region us-west-2 --region us-west-1

delete failed: s3://xxxxxx.com.local.daily/assets/uploads/img/00fd4ee8ab8969d0d98690064481d62d.jpg A client error (PermanentRedirect) occurred when calling the DeleteObject operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint: xxxxxx.com.local.daily.s3-us-west-1.amazonaws.com
You can fix this issue by explicitly providing the correct region location using the --region argument, the AWS_DEFAULT_REGION environment variable, or the region variable in the AWS CLI configuration file.  You can get the bucket's location by running "aws s3api get-bucket-location --bucket BUCKET".
Completed 1 part(s) with ... file(s) remaining

[balthatrix]

I got the following when I ran
$ aws s3api get-bucket-location --bucket

None

BTW I'm trying to simply add files to s3 as custom maintenance pages for our site. When I run
$ aws s3 ls s3://
I get

A client error (PermanentRedirect) occurred when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
Any ideas?

EDIT: Here's the output with the --debug option, same command:
2015-01-15 10:33:46,360 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.5.3 Python/2.7.6 Darwin/14.0.0, botocore version: 0.66.0
2015-01-15 10:33:46,360 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler
2015-01-15 10:33:46,361 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler
2015-01-15 10:33:46,361 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler
2015-01-15 10:33:46,361 - MainThread - botocore.service - DEBUG - Creating service object for: s3
2015-01-15 10:33:46,376 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler
2015-01-15 10:33:46,377 - MainThread - botocore.hooks - DEBUG - Event service-created: calling handler
2015-01-15 10:33:46,378 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2015-01-15 10:33:46,379 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2015-01-15 10:33:46,379 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2015-01-15 10:33:46,379 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2015-01-15 10:33:46,384 - MainThread - botocore.service - DEBUG - Creating operation objects for: Service(s3)
2015-01-15 10:33:46,388 - MainThread - botocore.operation - DEBUG - Operation:ListObjects called with kwargs: {'prefix': '', 'bucket': u'thoughtstem.cms.dev', 'delimiter': '/'}
2015-01-15 10:33:46,389 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjects: calling handler
2015-01-15 10:33:46,389 - MainThread - botocore.endpoint - DEBUG - Making request for (verify_ssl=True) with params: {'query_string': {u'delimiter': '/', u'prefix': ''}, 'headers': {}, 'url_path': u'/thoughtstem.cms.dev', 'body': '', 'method': u'GET'}
2015-01-15 10:33:46,389 - MainThread - botocore.hooks - DEBUG - Event before-auth.s3: calling handler
2015-01-15 10:33:46,389 - MainThread - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3-us-west-1.amazonaws.com/thoughtstem.cms.dev?delimiter=%2F&prefix=
2015-01-15 10:33:46,389 - MainThread - botocore.handlers - DEBUG - Not changing URI, bucket is not DNS compatible: thoughtstem.cms.dev
2015-01-15 10:33:46,389 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2015-01-15 10:33:46,389 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2015-01-15 10:33:46,389 - MainThread - botocore.auth - DEBUG - StringToSign:
GET

Thu, 15 Jan 2015 18:33:46 GMT
/thoughtstem.cms.dev
2015-01-15 10:33:46,394 - MainThread - botocore.endpoint - DEBUG - Sending http request: 2015-01-15 10:33:46,394 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): s3-us-west-1.amazonaws.com
2015-01-15 10:33:46,622 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /thoughtstem.cms.dev?delimiter=%2F&prefix= HTTP/1.1" 301 None
2015-01-15 10:33:46,623 - MainThread - botocore.parsers - DEBUG - Response body:

PermanentRedirectThe bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.thoughtstem.cms.devs3.amazonaws.com83DBEBC97AD88B6F0pb8IJOEjl4uM8GNb/ioUvfdZG30sTIIS3a4/pBZt9Z6tZfSL6Fd0gDJXiFXPs45hosEPL1/+uU=
2015-01-15 10:33:46,624 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjects: calling handler
2015-01-15 10:33:46,624 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2015-01-15 10:33:46,624 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjects: calling handler
2015-01-15 10:33:46,624 - MainThread - awscli.errorhandler - DEBUG - HTTP Response Code: 301
2015-01-15 10:33:46,624 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/usr/local/Cellar/awscli/1.5.3/lib/python2.7/site-packages/awscli/clidriver.py", line 206, in main
return command_tableparsed_args.command
File "/usr/local/Cellar/awscli/1.5.3/lib/python2.7/site-packages/awscli/customizations/commands.py", line 184, in call
parsed_globals)
File "/usr/local/Cellar/awscli/1.5.3/lib/python2.7/site-packages/awscli/customizations/commands.py", line 181, in call
return self._run_main(parsed_args, parsed_globals)
File "/usr/local/Cellar/awscli/1.5.3/lib/python2.7/site-packages/awscli/customizations/s3/subcommands.py", line 251, in _run_main
self._list_all_objects(bucket, key, parsed_globals.page_size)
File "/usr/local/Cellar/awscli/1.5.3/lib/python2.7/site-packages/awscli/customizations/s3/subcommands.py", line 260, in _list_all_objects
for _, response_data in iterator:
File "/usr/local/Cellar/awscli/1.5.3/libexec/lib/python2.7/site-packages/botocore-0.66.0-py2.7.egg/botocore/paginate.py", line 69, in __iter__
response = self._make_request(current_kwargs)
File "/usr/local/Cellar/awscli/1.5.3/libexec/lib/python2.7/site-packages/botocore-0.66.0-py2.7.egg/botocore/paginate.py", line 386, in _make_request
return self._operation.call(self._endpoint, current_kwargs)
File "/usr/local/Cellar/awscli/1.5.3/libexec/lib/python2.7/site-packages/botocore-0.66.0-py2.7.egg/botocore/operation.py", line 93, in call
parsed=response[1])
File "/usr/local/Cellar/awscli/1.5.3/libexec/lib/python2.7/site-packages/botocore-0.66.0-py2.7.egg/botocore/session.py", line 712, in emit
return self._events.emit(event_name, **kwargs)
File "/usr/local/Cellar/awscli/1.5.3/libexec/lib/python2.7/site-packages/botocore-0.66.0-py2.7.egg/botocore/hooks.py", line 152, in emit
response = handler(kwargs)
File "/usr/local/Cellar/awscli/1.5.3/lib/python2.7/site-packages/awscli/errorhandler.py", line 70, in __call__
http_status_code=http_response.status_code)
ClientError: A client error (PermanentRedirect) occurred when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
2015-01-15 10:33:46,626 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

A client error (PermanentRedirect) occurred when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

[jamesls]

Can you add a --debug log output?

[HouCoder]

I've resolved this issue by upgrading to the latest aws-cli.

[styk-tv]

So that was a silly bug. Thanks for the fix. Just a quick question, what version would that translate to against Xenial repos?

awscli is already the newest version (1.11.13-1ubuntu1~16.04.0).

ubuntu@ip-10-99-101-18:/$ aws s3api get-bucket-location --bucket my-super-bucket
{
    "LocationConstraint": null
}

EDIT: removed package installed by pip still no go

aws-cli/1.11.61 Python/2.7.12 Linux/4.4.0-57-generic botocore/1.5.24