Aws-cli: `s3 cp <object>` requires ListBucket permissions

[mlogan]

The aws cli can't copy an object out of s3 unless the user has ListBucket permissions for the object's bucket.

Steps to reproduce:

1. Create an s3 bucket called test-bucket, or use an existing bucket.
2. Add an object to that bucket called test-object, or use an existing object.
3. Create a new user, and give it a policy like this:

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:GetObject" ], "Sid": "Stmt1378493159000", "Resource": [ "arn:aws:s3:::test-bucket/*", "arn:aws:s3:::test-bucket" ], "Effect": "Allow" } ] }

1. Install the new user's credentials in your environment.
2. Run:

$ aws --debug --region us-west-1 s3 cp s3://test-bucket/test-object /tmp/object

1. You should see the following output:

```
2013-09-06 11:58:01,700 - botocore.auth - DEBUG - HTTP request method: GET
2013-09-06 11:58:01,700 - botocore.auth - DEBUG - StringToSign:
GET

Fri, 06 Sep 2013 18:58:01 GMT
/test-bucket/
2013-09-06 11:58:01,709 - botocore.endpoint - DEBUG - Sending http request: 2013-09-06 11:58:01,789 - botocore.response - DEBUG - Response Body:

AccessDeniedAccess Denied132435C85AAD5EB03t89YS52j6/GJQZXU/xQ4r8nvRqwzp7Qe8d6HYwF072oVpthXcTxFKH5ITYe8oML
2013-09-06 11:58:01,790 - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjects: calling handler
2013-09-06 11:58:01,790 - botocore.retryhandler - DEBUG - No retry needed.
2013-09-06 11:58:01,790 - botocore.hooks - DEBUG - Event after-call.s3.ListObjects: calling handler
2013-09-06 11:58:01,790 - awscli.errorhandler - DEBUG - HTTP Response Code: 403
2013-09-06 11:58:01,790 - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 184, in main
return command_tableparsed_args.command
File "/Library/Python/2.7/site-packages/awscli/customizations/s3/s3.py", line 308, in __call__
return self.op_tableparsed_args.operation
File "/Library/Python/2.7/site-packages/awscli/customizations/s3/s3.py", line 397, in __call__
cmd_params.add_paths(parsed_args.paths)
File "/Library/Python/2.7/site-packages/awscli/customizations/s3/s3.py", line 614, in add_paths
self.check_src_path(paths)
File "/Library/Python/2.7/site-packages/awscli/customizations/s3/s3.py", line 677, in check_src_path
delimiter='/')
File "/Library/Python/2.7/site-packages/botocore/operation.py", line 82, in call
parsed=response[1])
File "/Library/Python/2.7/site-packages/botocore/session.py", line 550, in emit
return self._events.emit(event_name, kwargs)
File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 158, in emit
response = handler(kwargs)
File "/Library/Python/2.7/site-packages/awscli/errorhandler.py", line 50, in __call__
raise ClientError(msg)
ClientError: A client error (AccessDenied) occurred: Access Denied
2013-09-06 11:58:01,791 - awscli.clidriver - DEBUG - Exiting with rc 255
A client error (AccessDenied) occurred: Access Denied
```

It appears that the solution is to use the Verify=false option in the appropriate boto call (which I haven't had time to dive in and locate). See: http://stackoverflow.com/questions/11478752/with-the-boto-library-can-i-avoid-granting-list-permissions-on-a-base-bucket-in.

Thanks,
Mark

[wvdhaute]

+1

[yannh]

Can confirm this, issue appeared in 1.1.1, downgrading to 1.1.0 or adding listbucket to policy fixes this. On a sidenote, having --debug print out the entire file you're uploading in hexa is a pain :)

Thanks

[axolx]

+1. Glad to find this issue documented. Was starting to lose my mind...

[maxrabin]

+1

[jspiro]

+1

[ghost]

+1
now I downgraded awscli to 1.1.0 and 'aws s3 cp' worked well.

[jamesls]

Looking into this.

[jamesls]

I'd like to summarize the issues highlighted in this thread so far so that it's easier to talk about the fixes.

The cp command supports both upload _and_ download (cp s3://foo/foo /tmp/foo and s3 /tmp/foo s3://foo/foo).

For what I've found, from >= 1.1.0, _uploads_ use a ListObjects operation to verify the bucket exists before uploading files. _Downloads_ has always required this operation so OP's problem won't be fixed by downgrading.

The issue is also slightly more complicated by the fact that cp also takes a --recursive argument. For a _recursive_ download, we need the ListObjects operation, because we need a list of objects to download. For the recusive _upload_, we don't need the ListObjects operation.

So the plan is:

Command	Recursive	Operations
cp (upload)	No	PutObject
cp (upload)	Yes	PutObject
cp (download)	No	GetObject
cp (download)	Yes	GetObject, ListObjects