Aws-cli: Support for requester-pays buckets

[rclark]

If a bucket is marked as a requester-pays bucket, then GetObject requests must specify the following header:

x-amz-request-payer: requester

There's no mention anywhere of how I might set this header using aws s3 cp or aws s3api get-object. Is it supported? Is there maybe a workaround in some fashion similar to https://github.com/aws/aws-sdk-js/issues/100#issuecomment-16217625?

[johnmu]

Any idea when this will be supported?

[chutium]

ping again, @jamesls there is a patch for s3cmd http://arxiv.org/help/faq/bulk_data_s3_patch.txt, i tried to find similar code related to headers in awscli/customizations/s3 , but i got nothing.

[jamesls]

This is now available in the 1.7.16 release of the AWS CLI. There's the control plane operations for configuring request payers on a bucket, and the various operation on an S3 object now have a --request-payer option you can provide.

[jure]

How about some documentation about this feature? "--request-payer" is not an option, as far as I can tell, and I can't find what the actual option is.

[jamesls]

@jure It should be documented on the various operations that support --request-payer, for example: http://docs.aws.amazon.com/cli/latest/reference/s3api/get-object.html#options shows:

--request-payer (string) Confirms that the requester knows that she or he will be charged for the request. Bucket owners need not specify this parameter in their requests. Documentation on downloading objects from requester pays buckets can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html

[jure]

I see, thank you @jamesls. I was trying to use aws s3 (which doens't have that option) instead of aws s3api (which does).

[eparejatobes]

@jamesls just to confirm, one is supposed to do something like

aws s3api get-object --request-payer requester --bucket <bucket> --key <key> <outfile>

[willywutang]

I also am confused about how to use this despite the comments above.

[jamesls]

That's correct that you would specify --request-payer requester using a command such as aws s3api get-object.

[danielecook]

Why isn't this part of aws s3? It is very hard to find information on this fundamental feature.

[jdellithorpe]

It looks like aws s3 ls has the --request-payer option, but other subcommands like sync do not. According to my understanding, s3 uses s3api to execute its commands, so adding this functionality would simply be a matter of passing along the argument.

[jdellithorpe]

For anyone else wanting to use aws s3 sync on a "Requester Pays" bucket...

aws s3 sync src dest --request-payer

With following changes made on awscli-1.11.172:

diff --git a/awscli/customizations/s3/subcommands.py b/awscli/customizations/s3/subcommands.py
index 4637dd8..88d9d92 100644
--- a/awscli/customizations/s3/subcommands.py
+++ b/awscli/customizations/s3/subcommands.py
@@ -434,7 +434,8 @@ TRANSFER_ARGS = [DRYRUN, QUIET, INCLUDE, EXCLUDE, ACL,
                  WEBSITE_REDIRECT, CONTENT_TYPE, CACHE_CONTROL,
                  CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE,
                  EXPIRES, SOURCE_REGION, ONLY_SHOW_ERRORS, NO_PROGRESS,
-                 PAGE_SIZE, IGNORE_GLACIER_WARNINGS, FORCE_GLACIER_TRANSFER]
+                 PAGE_SIZE, IGNORE_GLACIER_WARNINGS, FORCE_GLACIER_TRANSFER,
+                 REQUEST_PAYER]


 def get_client(session, region, endpoint_url, verify, config=None):
diff --git a/awscli/customizations/s3/utils.py b/awscli/customizations/s3/utils.py
index 9d3e491..6f725a3 100644
--- a/awscli/customizations/s3/utils.py
+++ b/awscli/customizations/s3/utils.py
@@ -363,6 +363,7 @@ class BucketLister(object):
             kwargs['Prefix'] = prefix

         paginator = self._client.get_paginator('list_objects')
+        kwargs['RequestPayer'] = 'requester'
         pages = paginator.paginate(**kwargs)
         for page in pages:
             contents = page.get('Contents', [])
@@ -428,6 +429,7 @@ class RequestParamsMapper(object):
     @classmethod
     def map_get_object_params(cls, request_params, cli_params):
         """Map CLI params to GetObject request params"""
+        cls._set_request_payer_params(request_params, cli_params)
         cls._set_sse_c_request_params(request_params, cli_params)

     @classmethod
@@ -512,6 +514,11 @@ class RequestParamsMapper(object):
                          'read|readacl|writeacl|full')

     @classmethod
+    def _set_request_payer_params(cls, request_params, cli_params):
+        if cli_params.get('request_payer'):
+            request_params['RequestPayer'] = cli_params['request_payer']
+
+    @classmethod
     def _set_metadata_params(cls, request_params, cli_params):
         if cli_params.get('metadata'):
             request_params['Metadata'] = cli_params['metadata']

FYI this is hack, don't use long-term...

[yonist]

@jdellithorpe
I applied your patch and I get the following error when running the sync command:
"an error occurred (AccessDenied) when calling the CopyObject operation: Access Denied"

It works perfectly without request-payer enabled on the bucket.
Any ideas?

Thanks.

[maheshbc]

@jdellithorpe : I have aws cli version 1.14.36

aws --version
aws-cli/1.14.36 Python/2.7.13 Darwin/15.6.0 botocore/1.8.40

I tried s3 sync with requester pays option

aws --profile s3 sync --request-payer

It says Unknown options: --request-payer

Any idea why?

[JiaweiZhuang]

It says Unknown options: --request-payer

Any idea why?

I had the same issue before. Updating awscli fixed it. --request-payer is an option in 1.14.42.

$pip install --upgrade awscli
$aws --version
aws-cli/1.14.42 Python/3.6.2 Darwin/16.7.0 botocore/1.8.46

[kyleknap]

Yep. We just added support for --request-payer for the cp, mv, sync, and rm commands in 1.14.42. Upgrade to that version to start using it.

[maheshbc]

Hello Jiawei and Kyle,

Thanks a lot for your comments and clarifications.

I was able to download the forked branch and make it work towards the end
of the last week.

But it is good to know that it is now available in 1.14.42. This is
excellent news.

I will test it out today itself.

• Mahesh

On Tue, Feb 20, 2018 at 9:00 AM, Kyle Knapp [e-mail]
wrote:

Yep. We just added support for --request-payer for the cp, mv, sync, and
rm commands in 1.14.42. Upgrade to that version to start using it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-cli/issues/797#issuecomment-367044692, or mute
the thread
https://github.com/notifications/unsubscribe-auth/Ais95pC6ZrJqmSDeGnCQVj829q9rR11eks5tWvofgaJpZM4B_j0w
.

[Brett55]

I'm still not seeing the request-payer argument

aws-cli/1.16.10 Python/2.7.15 Darwin/16.7.0 botocore/1.12.0

[maheshbc]

Hello Brett,

Did you try the following command:

aws s3 sync src_s3_bucket dest_s3_bucket --request-payer requester

or

aws s3 ls s3://src_s3_bucket/object --request-payer requester

It has been working for me since version 1.14.x

Thanks,
Mahesh

On Mon, Sep 10, 2018 at 3:01 PM Brett Cullen [e-mail]
wrote:

I'm still not seeing the request-payer argument

aws-cli/1.16.10 Python/2.7.15 Darwin/16.7.0 botocore/1.12.0

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-cli/issues/797#issuecomment-420075792, or mute
the thread
https://github.com/notifications/unsubscribe-auth/Ais95oVN1E-iDTZS6eDAkAY5VfN1ELocks5uZuEqgaJpZM4B_j0w
.

[Brett55]

I can get aws s3 ls to work with the --request-payer argument but not aws s3 cp. aws s3 sync also works