Aws-cli: Downloading S3 object encrypted using KMS keys fails with error: "Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4"

[maitreya1975]

I uploaded an object to S3 encrypted with a KMS managed key using the S3 Console.
I am using:

$ aws --version
aws-cli/1.6.2 Python/2.7.6 Darwin/13.4.0

When I tried to download the object using aws-cli, I got the following error:

aws s3 cp s3://xxxx/yyyy.jpg ./
download failed: s3://xxxx/yyyy..jpg to ./ngc6960_FinalPugh900.jpg A client error (InvalidArgument) occurred when calling the GetObject operation: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.

I get the same error when using the low level s3api:

685b3588202f:lego_iam maitreyr$ aws s3api get-object --bucket maitreyr-kms-test --key ngc6960_FinalPugh900.jpg ./ngc.jpg --debug
2014-11-16 10:58:31,128 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.6.2 Python/2.7.6 Darwin/13.4.0, botocore version: 0.73.0
2014-11-16 10:58:31,128 - MainThread - botocore.service - DEBUG - Creating service object for: s3
2014-11-16 10:58:31,156 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler <function signature_overrides at 0x108628398>
2014-11-16 10:58:31,156 - MainThread - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x108628230>
2014-11-16 10:58:31,159 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2014-11-16 10:58:31,160 - MainThread - botocore.service - DEBUG - Creating operation objects for: Service(s3)
2014-11-16 10:58:31,168 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler <function inject_assume_role_provider at 0x1089759b0>
2014-11-16 10:58:31,168 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler <function add_waiters at 0x108980758>
2014-11-16 10:58:31,173 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'bucket', <awscli.arguments.CLIArgument object at 0x108ab3550>), (u'if-match', <awscli.arguments.CLIArgument object at 0x108ab3590>), (u'if-modified-since', <awscli.arguments.CLIArgument object at 0x108ab35d0>), (u'if-none-match', <awscli.arguments.CLIArgument object at 0x108ab3610>), (u'if-unmodified-since', <awscli.arguments.CLIArgument object at 0x108ab3650>), (u'key', <awscli.arguments.CLIArgument object at 0x108ab3690>), (u'range', <awscli.arguments.CLIArgument object at 0x108ab36d0>), (u'response-cache-control', <awscli.arguments.CLIArgument object at 0x108ab3710>), (u'response-content-disposition', <awscli.arguments.CLIArgument object at 0x108ab3750>), (u'response-content-encoding', <awscli.arguments.CLIArgument object at 0x108ab37d0>), (u'response-content-language', <awscli.arguments.CLIArgument object at 0x108ab3810>), (u'response-content-type', <awscli.arguments.CLIArgument object at 0x108ab3850>), (u'response-expires', <awscli.arguments.CLIArgument object at 0x108ab3890>), (u'version-id', <awscli.arguments.CLIArgument object at 0x108ab38d0>), (u'sse-customer-algorithm', <awscli.arguments.CLIArgument object at 0x108ab3910>), (u'sse-customer-key', <awscli.arguments.CLIArgument object at 0x108ab3950>), (u'sse-customer-key-md5', <awscli.arguments.CLIArgument object at 0x108ab3990>), (u'ssekms-key-id', <awscli.arguments.CLIArgument object at 0x108ab39d0>)])
2014-11-16 10:58:31,173 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function add_streaming_output_arg at 0x108894320>
2014-11-16 10:58:31,175 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function add_cli_input_json at 0x10896db90>
2014-11-16 10:58:31,175 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function unify_paging_params at 0x1088971b8>
2014-11-16 10:58:31,175 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function add_generate_skeleton at 0x108975140>
2014-11-16 10:58:31,177 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.bucket: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,177 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.get-object: calling handler <awscli.argprocess.ParamShorthand object at 0x1089b1450>
2014-11-16 10:58:31,177 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2014-11-16 10:58:31,178 - MainThread - awscli.arguments - DEBUG - Unpacked value of "maitreyr-kms-test" for parameter "bucket": maitreyr-kms-test
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.if-match: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.if-modified-since: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.if-none-match: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.if-unmodified-since: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.key: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,178 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.get-object: calling handler <awscli.argprocess.ParamShorthand object at 0x1089b1450>
2014-11-16 10:58:31,178 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2014-11-16 10:58:31,178 - MainThread - awscli.arguments - DEBUG - Unpacked value of "ngc6960_FinalPugh900.jpg" for parameter "key": ngc6960_FinalPugh900.jpg
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.range: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-cache-control: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-content-disposition: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-content-encoding: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-content-language: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-content-type: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.response-expires: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.version-id: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.sse-customer-algorithm: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,179 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.sse-customer-key: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,180 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.sse-customer-key-md5: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,180 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.ssekms-key-id: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,180 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.get-object.outfile: calling handler <function uri_param at 0x108868f50>
2014-11-16 10:58:31,180 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2014-11-16 10:58:31,180 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2014-11-16 10:58:31,180 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2014-11-16 10:58:31,181 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2014-11-16 10:58:31,219 - MainThread - botocore.operation - DEBUG - Operation:GetObject called with kwargs: {u'Bucket': u'maitreyr-kms-test', u'Key': u'ngc6960_FinalPugh900.jpg'}
2014-11-16 10:58:31,221 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function sse_md5 at 0x108628050>
2014-11-16 10:58:31,222 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x108628410>
2014-11-16 10:58:31,222 - MainThread - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x108aac490> (verify_ssl=True) with params: {'query_string': {}, 'headers': {}, 'url_path': u'/maitreyr-kms-test/ngc6960_FinalPugh900.jpg', 'body': '', 'method': u'GET'}
2014-11-16 10:58:31,222 - MainThread - botocore.hooks - DEBUG - Event before-auth.s3: calling handler <function fix_s3_host at 0x108628140>
2014-11-16 10:58:31,222 - MainThread - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3-us-west-2.amazonaws.com/maitreyr-kms-test/ngc6960_FinalPugh900.jpg
2014-11-16 10:58:31,222 - MainThread - botocore.handlers - DEBUG - URI updated to: https://maitreyr-kms-test.s3.amazonaws.com/ngc6960_FinalPugh900.jpg
2014-11-16 10:58:31,222 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2014-11-16 10:58:31,223 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2014-11-16 10:58:31,223 - MainThread - botocore.auth - DEBUG - StringToSign:
GET


Sun, 16 Nov 2014 16:58:31 GMT
/maitreyr-kms-test/ngc6960_FinalPugh900.jpg
2014-11-16 10:58:31,230 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [GET]>
2014-11-16 10:58:31,231 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): maitreyr-kms-test.s3.amazonaws.com
2014-11-16 10:58:31,481 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /ngc6960_FinalPugh900.jpg HTTP/1.1" 307 None
2014-11-16 10:58:31,483 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): maitreyr-kms-test.s3-us-west-2.amazonaws.com
2014-11-16 10:58:32,419 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /ngc6960_FinalPugh900.jpg HTTP/1.1" 400 None
2014-11-16 10:58:32,423 - MainThread - botocore.parsers - DEBUG - Response headers:
{'connection': 'close',
 'content-type': 'application/xml',
 'date': 'Sun, 16 Nov 2014 16:58:17 GMT',
 'server': 'AmazonS3',
 'transfer-encoding': 'chunked',
 'x-amz-id-2': 'VCLi4Tph0mzbatYuzb3vNY16/eY0hJn9KcN/oMlMbLYS/gMBQI1+YxvpeatReZ1PLzp8ahhN1HQ=',
 'x-amz-request-id': 'AD42152AA3A28C6A'}
2014-11-16 10:58:32,423 - MainThread - botocore.parsers - DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>null</ArgumentValue><RequestId>AD42152AA3A28C6A</RequestId><HostId>VCLi4Tph0mzbatYuzb3vNY16/eY0hJn9KcN/oMlMbLYS/gMBQI1+YxvpeatReZ1PLzp8ahhN1HQ=</HostId></Error>
2014-11-16 10:58:32,424 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x108a1f850>
2014-11-16 10:58:32,424 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2014-11-16 10:58:32,424 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <bound method StreamingOutputArgument.save_file of <awscli.customizations.streamingoutputarg.StreamingOutputArgument object at 0x108ab3a10>>
2014-11-16 10:58:32,424 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 359, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 494, in __call__
    self._operation_object, call_parameters, parsed_globals)
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 599, in invoke
    **parameters)
  File "/usr/local/lib/python2.7/site-packages/botocore/operation.py", line 98, in call
    parsed=response[1])
  File "/usr/local/lib/python2.7/site-packages/botocore/session.py", line 729, in emit
    return self._events.emit(event_name, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/hooks.py", line 185, in emit
    response = handler(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/awscli/customizations/streamingoutputarg.py", line 92, in save_file
    body = parsed[self._response_key]
KeyError: u'Body'
2014-11-16 10:58:32,425 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

u'Body'

[jamesls]

Looks like there's two issues here. The first issue is fixed via boto/botocore#382, which is that there needs to be a way to enable signature version 4 for S3. You can do this by running:

aws configure set default.s3.signature_version s3v4

# Or for a specific profile
aws configure set profile.<profilename>.s3.signature_version s3v4

The second issue is that we aren't handling the specific case where we're streaming to a file and receive an error response from S3, which I'm fixing now.