Aws-cli: Access denied when uploading multipart that requires --acl bucket-owner-full-control

Created on 9 Dec 2015  路  21Comments  路  Source: aws/aws-cli

I have a problem uploading relatively big files on s3 to another account's s3 bucket. Small files uploaded OK, the ones that do it multipart - fail.

Below are bucket settings from bucket owner's account and output of aws s3 cp with --debug. Boto uploads same files without error. Not sure if it uses multipart, files up to 100MB in size.

The settings do not allow uploads without --acl bucket-owner-full-control

{
    "Version": "2012-10-17",
    "Id": "Policy1433167545906",
    "Statement": [
        {
            "Sid": "AnotherAccountRoleAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<id_here>:role/another"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::bucket-name/*",
                "arn:aws:s3:::bucket-name"
            ]
        },
        {
            "Sid": "AnotherAccountRoleAccess",
            "Effect": "Deny",
            "Principal": {
                "AWS": "arn:aws:iam::<id_here>:role/another"
            },
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::bucket-name/*",
                "arn:aws:s3:::bucket-name"
            ],
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        }
    ]
}

I stripped several long hash-like strings from the log, I hope nobody needs them.

ubuntu@ip-10-64-49-201:~$ aws --version
aws-cli/1.9.9 Python/2.7.6 Linux/3.13.0-48-generic botocore/1.3.9
ubuntu@ip-10-64-49-201:~$ aws s3 cp test_file s3://bucket-name/dir-name/ --acl bucket-owner-full-control --debug
2015-12-09 13:03:33,306 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.9.9 Python/2.7.6 Linux/3.13.0-48-generic botocore/1.3.9
2015-12-09 13:03:33,306 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', 'test_file', 's3://bucket-name/dir-name/', '--acl', 'bucket-owner-full-control', '--debug']
2015-12-09 13:03:33,307 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7fb00708aed8>
2015-12-09 13:03:33,307 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider at 0x7fb0072dfaa0>
2015-12-09 13:03:33,307 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2015-12-09 13:03:33,307 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7fb0072ed8c0>
2015-12-09 13:03:33,308 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,308 - MainThread - botocore.hooks - DEBUG - Event building-command-table.cp: calling handler <function add_waiters at 0x7fb0072ed8c0>
2015-12-09 13:03:33,310 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-key: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,311 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,311 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,311 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,311 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,311 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,311 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source-key: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,311 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,311 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,312 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,312 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,312 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,312 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,312 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,312 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7fb0070a80d0>
2015-12-09 13:03:33,312 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,312 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,312 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,313 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.ignore-glacier-warnings: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,313 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7fb0070a80d0>
2015-12-09 13:03:33,313 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,313 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,313 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,313 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,313 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7fb0070a80d0>
2015-12-09 13:03:33,313 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,313 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,314 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7fb0070a80d0>
2015-12-09 13:03:33,314 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,314 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7fb0070a80d0>
2015-12-09 13:03:33,314 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,314 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-kms-key-id: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,314 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,314 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler <function uri_param at 0x7fb007467c80>
2015-12-09 13:03:33,314 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7fb0070a80d0>
2015-12-09 13:03:33,314 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2015-12-09 13:03:33,315 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2015-12-09 13:03:33,315 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2015-12-09 13:03:33,315 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2015-12-09 13:03:33,315 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2015-12-09 13:03:33,315 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2015-12-09 13:03:33,315 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2015-12-09 13:03:33,321 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2015-12-09 13:03:33,322 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 7
2015-12-09 13:03:33,323 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2015-12-09 13:03:33,324 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/fetcher HTTP/1.1" 200 930
2015-12-09 13:03:33,325 - MainThread - botocore.credentials - INFO - Found credentials from IAM Role: fetcher
2015-12-09 13:03:33,350 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2015-12-09 13:03:33,353 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fb007878cf8>
2015-12-09 13:03:33,354 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fb007861b90>
2015-12-09 13:03:33,356 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2015-12-09 13:03:33,357 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2015-12-09 13:03:33,359 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fb007878cf8>
2015-12-09 13:03:33,359 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fb007861b90>
2015-12-09 13:03:33,361 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2015-12-09 13:03:33,361 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2015-12-09 13:03:33,362 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fb007878cf8>
2015-12-09 13:03:33,363 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fb007861b90>
2015-12-09 13:03:33,364 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2015-12-09 13:03:33,365 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608
2015-12-09 13:03:33,365 - MainThread - awscli.customizations.s3.executor - DEBUG - Using max queue size for s3 tasks of: 1000
2015-12-09 13:03:33,365 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608
2015-12-09 13:03:33,365 - MainThread - awscli.customizations.s3.executor - DEBUG - Using max queue size for s3 tasks of: 2
2015-12-09 13:03:33,366 - MainThread - awscli.customizations.s3.executor - DEBUG - Using a threadpool size of: 10
2015-12-09 13:03:33,368 - MainThread - awscli.customizations.s3.executor - DEBUG - Submitting task: <awscli.customizations.s3.tasks.CreateMultipartUploadTask object at 0x7fb0069a1e50>
2015-12-09 13:03:33,368 - MainThread - awscli.customizations.s3.executor - DEBUG - Submitting task: <awscli.customizations.s3.tasks.UploadPartTask object at 0x7fb0069a1f10>
2015-12-09 13:03:33,368 - MainThread - awscli.customizations.s3.executor - DEBUG - Submitting task: <awscli.customizations.s3.tasks.UploadPartTask object at 0x7fb0069a1ed0>
2015-12-09 13:03:33,368 - Thread-6 - awscli.customizations.s3.executor - DEBUG - Worker thread invoking task: <awscli.customizations.s3.tasks.CreateMultipartUploadTask object at 0x7fb0069a1e50>
2015-12-09 13:03:33,369 - Thread-6 - awscli.customizations.s3.tasks - DEBUG - Creating multipart upload for file: /home/ubuntu/test_file
2015-12-09 13:03:33,369 - Thread-5 - awscli.customizations.s3.executor - DEBUG - Worker thread invoking task: <awscli.customizations.s3.tasks.UploadPartTask object at 0x7fb0069a1f10>
2015-12-09 13:03:33,369 - Thread-5 - awscli.customizations.s3.tasks - DEBUG - Uploading part 1 for filename: /home/ubuntu/test_file
2015-12-09 13:03:33,370 - Thread-5 - awscli.customizations.s3.tasks - DEBUG - Waiting for upload id.
2015-12-09 13:03:33,369 - MainThread - awscli.customizations.s3.executor - DEBUG - Submitting task: <awscli.customizations.s3.tasks.UploadPartTask object at 0x7fb0069310d0>
2015-12-09 13:03:33,370 - Thread-7 - awscli.customizations.s3.executor - DEBUG - Worker thread invoking task: <awscli.customizations.s3.tasks.UploadPartTask object at 0x7fb0069a1ed0>
2015-12-09 13:03:33,370 - Thread-7 - awscli.customizations.s3.tasks - DEBUG - Uploading part 2 for filename: /home/ubuntu/test_file
2015-12-09 13:03:33,370 - Thread-7 - awscli.customizations.s3.tasks - DEBUG - Waiting for upload id.
2015-12-09 13:03:33,370 - MainThread - awscli.customizations.s3.executor - DEBUG - Submitting task: <awscli.customizations.s3.tasks.CompleteMultipartUploadTask object at 0x7fb0069a1e90>
2015-12-09 13:03:33,371 - Thread-8 - awscli.customizations.s3.executor - DEBUG - Worker thread invoking task: <awscli.customizations.s3.tasks.UploadPartTask object at 0x7fb0069310d0>
2015-12-09 13:03:33,371 - Thread-8 - awscli.customizations.s3.tasks - DEBUG - Uploading part 3 for filename: /home/ubuntu/test_file
2015-12-09 13:03:33,371 - Thread-9 - awscli.customizations.s3.executor - DEBUG - Worker thread invoking task: <awscli.customizations.s3.tasks.CompleteMultipartUploadTask object at 0x7fb0069a1e90>
2015-12-09 13:03:33,372 - Thread-8 - awscli.customizations.s3.tasks - DEBUG - Waiting for upload id.
2015-12-09 13:03:33,372 - Thread-9 - awscli.customizations.s3.tasks - DEBUG - Completing multipart upload for file: /home/ubuntu/test_file
2015-12-09 13:03:33,372 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,373 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,373 - Thread-10 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:33,373 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,373 - Thread-11 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:33,374 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,375 - Thread-12 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:33,375 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,375 - Thread-13 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:33,375 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,377 - Thread-6 - botocore.hooks - DEBUG - Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function sse_md5 at 0x7fb0078152a8>
2015-12-09 13:03:33,377 - Thread-6 - botocore.hooks - DEBUG - Event before-parameter-build.s3.CreateMultipartUpload: calling handler <function validate_bucket_name at 0x7fb007815230>
2015-12-09 13:03:33,375 - Thread-14 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:33,380 - Thread-6 - botocore.hooks - DEBUG - Event before-call.s3.CreateMultipartUpload: calling handler <function add_expect_header at 0x7fb007815668>
2015-12-09 13:03:33,377 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,380 - Thread-6 - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x7fb006c6f510> (verify_ssl=True) with params: {'body': '', 'url': u'https://s3.amazonaws.com/bucket-name/dir-name/test_file?uploads', 'headers': {u'x-amz-acl': u'bucket-owner-full-control', 'User-Agent': 'aws-cli/1.9.9 Python/2.7.6 Linux/3.13.0-48-generic botocore/1.3.9'}, 'query_string': {}, 'url_path': u'/bucket-name/dir-name/test_file?uploads', 'method': u'POST'}
2015-12-09 13:03:33,381 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,381 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,381 - Thread-6 - botocore.hooks - DEBUG - Event request-created.s3.CreateMultipartUpload: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fb006c8a450>>
2015-12-09 13:03:33,382 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2015-12-09 13:03:33,381 - Thread-6 - botocore.hooks - DEBUG - Event before-sign.s3.CreateMultipartUpload: calling handler <function fix_s3_host at 0x7fb007912398>
2015-12-09 13:03:33,382 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-5, started daemon 140393984423680)>
2015-12-09 13:03:33,382 - Thread-6 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/bucket-name/dir-name/test_file?uploads
2015-12-09 13:03:33,382 - Thread-6 - botocore.utils - DEBUG - URI updated to: https://bucket-name.s3.amazonaws.com/dir-name/test_file?uploads
2015-12-09 13:03:33,382 - Thread-6 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2015-12-09 13:03:33,382 - Thread-6 - botocore.auth - DEBUG - HTTP request method: POST
2015-12-09 13:03:33,383 - Thread-6 - botocore.auth - DEBUG - StringToSign:
POST


Wed, 09 Dec 2015 13:03:33 GMT
x-amz-acl:bucket-owner-full-control
x-amz-security-token:long_string_i_hope_nobody_needs_1
/bucket-name/dir-name/test_file?uploads
2015-12-09 13:03:33,383 - Thread-6 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>
2015-12-09 13:03:33,384 - Thread-6 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): bucket-name.s3.amazonaws.com
2015-12-09 13:03:33,805 - Thread-6 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST /dir-name/test_file?uploads HTTP/1.1" 200 None
2015-12-09 13:03:33,806 - Thread-6 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '9UvH0dfbFUMo+oBP+TXPQksgCrPL4tfiyG2nnRRbTQ8baNTZLFy/19T+MFZMp3Uoi4L+S6lH7Kg=', 'date': 'Wed, 09 Dec 2015 13:03:34 GMT', 'transfer-encoding': 'chunked', 'x-amz-request-id': 'D52F86A31E708F82', 'server': 'AmazonS3'}
2015-12-09 13:03:33,806 - Thread-6 - botocore.parsers - DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>bucket-name</Bucket><Key>dir-name/test_file</Key><UploadId>long_string_i_hope_nobody_needs_2</UploadId></InitiateMultipartUploadResult>
2015-12-09 13:03:33,807 - Thread-6 - botocore.hooks - DEBUG - Event needs-retry.s3.CreateMultipartUpload: calling handler <botocore.retryhandler.RetryHandler object at 0x7fb0069e6a10>
2015-12-09 13:03:33,807 - Thread-6 - botocore.retryhandler - DEBUG - No retry needed.
2015-12-09 13:03:33,807 - Thread-6 - botocore.hooks - DEBUG - Event after-call.s3.CreateMultipartUpload: calling handler <function enhance_error_msg at 0x7fb0072d8b18>
2015-12-09 13:03:33,807 - Thread-6 - botocore.hooks - DEBUG - Event after-call.s3.CreateMultipartUpload: calling handler <awscli.errorhandler.ErrorHandler object at 0x7fb0070a8190>
2015-12-09 13:03:33,808 - Thread-6 - awscli.errorhandler - DEBUG - HTTP Response Code: 200
2015-12-09 13:03:33,808 - Thread-6 - awscli.customizations.s3.tasks - DEBUG - Announcing upload id: long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,808 - Thread-6 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:33,838 - Thread-7 - botocore.hooks - DEBUG - Event before-parameter-build.s3.UploadPart: calling handler <function sse_md5 at 0x7fb0078152a8>
2015-12-09 13:03:33,839 - Thread-7 - botocore.hooks - DEBUG - Event before-parameter-build.s3.UploadPart: calling handler <function validate_bucket_name at 0x7fb007815230>
2015-12-09 13:03:33,839 - Thread-5 - botocore.hooks - DEBUG - Event before-parameter-build.s3.UploadPart: calling handler <function sse_md5 at 0x7fb0078152a8>
2015-12-09 13:03:33,839 - Thread-8 - botocore.hooks - DEBUG - Event before-parameter-build.s3.UploadPart: calling handler <function sse_md5 at 0x7fb0078152a8>
2015-12-09 13:03:33,841 - Thread-7 - botocore.hooks - DEBUG - Event before-call.s3.UploadPart: calling handler <function conditionally_calculate_md5 at 0x7fb0078151b8>
2015-12-09 13:03:33,841 - Thread-5 - botocore.hooks - DEBUG - Event before-parameter-build.s3.UploadPart: calling handler <function validate_bucket_name at 0x7fb007815230>
2015-12-09 13:03:33,844 - Thread-5 - botocore.hooks - DEBUG - Event before-call.s3.UploadPart: calling handler <function conditionally_calculate_md5 at 0x7fb0078151b8>
2015-12-09 13:03:33,841 - Thread-8 - botocore.hooks - DEBUG - Event before-parameter-build.s3.UploadPart: calling handler <function validate_bucket_name at 0x7fb007815230>
2015-12-09 13:03:33,845 - Thread-8 - botocore.hooks - DEBUG - Event before-call.s3.UploadPart: calling handler <function conditionally_calculate_md5 at 0x7fb0078151b8>
2015-12-09 13:03:33,854 - Thread-8 - botocore.hooks - DEBUG - Event before-call.s3.UploadPart: calling handler <function add_expect_header at 0x7fb007815668>
2015-12-09 13:03:33,856 - Thread-8 - botocore.handlers - DEBUG - Adding expect 100 continue header to request.
2015-12-09 13:03:33,857 - Thread-8 - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x7fb006c8a250> (verify_ssl=True) with params: {'body': <awscli.customizations.s3.utils.ReadFileChunk object at 0x7fb00694aa50>, 'url': u'https://s3.amazonaws.com/bucket-name/dir-name/test_file?partNumber=3&uploadId=long_string_i_hope_nobody_needs_2', 'headers': {'Content-MD5': u'NsGbEOFuO6hRTXAFlHbEnA==', 'Expect': '100-continue', 'User-Agent': 'aws-cli/1.9.9 Python/2.7.6 Linux/3.13.0-48-generic botocore/1.3.9'}, 'query_string': {u'partNumber': 3, u'uploadId': 'long_string_i_hope_nobody_needs_2'}, 'url_path': u'/bucket-name/dir-name/test_file', 'method': u'PUT'}
2015-12-09 13:03:33,864 - Thread-8 - botocore.hooks - DEBUG - Event request-created.s3.UploadPart: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fb006c8a450>>
2015-12-09 13:03:33,864 - Thread-8 - botocore.hooks - DEBUG - Event before-sign.s3.UploadPart: calling handler <function fix_s3_host at 0x7fb007912398>
2015-12-09 13:03:33,864 - Thread-8 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/bucket-name/dir-name/test_file?partNumber=3&uploadId=long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,864 - Thread-8 - botocore.utils - DEBUG - URI updated to: https://bucket-name.s3.amazonaws.com/dir-name/test_file?partNumber=3&uploadId=long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,865 - Thread-8 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2015-12-09 13:03:33,865 - Thread-8 - botocore.auth - DEBUG - HTTP request method: PUT
2015-12-09 13:03:33,865 - Thread-8 - botocore.auth - DEBUG - StringToSign:
PUT
NsGbEOFuO6hRTXAFlHbEnA==

Wed, 09 Dec 2015 13:03:33 GMT
x-amz-security-token:long_string_i_hope_nobody_needs_1
/bucket-name/dir-name/test_file?partNumber=3&uploadId=long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,866 - Thread-8 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2015-12-09 13:03:33,867 - Thread-8 - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2015-12-09 13:03:33,870 - Thread-5 - botocore.hooks - DEBUG - Event before-call.s3.UploadPart: calling handler <function add_expect_header at 0x7fb007815668>
2015-12-09 13:03:33,871 - Thread-5 - botocore.handlers - DEBUG - Adding expect 100 continue header to request.
2015-12-09 13:03:33,871 - Thread-5 - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x7fb006c8a250> (verify_ssl=True) with params: {'body': <awscli.customizations.s3.utils.ReadFileChunk object at 0x7fb00694ac10>, 'url': u'https://s3.amazonaws.com/bucket-name/dir-name/test_file?partNumber=1&uploadId=long_string_i_hope_nobody_needs_2', 'headers': {'Content-MD5': u'L6rjvUywbKksMPhJoyk9Qw==', 'Expect': '100-continue', 'User-Agent': 'aws-cli/1.9.9 Python/2.7.6 Linux/3.13.0-48-generic botocore/1.3.9'}, 'query_string': {u'partNumber': 1, u'uploadId': 'long_string_i_hope_nobody_needs_2'}, 'url_path': u'/bucket-name/dir-name/test_file', 'method': u'PUT'}
2015-12-09 13:03:33,871 - Thread-5 - botocore.hooks - DEBUG - Event request-created.s3.UploadPart: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fb006c8a450>>
2015-12-09 13:03:33,871 - Thread-5 - botocore.hooks - DEBUG - Event before-sign.s3.UploadPart: calling handler <function fix_s3_host at 0x7fb007912398>
2015-12-09 13:03:33,871 - Thread-5 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/bucket-name/dir-name/test_file?partNumber=1&uploadId=long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,871 - Thread-5 - botocore.utils - DEBUG - URI updated to: https://bucket-name.s3.amazonaws.com/dir-name/test_file?partNumber=1&uploadId=long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,872 - Thread-5 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2015-12-09 13:03:33,872 - Thread-5 - botocore.auth - DEBUG - HTTP request method: PUT
2015-12-09 13:03:33,872 - Thread-5 - botocore.auth - DEBUG - StringToSign:
PUT
L6rjvUywbKksMPhJoyk9Qw==

Wed, 09 Dec 2015 13:03:33 GMT
x-amz-security-token:long_string_i_hope_nobody_needs_1
/bucket-name/dir-name/test_file?partNumber=1&uploadId=long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,873 - Thread-5 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2015-12-09 13:03:33,873 - Thread-5 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (2): bucket-name.s3.amazonaws.com
2015-12-09 13:03:33,876 - Thread-7 - botocore.hooks - DEBUG - Event before-call.s3.UploadPart: calling handler <function add_expect_header at 0x7fb007815668>
2015-12-09 13:03:33,876 - Thread-7 - botocore.handlers - DEBUG - Adding expect 100 continue header to request.
2015-12-09 13:03:33,876 - Thread-7 - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x7fb006c8a250> (verify_ssl=True) with params: {'body': <awscli.customizations.s3.utils.ReadFileChunk object at 0x7fb006931890>, 'url': u'https://s3.amazonaws.com/bucket-name/dir-name/test_file?partNumber=2&uploadId=long_string_i_hope_nobody_needs_2', 'headers': {'Content-MD5': u'vLB9RP4/ppyWBkQxRAUnaA==', 'Expect': '100-continue', 'User-Agent': 'aws-cli/1.9.9 Python/2.7.6 Linux/3.13.0-48-generic botocore/1.3.9'}, 'query_string': {u'partNumber': 2, u'uploadId': 'long_string_i_hope_nobody_needs_2'}, 'url_path': u'/bucket-name/dir-name/test_file', 'method': u'PUT'}
2015-12-09 13:03:33,876 - Thread-7 - botocore.hooks - DEBUG - Event request-created.s3.UploadPart: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fb006c8a450>>
2015-12-09 13:03:33,876 - Thread-7 - botocore.hooks - DEBUG - Event before-sign.s3.UploadPart: calling handler <function fix_s3_host at 0x7fb007912398>
2015-12-09 13:03:33,876 - Thread-7 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/bucket-name/dir-name/test_file?partNumber=2&uploadId=long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,876 - Thread-7 - botocore.utils - DEBUG - URI updated to: https://bucket-name.s3.amazonaws.com/dir-name/test_file?partNumber=2&uploadId=long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,876 - Thread-7 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2015-12-09 13:03:33,876 - Thread-7 - botocore.auth - DEBUG - HTTP request method: PUT
2015-12-09 13:03:33,877 - Thread-7 - botocore.auth - DEBUG - StringToSign:
PUT
vLB9RP4/ppyWBkQxRAUnaA==

Wed, 09 Dec 2015 13:03:33 GMT
x-amz-security-token:long_string_i_hope_nobody_needs_1
/bucket-name/dir-name/test_file?partNumber=2&uploadId=long_string_i_hope_nobody_needs_2
2015-12-09 13:03:33,877 - Thread-7 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2015-12-09 13:03:33,877 - Thread-7 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (3): bucket-name.s3.amazonaws.com
2015-12-09 13:03:33,971 - Thread-8 - botocore.awsrequest - DEBUG - Received a non 100 Continue response from the server, NOT sending request body.
2015-12-09 13:03:33,972 - Thread-8 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "PUT /dir-name/test_file?partNumber=3&uploadId=long_string_i_hope_nobody_needs_2 HTTP/1.1" 403 None
2015-12-09 13:03:33,974 - Thread-8 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'HdrSGG2LMtIAY2eVEIJdKP2J08mnLN6ouJog/Dc6AhE1TOLFMYQUP7/cRZFZvT1fC80cZ4xMcBc=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'close', 'x-amz-request-id': '664B21F684C8C7E2', 'date': 'Wed, 09 Dec 2015 13:03:33 GMT', 'content-type': 'application/xml'}
2015-12-09 13:03:33,975 - Thread-8 - botocore.parsers - DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>664B21F684C8C7E2</RequestId><HostId>HdrSGG2LMtIAY2eVEIJdKP2J08mnLN6ouJog/Dc6AhE1TOLFMYQUP7/cRZFZvT1fC80cZ4xMcBc=</HostId></Error>
2015-12-09 13:03:33,975 - Thread-8 - botocore.hooks - DEBUG - Event needs-retry.s3.UploadPart: calling handler <botocore.retryhandler.RetryHandler object at 0x7fb0069e6a10>
2015-12-09 13:03:33,975 - Thread-8 - botocore.retryhandler - DEBUG - No retry needed.
2015-12-09 13:03:33,976 - Thread-8 - botocore.hooks - DEBUG - Event after-call.s3.UploadPart: calling handler <function enhance_error_msg at 0x7fb0072d8b18>
2015-12-09 13:03:33,976 - Thread-8 - botocore.hooks - DEBUG - Event after-call.s3.UploadPart: calling handler <awscli.errorhandler.ErrorHandler object at 0x7fb0070a8190>
2015-12-09 13:03:33,976 - Thread-8 - awscli.errorhandler - DEBUG - HTTP Response Code: 403
2015-12-09 13:03:33,976 - Thread-8 - awscli.customizations.s3.tasks - DEBUG - Error during part upload: A client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/tasks.py", line 241, in __call__
    response_data = self._filename.client.upload_part(**params)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 391, in _make_api_call
    model=operation_model
  File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/awscli/errorhandler.py", line 70, in __call__
    http_status_code=http_response.status_code)
ClientError: A client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied
2015-12-09 13:03:33,978 - Thread-8 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:33,978 - Thread-1 - awscli.customizations.s3.executor - DEBUG - Received print task: PrintTask(message=u'upload failed: ./test_file to s3://bucket-name/dir-name/test_file\nA client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied', error=True, total_parts=None, warning=None)
upload failed: ./test_file to s3://bucket-name/dir-name/test_file
A client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied
2015-12-09 13:03:34,163 - Thread-7 - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2015-12-09 13:03:34,170 - Thread-5 - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2015-12-09 13:03:34,277 - Thread-7 - botocore.awsrequest - DEBUG - Received a non 100 Continue response from the server, NOT sending request body.
2015-12-09 13:03:34,277 - Thread-7 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "PUT /dir-name/test_file?partNumber=2&uploadId=long_string_i_hope_nobody_needs_2 HTTP/1.1" 403 None
2015-12-09 13:03:34,279 - Thread-7 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'long_string_i_hope_nobody_needs_3', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'close', 'x-amz-request-id': 'A0B60AA2ABB01032', 'date': 'Wed, 09 Dec 2015 13:03:33 GMT', 'content-type': 'application/xml'}
2015-12-09 13:03:34,279 - Thread-7 - botocore.parsers - DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>A0B60AA2ABB01032</RequestId><HostId>long_string_i_hope_nobody_needs_3</HostId></Error>
2015-12-09 13:03:34,279 - Thread-7 - botocore.hooks - DEBUG - Event needs-retry.s3.UploadPart: calling handler <botocore.retryhandler.RetryHandler object at 0x7fb0069e6a10>
2015-12-09 13:03:34,279 - Thread-7 - botocore.retryhandler - DEBUG - No retry needed.
2015-12-09 13:03:34,280 - Thread-7 - botocore.hooks - DEBUG - Event after-call.s3.UploadPart: calling handler <function enhance_error_msg at 0x7fb0072d8b18>
2015-12-09 13:03:34,280 - Thread-7 - botocore.hooks - DEBUG - Event after-call.s3.UploadPart: calling handler <awscli.errorhandler.ErrorHandler object at 0x7fb0070a8190>
2015-12-09 13:03:34,280 - Thread-7 - awscli.errorhandler - DEBUG - HTTP Response Code: 403
2015-12-09 13:03:34,280 - Thread-7 - awscli.customizations.s3.tasks - DEBUG - Error during part upload: A client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/tasks.py", line 241, in __call__
    response_data = self._filename.client.upload_part(**params)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 391, in _make_api_call
    model=operation_model
  File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/awscli/errorhandler.py", line 70, in __call__
    http_status_code=http_response.status_code)
ClientError: A client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied
2015-12-09 13:03:34,280 - Thread-7 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:34,281 - Thread-1 - awscli.customizations.s3.executor - DEBUG - Received print task: PrintTask(message=u'upload failed: ./test_file to s3://bucket-name/dir-name/test_file\nA client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied', error=True, total_parts=None, warning=None)
upload failed: ./test_file to s3://bucket-name/dir-name/test_file
A client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied
2015-12-09 13:03:34,331 - Thread-5 - botocore.awsrequest - DEBUG - Received a non 100 Continue response from the server, NOT sending request body.
2015-12-09 13:03:34,332 - Thread-5 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "PUT /dir-name/test_file?partNumber=1&uploadId=long_string_i_hope_nobody_needs_2 HTTP/1.1" 403 None
2015-12-09 13:03:34,333 - Thread-5 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'long_string_i_hope_nobody_needs_4', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'close', 'x-amz-request-id': 'B58B1184631E951C', 'date': 'Wed, 09 Dec 2015 13:03:33 GMT', 'content-type': 'application/xml'}
2015-12-09 13:03:34,333 - Thread-5 - botocore.parsers - DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>B58B1184631E951C</RequestId><HostId>long_string_i_hope_nobody_needs_4</HostId></Error>
2015-12-09 13:03:34,334 - Thread-5 - botocore.hooks - DEBUG - Event needs-retry.s3.UploadPart: calling handler <botocore.retryhandler.RetryHandler object at 0x7fb0069e6a10>
2015-12-09 13:03:34,334 - Thread-5 - botocore.retryhandler - DEBUG - No retry needed.
2015-12-09 13:03:34,334 - Thread-5 - botocore.hooks - DEBUG - Event after-call.s3.UploadPart: calling handler <function enhance_error_msg at 0x7fb0072d8b18>
2015-12-09 13:03:34,334 - Thread-5 - botocore.hooks - DEBUG - Event after-call.s3.UploadPart: calling handler <awscli.errorhandler.ErrorHandler object at 0x7fb0070a8190>
2015-12-09 13:03:34,334 - Thread-5 - awscli.errorhandler - DEBUG - HTTP Response Code: 403
2015-12-09 13:03:34,334 - Thread-5 - awscli.customizations.s3.tasks - DEBUG - Error during part upload: A client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/tasks.py", line 241, in __call__
    response_data = self._filename.client.upload_part(**params)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 391, in _make_api_call
    model=operation_model
  File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/awscli/errorhandler.py", line 70, in __call__
    http_status_code=http_response.status_code)
ClientError: A client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied
2015-12-09 13:03:34,335 - Thread-5 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:34,335 - Thread-1 - awscli.customizations.s3.executor - DEBUG - Received print task: PrintTask(message=u'upload failed: ./test_file to s3://bucket-name/dir-name/test_file\nA client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied', error=True, total_parts=None, warning=None)
upload failed: ./test_file to s3://bucket-name/dir-name/test_file
A client error (AccessDenied) occurred when calling the UploadPart operation: Access Denied
2015-12-09 13:03:34,350 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-5, stopped daemon 140393984423680)>
2015-12-09 13:03:34,350 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-6, stopped daemon 140393976030976)>
2015-12-09 13:03:34,350 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-6, stopped daemon 140393976030976)>
2015-12-09 13:03:34,350 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-7, stopped daemon 140393967638272)>
2015-12-09 13:03:34,350 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-7, stopped daemon 140393967638272)>
2015-12-09 13:03:34,350 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-8, stopped daemon 140393622533888)>
2015-12-09 13:03:34,350 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-8, stopped daemon 140393622533888)>
2015-12-09 13:03:34,350 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-9, started daemon 140393614141184)>
2015-12-09 13:03:34,838 - Thread-9 - awscli.customizations.s3.executor - DEBUG - Error calling task: Upload has been cancelled.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/executor.py", line 226, in run
    function()
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/tasks.py", line 504, in __call__
    parts = self._upload_context.wait_for_parts_to_finish()
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/tasks.py", line 612, in wait_for_parts_to_finish
    raise UploadCancelledError("Upload has been cancelled.")
UploadCancelledError: Upload has been cancelled.
2015-12-09 13:03:34,839 - Thread-9 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-12-09 13:03:34,865 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-9, stopped daemon 140393614141184)>
2015-12-09 13:03:34,865 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-10, stopped daemon 140393605748480)>
2015-12-09 13:03:34,865 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-10, stopped daemon 140393605748480)>
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-11, stopped daemon 140393597355776)>
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-11, stopped daemon 140393597355776)>
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-12, stopped daemon 140393588963072)>
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-12, stopped daemon 140393588963072)>
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-13, stopped daemon 140393580570368)>
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-13, stopped daemon 140393580570368)>
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-14, stopped daemon 140393572177664)>
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-14, stopped daemon 140393572177664)>
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for result thread.
2015-12-09 13:03:34,866 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for IO thread.
2015-12-09 13:03:34,867 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for result thread to shutdown.
2015-12-09 13:03:34,867 - Thread-2 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in IO thread, shutting down.
2015-12-09 13:03:34,867 - Thread-1 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in print thread, shutting down print thread.
2015-12-09 13:03:34,867 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for IO thread to shutdown.
2015-12-09 13:03:34,867 - MainThread - awscli.customizations.s3.executor - DEBUG - All threads have been shutdown.
closing-soon guidance s3 service-api
Source
picture artem-cliqz
👍10

Most helpful comment

This looks like a bug in the S3/IAM integration internals to me. The problem of objects not being modifiable by other users even if they have permission on the bucket is a popular one. In response, AWS has published an example bucket policy to force users to use --acl bucket-owner-full-control. The fact that UploadPart reuses the permissions from PutObject makes it impossible to restrict access like this; their example is broken and doesn't allow any multipart uploads, even if they have the correct ACL set.

Per https://github.com/aws/aws-cli/issues/1674#issuecomment-163717329, awscli can't even work around it by sending the s3:x-amz-acl=bucket-owner-full-control header for every UploadPart operation, so there seems to be no alternative to either (1) not using multipart uploads, or (2) not using the ACL enforcement policy.

picture jonasschneider on 2 May 2017
👍20

All 21 comments

Unfortunately there is not really much we can. The request is failing on UploadPart operation. This operation is encompassed by the s3:PutObject action in the bucket policy. However, s3 does not permit sending the x-amz-acl for that operation. (Note this is just a python script that I wrote to test it by injecting the x-amz-acl header):

import boto3

s3 = boto3.client('s3')

def add_header(params, **kwargs):
    params['headers']['x-amz-acl'] = 'bucket-owner-full-control'

s3.meta.events.register('before-call.s3', add_header)


boto3.set_stream_logger('botocore')
response = s3.create_multipart_upload(
    Bucket='mybucket',
    Key='foo'
)

id = response['UploadId']

s3.upload_part(
    Bucket='mybucket',
    Key='foo',
    PartNumber=1,
    UploadId=id
)

This errors out on the upload_part method with:

botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the UploadPart operation: The specified header is not valid in this context

The best you could do is to set the multipart_threshold in ~/.aws/config to a size where multipart uploads do not happen for the data you are sending. However, that will make the upload slower. Here is the guide on how to do that: http://docs.aws.amazon.com/cli/latest/topic/s3-config.html#cli-aws-help-s3-config. Let me know if you have anymore questions.

kyleknap picture kyleknap on 10 Dec 2015

Thanks a lot for your time to reproduce the issue. Appreciate it. It sound like AWS S3 api is not fully functional. I believe the api developers should be aware of the problem. Do you think it worth contacting them?

artem-cliqz picture artem-cliqz on 10 Dec 2015

Thanks for the feedback. We will pass it along to S3 and see what can be done.

kyleknap picture kyleknap on 10 Dec 2015

I'm also having this issue with files of around 150MB. It used to work but I had to disable multipart upload since I started adding the flags for server-side encryption using KMS:

aws s3 cp file.sql.gz "s3://bucketname/backup/daily/file.sql.gz --sse aws:kms --sse-kms-key-id "99999999-9999-9999-9999-999999999999" --region eu-west-1
vincent-dm picture vincent-dm on 27 Jan 2016
👍4

I ran into this too. I hope (but sort of doubt) the S3 team is reading this thread 馃槃

copumpkin picture copumpkin on 9 Sep 2016

I'm also seeing this bug.

I have two accounts: A and B. The S3 bucket lives in account A, and I would like to write from B while maintaining object ownership in account A. Here is my bucket policy:

{
  "Version":"2012-10-17",
  "Statement": [ 
    {
      "Sid":"AllowWritesWhileMaintainingOwnership",
        "Effect":"Allow",
      "Principal": {
            "AWS": [
              "arn:aws:iam::ACCOUNT_A:root",
              "arn:aws:iam::ACCOUNT_B:root"
            ]
        },
      "Action":["s3:PutObject"],
      "Resource": [
        "arn:aws:s3:::MY_BUCKET/*"
      ],
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}

Now the following steps are done using ACCOUNT_B's credentials:

user:~/test-s3-bucket $ head -c 2097152 /dev/urandom > test.txt
user:~/test-s3-bucket $ head -c 53477376 /dev/urandom > test2.txt
user:~/test-s3-bucket $ du -hs *
2.0M    test.txt
 51M    test2.txt
user:~/test-s3-bucket $ aws s3 cp test.txt s3://MY_BUCKET --acl bucket-owner-full-control
upload: ./test.txt to s3://MY_BUCKET/test.txt
user:~/test-s3-bucket $ aws s3 cp test2.txt s3://MY_BUCKET --acl bucket-owner-full-control
upload failed: ./test2.txt to s3://MY_BUCKET/test2.txt
An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied
upload failed: ./test2.txt to s3://MY_BUCKET/test2.txt
An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied
upload failed: ./test2.txt to s3://MY_BUCKET/test2.txt
An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied
upload failed: ./test2.txt to s3://MY_BUCKET/test2.txt
An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied
upload failed: ./test2.txt to s3://MY_BUCKET/test2.txt
An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied
upload failed: ./test2.txt to s3://MY_BUCKET/test2.txt
An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied
upload failed: ./test2.txt to s3://MY_BUCKET/test2.txt
An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied
Completed 7 of 7 part(s) with -6 file(s) remaining
user:~/test-s3-bucket $ aws configure set default.s3.multipart_threshold 52MB
user:~/test-s3-bucket $ aws s3 cp test2.txt s3://MY_BUCKET --acl bucket-owner-full-control
upload: ./test2.txt to s3://MY_BUCKET/test2.txt

You can see the workaround of setting the multipart object threshold to be bigger than the files you are uploading as mentioned above, but that is not an ideal solution.

pc-0 picture pc-0 on 9 Sep 2016

In fact the bug is still in place and even with setting high _multipart_threshold_ or using aws s3api put-object --acl bucket-owner-full-control ... command you are limited to maximum of 5GB file upload. Else, you get

aws s3 cp 6g.file s3://my_bucket --acl bucket-owner-full-control
upload failed: ./6g.file to s3://my_bucket/6g.file An error occurred (EntityTooLarge) when calling the PutObject operation: Your proposed upload exceeds the maximum allowed size
rparkhunovskyi picture rparkhunovskyi on 11 Apr 2017
👍2

Hello - this is not a bug. However there are some ways to get round the issue. Using assume-roles - here is an article that speaks to how you may want to solve this in your-use case: http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example4.html

etiennemunnich picture etiennemunnich on 21 Apr 2017
👎1

This looks like a bug in the S3/IAM integration internals to me. The problem of objects not being modifiable by other users even if they have permission on the bucket is a popular one. In response, AWS has published an example bucket policy to force users to use --acl bucket-owner-full-control. The fact that UploadPart reuses the permissions from PutObject makes it impossible to restrict access like this; their example is broken and doesn't allow any multipart uploads, even if they have the correct ACL set.

Per https://github.com/aws/aws-cli/issues/1674#issuecomment-163717329, awscli can't even work around it by sending the s3:x-amz-acl=bucket-owner-full-control header for every UploadPart operation, so there seems to be no alternative to either (1) not using multipart uploads, or (2) not using the ACL enforcement policy.

jonasschneider picture jonasschneider on 2 May 2017
👍20

@kyleknap why was this issued closed? Is there another group that is in charge of this problem now? Is there a way to track the progress on this bug? Thanks!

rmmcnulty9 picture rmmcnulty9 on 24 Aug 2017
👍3

I just got this issue yesterday -- it seems to still be a problem.

aws --profile <Account> s3 cp <filename> <bucket> --acl bucket-owner-full-control --sse AES256

Can someone take a step back and explain at a high level what is causing this behavior exactly? Has it to do with object acl specifically with multi-part-uploads?

peterpod-c1 picture peterpod-c1 on 29 Aug 2017

Hello, @kyleknap, how to upload file bigger than 5Gb?

An error occurred (EntityTooLarge) when calling the PutObject operation: Your proposed upload exceeds the maximum allowed size
taraslayshchuk picture taraslayshchuk on 4 Sep 2017

Any updates on this on the s3 side?

cancan101 picture cancan101 on 4 Nov 2017
👍2

+1 this certainly seems like a bug.

I had trouble figuring out how to increase the multipart threshold value as a workaround, so posting here in case it helps anyone else with files under 5gb in size:

import boto3
from boto3.s3.transfer import TransferConfig

boto3.client('s3').upload_file(
    Bucket='my-bucket',
    Key='big-file.csv',
    Filename='/tmp/big-file.csv',
    Config=TransferConfig(5*(1024**3)),
    ExtraArgs={'ACL': 'bucket-owner-full-control'},
)

More details here: http://boto3.readthedocs.io/en/latest/guide/s3.html#configuration-settings

ryantuck picture ryantuck on 21 Nov 2017
👍5 👎1

Would love to get an update on this too, as I'm not finding any workarounds.

tomauer picture tomauer on 14 Dec 2017
👍3

Has anyone found any workaround for this? Because not having bucket-owner-full-control condition in bucket policy might cause permissions issue if any user won't attach this header.

sfaiz picture sfaiz on 14 Dec 2017
👍2

You could replace your condition with something like this:
@sfaiz
"Condition": { "StringEqualsIfExists": { "s3:x-amz-acl": "bucket-owner-full-control" }

AbbTek picture AbbTek on 16 Jan 2018
👍1

@AbbTek If the goal of such a policy is to prevent people writing into your bucket when they forget the ACL, wouldn't using IfExists mean that a simple aws s3 cp x s3://dest without any --acl upload, thus not actually enforcing the ACL?

ashb picture ashb on 25 Jan 2018
👍2

AWS confirmed that this is still a bug:

The cp command under the hood initiates a multi part upload for objects larger than 8 MB. I took a look at our API reference for upload part and noticed that the UploadPart API cannot pass any x-amz- headers with the request, hence, it cannot pass the x-amz- bucket-owner-full-control which ends up denying the request due to the bucket policy only allowing puts with x-amz header bucket-owner-full-control.

Workaround is to avoid multi-part uploads, by either:

  • Increase your triggering limit for s3 cp command from the default 8MB to higher, e.g. 5GB (see https://docs.aws.amazon.com/cli/latest/topic/s3-config.html)
  • Use aws s3api put-object instead - it doesn't automatically switch to multi-part uploads.

Multi-part uploads are auto-enabled in s3 cp for performance and fault-tolerancy.

tuukkamustonen picture tuukkamustonen on 14 Jun 2018
👍4

I was getting a multi-part upload error so I found this thread and assumed that was my problem. I have a very restrictive, write only policy for the backup task. I did not increase the default upload size or make any global configuration changes, but I found and added permissions related to multipart and this policy is now working for me. 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:PutEncryptionConfiguration",
        "s3:PutObject",
        "s3:ListBucketMultipartUploads",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "*"
    }
  ]
}

I think maybe the problem discussed in this thread is fixed. My pip list shows:

awscli          1.15.63
botocore        1.10.62
s3transfer      0.1.13

(As an aside: ECS network mode awsvpc was failing to reach S3, while ECS network mode host worked fine. Fargate seems like a perfect use case for backing up databases to S3, but oh well.)

normangilmore picture normangilmore on 21 Jul 2018
👍1

I think maybe the problem discussed in this thread is fixed.

Indeed, it should be fixed. A few weeks ago they pushed fix for this. It took some days to propagate on all regions but should probably be in now.

It was a fix on AWS server/API side, nothing was changed (AFAIK) in client libraries.

tuukkamustonen picture tuukkamustonen on 23 Jul 2018
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ypant picture ypant  路  3Comments
hapx101 picture hapx101  路  3Comments
KimberleySDU picture KimberleySDU  路  3Comments
maanbsat picture maanbsat  路  3Comments
levequej picture levequej  路  3Comments