Aws-cli: Unicode characters in s3 command: 'ascii' codec can't encode characters in position 32-33: ordinal not in range(128)

Created on 3 Aug 2015  ·  19Comments  ·  Source: aws/aws-cli

I've seen similar questions asked for different aws-cli products, and the answer always has to do with the locale, though the asker, like myself, generally has their locale in order.

> locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
> aws s3 cp "s3://my-bucket/사본.jpg" /tmp/outfile.jpg
'ascii' codec can't encode characters in position 5-6: ordinal not in range(128)
Completed 1 part(s) with ... file(s) remaining
> aws --version
aws-cli/1.7.42 Python/2.7.6 Linux/3.13.0-61-generic

How can I force using a different codec instead of ascii?

locale s3 unicode
Source
picture timmytofu
👍2

Most helpful comment

Running into a similar problem trying to use aws s3 ls s3://$BUCKET/ : 

'ascii' codec can't encode character u'\xe9' in position 181: ordinal not in range(128)

my locale is 'en_CA.UTF-8'
aws-cli/1.11.44 Python/2.6.6 Linux/2.6.32-042stab093.4 botocore/1.5.7
(using CentOS 6.9 + cPanel, what the webhosting company gave me to use.)

Could not reproduce on another machine that is using Python 2.7 and Ubuntu 14.04 (instead of webhost-enforced CentOS + cPanel).

picture moses-moore-spafax on 14 Jun 2017
👍2

All 19 comments

Can we get a --debug log of this? It'd be helpful to see where that encoding error is coming from. I'm not able to repro.

$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
$ touch  사본.jpg
$ aws s3 cp  사본.jpg s3://bucket/사본.jpg
upload: ./사본.jpg to s3://bucket/사본.jpg

$ aws s3 cp s3://bucket/사본.jpg downloaded.jpg
download: s3://bucket/사본.jpg to ./downloaded.jpg
jamesls picture jamesls on 10 Aug 2015

Here you go. It looks like it's coming back as a 404, though it's clearly there, so maybe it's a deeper encoding issue on S3, and the ascii encoding bit is from an attempt to output an error message with ascii encoding, rather than coming from the copy? I can give you access to that bucket if you'd like.

tim 2015Aug11 12:44:34> aws s3 ls s3://qwer-bucket/
2015-08-11 12:11:55    5175268 PpK4v2SSGoGndguk1YgQ_Page1 ᄉᄇ.jpg
tim 2015Aug11 12:52:17> aws --debug s3 cp "s3://qwer-bucket/PpK4v2SSGoGndguk1YgQ_Page1 ᄉᄇ.jpg" /tmp/qwer 2> /tmp/err
Completed 1 part(s) with ... file(s) remaining
[1] tim 2015Aug11 12:52:54> cat /tmp/err
2015-08-11 12:52:54,125 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.7.42 Python/2.7.6 Linux/3.13.0-61-generic, botocore version: 1.1.5
2015-08-11 12:52:54,125 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 's3', 'cp', 's3://qwer-bucket/PpK4v2SSGoGndguk1YgQ_Page1 \xe1\x84\x89\xe1\x84\x87.jpg', '/tmp/qwer']
2015-08-11 12:52:54,125 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f417962da28>
2015-08-11 12:52:54,126 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider at 0x7f417985e7d0>
2015-08-11 12:52:54,126 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7f41798695f0>
2015-08-11 12:52:54,126 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,126 - MainThread - botocore.hooks - DEBUG - Event building-command-table.cp: calling handler <function add_waiters at 0x7f41798695f0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7f41796348d0>
2015-08-11 12:52:54,128 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,128 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7f41796348d0>
2015-08-11 12:52:54,129 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7f41796348d0>
2015-08-11 12:52:54,129 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7f41796348d0>
2015-08-11 12:52:54,129 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler <function uri_param at 0x7f41799f18c0>
2015-08-11 12:52:54,129 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthand object at 0x7f41796348d0>
2015-08-11 12:52:54,129 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-08-11 12:52:54,129 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2015-08-11 12:52:54,129 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2015-08-11 12:52:54,129 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2015-08-11 12:52:54,129 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2015-08-11 12:52:54,130 - MainThread - botocore.credentials - INFO - Credentials found in config file: ~/.aws/config
2015-08-11 12:52:54,146 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2015-08-11 12:52:54,147 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f4179df6398>
2015-08-11 12:52:54,147 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f4179df2320>
2015-08-11 12:52:54,149 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2015-08-11 12:52:54,150 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f4179df6398>
2015-08-11 12:52:54,150 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f4179df2320>
2015-08-11 12:52:54,151 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2015-08-11 12:52:54,151 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f4179df6398>
2015-08-11 12:52:54,151 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f4179df2320>
2015-08-11 12:52:54,153 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608
2015-08-11 12:52:54,153 - MainThread - awscli.customizations.s3.executor - DEBUG - Using max queue size for s3 tasks of: 1000
2015-08-11 12:52:54,153 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608
2015-08-11 12:52:54,153 - MainThread - awscli.customizations.s3.executor - DEBUG - Using max queue size for s3 tasks of: 2
2015-08-11 12:52:54,156 - MainThread - awscli.customizations.s3.executor - DEBUG - Using a threadpool size of: 10
2015-08-11 12:52:54,159 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7f4179dd10c8>
2015-08-11 12:52:54,161 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7f4179dd1398>
2015-08-11 12:52:54,161 - MainThread - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x7f4178fe83d0> (verify_ssl=True) with params: {'body': '', 'url': u'https://s3.amazonaws.com/qwer-bucket/PpK4v2SSGoGndguk1YgQ_Page1%20%E1%84%89%E1%84%87.jpg', 'headers': {'User-Agent': 'aws-cli/1.7.42 Python/2.7.6 Linux/3.13.0-61-generic'}, 'query_string': {}, 'url_path': u'/qwer-bucket/PpK4v2SSGoGndguk1YgQ_Page1%20%E1%84%89%E1%84%87.jpg', 'method': u'HEAD'}
2015-08-11 12:52:54,161 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.HeadObject: calling handler <bound method S3._sign_request of <botocore.client.S3 object at 0x7f41790302d0>>
2015-08-11 12:52:54,161 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7f4179f0aaa0>
2015-08-11 12:52:54,161 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/qwer-bucket/PpK4v2SSGoGndguk1YgQ_Page1%20%E1%84%89%E1%84%87.jpg
2015-08-11 12:52:54,162 - MainThread - botocore.utils - DEBUG - URI updated to: https://qwer-bucket.s3.amazonaws.com/PpK4v2SSGoGndguk1YgQ_Page1%20%E1%84%89%E1%84%87.jpg
2015-08-11 12:52:54,162 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2015-08-11 12:52:54,162 - MainThread - botocore.auth - DEBUG - HTTP request method: HEAD
2015-08-11 12:52:54,162 - MainThread - botocore.auth - DEBUG - StringToSign:
HEAD


Tue, 11 Aug 2015 16:52:54 GMT
/qwer-bucket/PpK4v2SSGoGndguk1YgQ_Page1%20%E1%84%89%E1%84%87.jpg
2015-08-11 12:52:54,166 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [HEAD]>
2015-08-11 12:52:54,166 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): qwer-bucket.s3.amazonaws.com
2015-08-11 12:52:54,295 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "HEAD /PpK4v2SSGoGndguk1YgQ_Page1%20%E1%84%89%E1%84%87.jpg HTTP/1.1" 404 0
2015-08-11 12:52:54,296 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'N7fEqkWNsocYuu1Ouw9DkEeqHv8Lang1RNkpyCv281ez/nccpojj8n1SyKfzDkBV', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': '422091AA6C35295D', 'date': 'Tue, 11 Aug 2015 16:52:52 GMT', 'content-type': 'application/xml'}
2015-08-11 12:52:54,296 - MainThread - botocore.parsers - DEBUG - Response body:

2015-08-11 12:52:54,296 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f41790b8250>
2015-08-11 12:52:54,296 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2015-08-11 12:52:54,296 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <function enhance_error_msg at 0x7f41798546e0>
2015-08-11 12:52:54,296 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <awscli.errorhandler.ErrorHandler object at 0x7f4179644510>
2015-08-11 12:52:54,296 - MainThread - awscli.errorhandler - DEBUG - HTTP Response Code: 404
2015-08-11 12:52:54,297 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Exception caught during task execution: 'ascii' codec can't encode characters in position 32-33: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/s3handler.py", line 97, in call
    total_files, total_parts = self._enqueue_tasks(files)
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/s3handler.py", line 175, in _enqueue_tasks
    for filename in files:
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/fileinfobuilder.py", line 31, in call
    for file_base in files:
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/filegenerator.py", line 138, in call
    for src_path, size, last_update in file_list:
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/filegenerator.py", line 279, in list_objects
    yield self._list_single_object(s3_path)
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/filegenerator.py", line 323, in _list_single_object
    raise ClientError(**copy_fields)
  File "/usr/local/lib/python2.7/dist-packages/awscli/errorhandler.py", line 27, in __init__
    error_type=error_type, operation_name=operation_name)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 32-33: ordinal not in range(128)
2015-08-11 12:52:54,300 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,301 - Thread-5 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,301 - Thread-1 - awscli.customizations.s3.executor - DEBUG - Received print task: PrintTask(message="'ascii' codec can't encode characters in position 32-33: ordinal not in range(128)", error=True, total_parts=None, warning=None)
'ascii' codec can't encode characters in position 32-33: ordinal not in range(128)
2015-08-11 12:52:54,301 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,301 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,301 - Thread-6 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,301 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,301 - Thread-7 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,301 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,301 - Thread-8 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,302 - Thread-9 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,302 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,302 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,302 - Thread-10 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,302 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,302 - Thread-11 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,302 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,302 - Thread-12 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,302 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 1)
2015-08-11 12:52:54,302 - Thread-13 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,302 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-5, stopped daemon 139919087363840)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-5, stopped daemon 139919087363840)>
2015-08-11 12:52:54,303 - Thread-14 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-6, stopped daemon 139919078971136)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-6, stopped daemon 139919078971136)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-7, stopped daemon 139919070578432)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-7, stopped daemon 139919070578432)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-8, stopped daemon 139919062185728)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-8, stopped daemon 139919062185728)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-9, stopped daemon 139919053793024)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-9, stopped daemon 139919053793024)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-10, stopped daemon 139919045400320)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-10, stopped daemon 139919045400320)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-11, stopped daemon 139918558885632)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-11, stopped daemon 139918558885632)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-12, stopped daemon 139918550492928)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-12, stopped daemon 139918550492928)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-13, stopped daemon 139918542100224)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-13, stopped daemon 139918542100224)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-14, stopped daemon 139918533707520)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-14, stopped daemon 139918533707520)>
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for result thread.
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for IO thread.
2015-08-11 12:52:54,303 - Thread-1 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in print thread, shutting down print thread.
2015-08-11 12:52:54,303 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for result thread to shutdown.
2015-08-11 12:52:54,303 - Thread-2 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in IO thread, shutting down.
2015-08-11 12:52:54,304 - MainThread - awscli.customizations.s3.executor - DEBUG - Waiting for IO thread to shutdown.
2015-08-11 12:52:54,304 - MainThread - awscli.customizations.s3.executor - DEBUG - All threads have been shutdown.
timmytofu picture timmytofu on 11 Aug 2015
👍1

I have this exact same issue... A lot

nagualcode picture nagualcode on 18 Dec 2015

I have the same issue which is preventing me to archive my S3 bucket. :(

download failed: s3://s.natalian.org/2014-06-19/Kai Hendry\ufffd\ufffd\ufffds _IMG_2955.JPG to s.natalian.org/2014-06-19/Kai Hendry\ufffd\ufffd\ufffds _IMG_2955.JPG 'ascii' codec can't encode characters in position 45-47: ordinal not in range(128)


'ascii' codec can't encode characters in position 45-47: ordinal not in range(128)
nuc:/mnt/2tb$ locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=
kaihendry picture kaihendry on 28 Jan 2016

Where are you using it?
In my case, was an embed computer, with a minial install of dedbian... And
i fixed this error adding "export "loall the locale settings" im the
begginung of the script I Had inside crontab, that started the awscli
jobs... So basically you install the utf8-us locale, and set it at the
begining of your script

On Thursday, 28 January 2016, Kai Hendry [email protected] wrote:

I have the same issue which is preventing me to archive my S3 bucket. :(

download failed: s3://s.natalian.org/2014-06-19/Kai Hendryufffdufffdufffds _IMG_2955.JPG to s.natalian.org/2014-06-19/Kai Hendryufffdufffdufffds _IMG_2955.JPG 'ascii' codec can't encode characters in position 45-47: ordinal not in range(128)

'ascii' codec can't encode characters in position 45-47: ordinal not in range(128)
nuc:/mnt/2tb$ locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

—
Reply to this email directly or view it on GitHub
https://github.com/aws/aws-cli/issues/1437#issuecomment-176117906.

nagualcode picture nagualcode on 28 Jan 2016

Actually once I setup locale, it now seems to work!

nuc:/mnt/2tb$ localectl
   System Locale: LANG=en_GB.UTF-8
       VC Keymap: n/a
      X11 Layout: n/a

Thank you

kaihendry picture kaihendry on 28 Jan 2016
👎1 👍1

I'm having the same issue on mac

aop picture aop on 1 Mar 2017

my advice? forget awscli and use rclone.org when possible.

nagualcode picture nagualcode on 1 Mar 2017
👍1

Running into a similar problem trying to use aws s3 ls s3://$BUCKET/ : 

'ascii' codec can't encode character u'\xe9' in position 181: ordinal not in range(128)

my locale is 'en_CA.UTF-8'
aws-cli/1.11.44 Python/2.6.6 Linux/2.6.32-042stab093.4 botocore/1.5.7
(using CentOS 6.9 + cPanel, what the webhosting company gave me to use.)

Could not reproduce on another machine that is using Python 2.7 and Ubuntu 14.04 (instead of webhost-enforced CentOS + cPanel).

moses-moore-spafax picture moses-moore-spafax on 14 Jun 2017
👍2

Cannot delete a certain object with a "ş" in its name:

aws s3 rm "s3://xxx/eleştirinin" --region eu-west-2

Returns with error code 0 but neither an error message, nor the expected confirmation. File is then still there. The AWS Console seems behave the same (no error, no confirmation, file remains in bucket).

With a slightly older CLI version I used to get the following error message:

fatal error: 'ascii' codec can't encode character u'\u015f' in position 71: ordinal not in range(128)

Now having upgraded to "aws-cli/1.11.113 Python/2.7.10 Darwin/16.6.0 botocore/1.5.76", I'm not getting anything back unless I use --debug...

[...]
2017-06-30 08:11:05,198 - Thread-3 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2017-06-30 08:11:05,198 - Thread-1 - awscli.customizations.s3.results - DEBUG - Error processing result SuccessResult(transfer_type='delete', src=u's3://xxx/ele\u015ftirinin', dest=None) with handler <awscli.customizations.s3.results.ResultRecorder object at 0x10bb94950>: 'ascii' codec can't encode character u'\u015f' in position 34: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 604, in _process_result
    result_handler(result)
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 241, in __call__
    result=result)
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 318, in _record_success_result
    self._pop_result_from_ongoing_dicts(result)
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 253, in _pop_result_from_ongoing_dicts
    ongoing_key = self._get_ongoing_dict_key(result)
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 250, in _get_ongoing_dict_key
    str(el) for el in [result.transfer_type, result.src, result.dest])
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 250, in <genexpr>
    str(el) for el in [result.transfer_type, result.src, result.dest])
UnicodeEncodeError: 'ascii' codec can't encode character u'\u015f' in position 34: ordinal not in range(128)
2017-06-30 08:11:05,198 - Thread-1 - awscli.customizations.s3.results - DEBUG - Error processing result SuccessResult(transfer_type='delete', src=u's3://xxx/ele\u015ftirinin', dest=None) with handler <awscli.customizations.s3.results.ResultPrinter object at 0x10bb949d0>: 'ascii' codec can't encode character u'\u015f' in position 34: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 604, in _process_result
    result_handler(result)
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 412, in __call__
    result=result)
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 429, in _print_success
    transfer_location=self._get_transfer_location(result)
  File "/usr/local/Cellar/awscli/1.11.113/libexec/lib/python2.7/site-packages/awscli/customizations/s3/results.py", line 464, in _get_transfer_location
    return self.SRC_TRANSFER_LOCATION_FORMAT.format(src=result.src)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u015f' in position 34: ordinal not in range(128)
2017-06-30 08:11:05,204 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

PS: I only managed to upload the file using aws s3 cp – sync always failed when used with the --include filename option.

Manc picture Manc on 30 Jun 2017

You can reproduce this issue by running the following verbatim (just copy and paste, src_bucket, target_bucket, etc don't have to exist)

aws s3 sync 's3://src_bucket/dir1/' 's3://target_bucket/dir1' --exclude '*'  --include '你好吗.txt'

Debug: 

2017-08-21 17:28:39,839 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/aws/local/lib/python2.7/site-packages/awscli/clidriver.py", line 200, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/aws/local/lib/python2.7/site-packages/awscli/customizations/commands.py", line 190, in __call__
    parsed_globals)
  File "/usr/local/aws/local/lib/python2.7/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/usr/local/aws/local/lib/python2.7/site-packages/awscli/customizations/s3/subcommands.py", line 688, in _run_main
    return cmd.run()
  File "/usr/local/aws/local/lib/python2.7/site-packages/awscli/customizations/s3/subcommands.py", line 1025, in run
    'filters': [create_filter(self.parameters),
  File "/usr/local/aws/local/lib/python2.7/site-packages/awscli/customizations/s3/filters.py", line 50, in create_filter
    return Filter(real_filters, src_rootdir, dst_rootdir)
  File "/usr/local/aws/local/lib/python2.7/site-packages/awscli/customizations/s3/filters.py", line 93, in __init__
    self.patterns = self._full_path_patterns(patterns, rootdir)
  File "/usr/local/aws/local/lib/python2.7/site-packages/awscli/customizations/s3/filters.py", line 103, in _full_path_patterns
    (pattern[0], os.path.join(rootdir, pattern[1])))
  File "/usr/local/aws/lib/python2.7/posixpath.py", line 71, in join
    path +=  b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)
2017-08-21 17:28:39,848 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Line that caused the bug:

        for pattern in original_patterns:
            full_patterns.append(
                (pattern[0], os.path.join(rootdir, pattern[1])))

aws --version
aws-cli/1.11.136 Python/2.7.12 Linux/4.4.0-92-generic botocore/1.6.3

locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

This issue occurs on Ubuntu 14.04 and 16.04.

mliang2 picture mliang2 on 21 Aug 2017
👍1

same problem:

# aws s3 ls s3://sa-prod-data/ 
'ascii' codec can't encode characters in position 31-40: ordinal not in range(128)

# aws --version
aws-cli/1.11.147 Python/2.6.6 Linux/2.6.32-642.el6.x86_64 botocore/1.7.5

# cat /etc/centos-release 
CentOS release 6.8 (Final)

# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
vbem picture vbem on 7 Sep 2017

Same here, with "iam get-user --user-name goo" on 16.04 LTS, pip install'd with --user

/usr/lib/python2.7/argparse.py:2140: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
if option_string == short_option_prefix:
Traceback (most recent call last):
File "/home/tchiverton/.local/bin/aws", line 27, in
sys.exit(main())
File "/home/tchiverton/.local/bin/aws", line 23, in main
return awscli.clidriver.main()
File "/home/tchiverton/.local/lib/python2.7/site-packages/awscli/clidriver.py", line 56, in main
return driver.main()
File "/home/tchiverton/.local/lib/python2.7/site-packages/awscli/clidriver.py", line 192, in main
parsed_args, remaining = parser.parse_known_args(args)
File "/home/tchiverton/.local/lib/python2.7/site-packages/awscli/argparser.py", line 91, in parse_known_args
parsed, remaining = super(CLIArgParser, self).parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib/python2.7/argparse.py", line 1774, in _parse_known_args
option_tuple = self._parse_optional(arg_string)
File "/usr/lib/python2.7/argparse.py", line 2082, in _parse_optional
option_tuples = self._get_option_tuples(arg_string)
File "/usr/lib/python2.7/argparse.py", line 2144, in _get_option_tuples
elif option_string.startswith(option_prefix):

tomchiverton picture tomchiverton on 22 Sep 2017

Experiencing a very similar issue where "aws s3 ls" returns "?" characters instead of the proper unicode character

2016-10-19 04:55:39 44984 uploads/editor/documents/data/7/??????????????_??????????_20161011.pdf 2016-10-19 04:56:21 435138 uploads/editor/documents/data/8/??????????_????????.pdf 2017-03-30 03:14:47 54489 uploads/editor/documents/data/9/??????????????_??????????_????????_0228.pdf 2017-09-06 08:09:10 1484173 uploads/summernote_file/file/101/_??_?????_????_2017_?????.pdf 2017-09-06 08:09:22 3156783 uploads/summernote_file/file/102/_???_???_????_???_2017_?????.pdf 2017-09-22 07:14:27 776428 uploads/summernote_file/file/142/_???_?????_2015_?????.pdf 2017-11-02 01:24:49 48738 uploads/summernote_file/file/180/?????_????_?????_????_.pdf 2017-07-03 00:39:51 802836 uploads/summernote_file/file/24/????-??????_????.pdf 2017-07-04 06:24:09 888522 uploads/summernote_file/file/25/?????_??_?_????_201707.pdf 2017-07-18 05:18:03 944292 uploads/summernote_file/file/48/????_??_???_????_??_.pdf

$ locale LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=

lrehmann picture lrehmann on 6 Dec 2017

Same issue when running:

aws cloudwatch put-metric-data --namespace Docker --metric-name DiskPoolSizeUsage --dimensions InstanceId=i-0a0dfe8e13 --value="20.27" -—unit Percent --region eu-west-1

[ec2-user@ip ~]$ aws --version
aws-cli/1.9.1 Python/2.7.10 Linux/4.1.10-17.31.amzn1.x86_64 botocore/1.3.1

[ec2-user@ip ~]$ locale
LANG=en_US.UTF-8
LC_CTYPE=en_GB.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Error:

/usr/lib64/python2.7/argparse.py:2140: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if option_string == short_option_prefix:
Traceback (most recent call last):
  File "/usr/bin/aws", line 27, in <module>
    sys.exit(main())
  File "/usr/bin/aws", line 23, in main
    return awscli.clidriver.main()
  File "/usr/lib/python2.7/dist-packages/awscli/clidriver.py", line 49, in main
    return driver.main()
  File "/usr/lib/python2.7/dist-packages/awscli/clidriver.py", line 176, in main
    parsed_args, remaining = parser.parse_known_args(args)
  File "/usr/lib/python2.7/dist-packages/awscli/argparser.py", line 62, in parse_known_args
    parsed, remaining = super(CLIArgParser, self).parse_known_args(args, namespace)
  File "/usr/lib64/python2.7/argparse.py", line 1733, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib64/python2.7/argparse.py", line 1774, in _parse_known_args
    option_tuple = self._parse_optional(arg_string)
  File "/usr/lib64/python2.7/argparse.py", line 2082, in _parse_optional
    option_tuples = self._get_option_tuples(arg_string)
  File "/usr/lib64/python2.7/argparse.py", line 2144, in _get_option_tuples
    elif option_string.startswith(option_prefix):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1: ordinal not in range(128)
nei picture nei on 28 Mar 2018

Actually, it could be the paste Ive done.. Ive rewritten the text and it worked. :(

nei picture nei on 28 Mar 2018

Rclone solved my backup problem with Portuguese characters / Windows ISO characters

https://rclone.org/s3/

Sync /home/local/directory to the remote bucket, _deleting any excess files in the bucket._

rclone sync /home/local/directory remote:bucket

hebertreis picture hebertreis on 8 Jun 2018
👍1

I solved this problem by configuring aws config:
aws configure
Obvious, but may help to someone new in aws (like me)

x-bogdanovich-x picture x-bogdanovich-x on 19 Apr 2019
👍1

I was able to fix this by dirty hacking aws file:
Replaced this:

if os.environ.get('LC_CTYPE', '') == 'UTF-8':
    os.environ['LC_CTYPE'] = 'en_US.UTF-8'

With this:

#if os.environ.get('LC_CTYPE', '') == 'UTF-8':
os.environ['LC_CTYPE'] = 'en_US.UTF-8'

My locale:

LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
vvrein picture vvrein on 25 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

maanbsat picture maanbsat  ·  3Comments
KimberleySDU picture KimberleySDU  ·  3Comments
rahul003 picture rahul003  ·  3Comments
ronaldpetty picture ronaldpetty  ·  3Comments
schams-net picture schams-net  ·  3Comments