Aws-cli: "'charmap' codec can't encode character" fetching CloudWatch logs and redirecting to a file on Windows 10

[aschmied]

The AWSCLI aborts and writes invalid JSON when I fetch logs containing non-ASCII characters and redirect the output to a file in Windows 10 cmd.exe. This looks related to #1070, but in this case it is log messages causing the problem, not filenames.

Steps

1. Create a test log group and stream:

   aws logs create-log-group --log-group-name anthony-test-group
   aws logs create-log-stream --log-group-name anthony-test-group --log-stream-name anthony-test-stream
   

2. Create a test message that contains a non-ASCII character:

   aws logs put-log-events --log-group-name anthony-test-group --log-stream-name anthony-test-stream --log-events "timestamp=1576691153000,message=Test ✓ message"
   

3. Verify the event was created successfully in the CloudWatch UI

4. Fetch JSON and print output to the console:

   aws logs get-log-events --log-group-name anthony-test-group --log-stream-name anthony-test-stream --output json
   

5. Fetch JSON and redirect output to file:

   aws logs get-log-events --log-group-name anthony-test-group --log-stream-name anthony-test-stream --output json > out.txt
   

Result

When the output is printed to the console the command succeeds. When the output is redirected to a file AWSCLI fails with the following message and exit code ????:

'charmap' codec can't encode character '\u2713' in position 6: character maps to <undefined>

It aborts processing at the non-ASCII character and writes invalid JSON to out.txt:

type out.txt
{
    "events": [
        {
            "timestamp": 1576691153000,
            "message":

The result is similar for "table" and "text" output formats.

Clean Up

aws logs delete-log-stream --log-group-name anthony-test-group --log-stream-name anthony-test-stream
aws logs delete-log-group --log-group-name anthony-test-group

[aschmied]

The linked branch provides a workaround: it adds a new output format ascii-json that escapes the non-ASCII UTF-8 in the output. I'm happy to PR this if it is an acceptable change.

[kdaily]

Hi @aschmied, the CLI uses whatever the system encoding is set to be. I tested this on Windows 10 by changing the system locale to use UTF-8 (see this Stack Overflow answer) and both cases were successful. Does this solution work for you?

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[kdaily]

If using AWS CLI v1 you can:

set PYTHONIOENCODING=UTF-8

to get text that is redirected to use UTF-8.