Azure-cli: Azure CLI commands return character '[0m' at the end of the command output

[jlausuch]

az feedback auto-generates most of the information requested below, as of CLI version 2.0.68

Describe the bug
I get the char sequence '[0m' at the end of the command output.

To Reproduce
any az command.
It doesn't happen if the command is pipped or redirected to a file.

Expected behavior
It should not print that character.

Environment summary

 sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
 zypper addrepo --name "Azure CLI" --check https://packages.microsoft.com/yumrepos/azure-cli azure-cli
 zypper in --from azure-cli -y azure-cli

Additional context

$ az -v

azure-cli                         2.0.68

command-modules-nspkg               2.0.3
core                              2.0.68
nspkg                              3.0.4
telemetry                          1.0.3

Python location '/usr/bin/python2'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 2.7.14 (default, Nov 30 2017, 11:33:17) [GCC]

Legal docs and information: aka.ms/AzureCliLegal


Your CLI is up-to-date.

[marstr]

Howdy jlausuch, is this new to 2.0.68 or has this behavior been present for older versions as well?

I suspect that the terminal you're using isn't playing nice with our colorizer. It would explain why an unexpected control character is being output, and also why it doesn't happen when you're not using an interactive terminal session.

[jlausuch]

It didn't happen with version 2.0.64 actually.

[charlesguoooo]

I have the same problem in pycharm.The version is 2.0.69.

[marstr]

@charlesguoooo, is your platform also SUSE?

[charlesguoooo]

@marstr no, It was happened in pycharm on mac. look this

#!/usr/bin/env python

import re
import json
import subprocess

result = subprocess.check_output(["az account show"],shell=True)

print json.dumps(result)

# return result include \u001b[0m in the end

But if not in pycharm,Just run python test.py,the return result is right

[marstr]

I'm going to unassign myself, as this does not relate to the Packaging as originally believed. @zikalino, @yugangw-msft you guys may want to re-triage or re-assign this.

[kh-sonrai]

Any updates on this issue? Still seeing this problem with az cli version 2.0.74 on mac when running in pycharm.
@zikalino @yugangw-msft

[alanjcastonguay]

Also observing trailing escape character in 2.0.70 in IntelliJ IDEA 2019.2 with above snippet, with or without --output json. Presumably there's some automagical terminal-detection thing going on here?

[jiasli]

\x1b[0m is appended by colorama to reset the terminal color. @jlausuch, which terminal you are using? Bash and zsh should work fine with this ANSI escape code.

[jiasli]

@charlesguoooo @kh-sonrai , the issue on PyCharm isn't limited to CLI. We can test by creating printcolor.py with only init():

from colorama import init
init()

Then call it in caller.py with PyCharm:

import subprocess

result = subprocess.check_output("<python_path>\\python.exe <absolute_path>\\printcolor.py")
decoded = result.decode(encoding='UTF-8')
print(decoded)

The actual output is empty but when debugging into it, we can see

result = {bytes} b'\x1b[0m'

This is because colorama registers reset_all for atexit which writes \x1b[0m at the end of stdout. https://github.com/tartley/colorama/blob/d7a538212c3b8d00dc0051a45f99dc9b201dd3b2/colorama/initialise.py#L45-L48

[Arnavion]

To give some urgency to this issue, note that it breaks scripting. For example piping the result through jq causes jq to complain because the trailing escape sequence is not valid JSON.

(Workaround: az ... | sed -e 's/\x1b\[0m$//' | ...)

[jiasli]

@Arnavion, what is the command you are running and what is your shell?

This command works good for me:

az vm list | jq '.[].name'

[Arnavion]

$ az vm list | jq '.[].name'

"vm-1"
      parse error: Invalid numeric literal at line 91, column 2

$ az vm list | tail -c 16 | xxd

00000000: 6c6c 0d0a 2020 7d0d 0a5d 0d0a 1b5b 306d  ll..  }..]...[0m

This is with mcr.microsoft.com/azure-cli:latest, which is currently :2.0.79. The shell is /bin/sh (since the docker image is based on alpine).

[jiasli]

We are not able to reproduce with docker on Mac.

docker run -it --rm mcr.microsoft.com/azure-cli:latest
az vm list | jq '.[].name'
az vm list | tail -c 16 | xxd
00000000: 6573 223a 206e 756c 6c0a 2020 7d0a 5d0a  es": null.  }.].

Are you running docker on Windows, Linux or Mac? If you run az vm list directly, is any colored output shown?

[Arnavion]

Okay, I know what the problem is.

My jq / tail commands are outside the Docker container. ie my command is doing the equivalent of docker run -it --rm mcr.microsoft.com/azure-cli az vm list | jq , because my az is a shell function that effectively wraps docker run -it --rm mcr.microsoft.com/azure-cli az "$@". The az program inside the container thinks its stdout is not being redirected in this case, so it includes the escape codes in its output.

The -t that I'm passing to the docker run is why az thinks its output is not being redirected. If I take the -t out then it stops including the escape codes as desired. I only noticed now because it hadn't been printing escape codes in the older versions.

Sorry for the false alarm.

[jiasli]

Thanks for the analysis. Per docker's doc,

-t : Allocate a pseudo-tty

This makes colorama think this is a tty and thus send the escaping chars.

[wouterhund]

Why does the azure-cli even output control characters when I request json output? And how can I turn this off? This is breaking my scripts

[marcin-vt]

It is still happening in az cli 2.5.1 (only in pycharm) and actually it is breaking DefaultAzureCredential (azure-identity 1.4.0b2) as it cannot parse token from az output (json.decoder.JSONDecodeError: Extra data)

[jiasli]

@marcin-vt, could you try disabling the color following #12601 and see if it still happens?

[marcin-vt]

Thnx @jiasli - that helped!

[yonzhan]

@marcin-vt was your issue resolved?

[marcin-vt]

Mine? Yes, adding the environmental variable solved the issue.

[jiasli]

I have created issue https://github.com/tartley/colorama/issues/263 on colorama's repo.