Azure-cli: Getting 'UnicodeEncodeError' on CentOS and Python 2.7

[dgard1981]

Describe the bug
Using any az acr command produces a 'UnicodeEncodeError'. Other 'az' commands work without issue.

To Reproduce
Either of the following examples produces the exact same error
az acr list
az acr login --name $ACRNAME

Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
msg = self.format(record)
File "/usr/lib64/az/lib/python2.7/site-packages/knack/log.py", line 78, in format
msg = logging.StreamHandler.format(self, record)
File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
return fmt.format(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 467, in format
s = self._fmt % record.__dict__
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 215: ordinal not in range(128)

Expected behavior
The expected behavior for the given examples is to list all ACR's in the current subscription, and then log in to the given ACR. The responses to these commands should be as described in the linked documentation.

Environment summary

• Install method: sudo yum install azure-cli
• CLI version: 2.0.55 (acr version 2.1.12).
• OS version: CentOS 7.3.1611 (bash)

Full CLI version output

azure-cli (2.0.55)

acr (2.1.12)
acs (2.3.14)
advisor (2.0.0)
ams (0.3.2)
appservice (0.2.11)
backup (1.2.1)
batch (3.4.1)
batchai (0.4.6)
billing (0.2.0)
botservice (0.1.4)
cdn (0.2.0)
cloud (2.1.0)
cognitiveservices (0.2.4)
command-modules-nspkg (2.0.2)
configure (2.0.20)
consumption (0.4.2)
container (0.3.11)
core (2.0.55)
cosmosdb (0.2.7)
dla (0.2.3)
dls (0.1.7)
dms (0.1.1)
eventgrid (0.2.0)
eventhubs (0.3.2)
extension (0.2.3)
feedback (2.1.4)
find (0.2.13)
hdinsight (0.2.0)
interactive (0.4.1)
iot (0.3.4)
iotcentral (0.1.5)
keyvault (2.2.9)
lab (0.1.5)
maps (0.3.3)
monitor (0.2.8)
network (2.3.0)
nspkg (3.0.3)
policyinsights (0.1.0)
profile (2.1.2)
rdbms (0.3.5)
redis (0.3.2)
relay (0.1.2)
reservations (0.4.1)
resource (2.1.8)
role (2.3.0)
search (0.1.1)
security (0.1.0)
servicebus (0.3.2)
servicefabric (0.1.11)
signalr (1.0.0)
sql (2.1.7)
storage (2.3.0)
telemetry (1.0.0)
vm (2.2.12)

Python location '/usr/lib64/az/bin/python'
Extensions directory '/home/dgard/.azure/cliextensions'

Python (Linux) 2.7.5 (default, Nov  6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

Additional context
This problem seems to be related specifically to the acr subset of commands. Other az commands work without error.

[ixxie]

Running into this as well in a similar environment (CentOS 7 and Python 2.7)

[tjprescott]

@djyou for comment

[djyou]

We invoke the SDK method directly for az acr list https://github.com/Azure/azure-cli/blob/ed7513985deb6f66205ed33a34320917eecda1ff/src/command_modules/azure-cli-acr/azure/cli/command_modules/acr/custom.py#L44

The error seems to be thrown from https://github.com/Azure/azure-cli/blob/ed7513985deb6f66205ed33a34320917eecda1ff/src/command_modules/azure-cli-acr/azure/cli/command_modules/acr/custom.py#L34

So I suppose all commands who invoke getLogger will end up in \logging\__init__.py and throw the exception.

sys.getdefaultencoding() is ascii for Python 2. Changing it may help but may also introduce other side effects. I would recommend upgrading to Python 3 if possible, or see if @tjprescott has other recommendations.

/cc @sajayantony

[ixxie]

I considered upgrading to Python 3, but it seems the installation process is quite involved - see https://github.com/Azure/azure-cli/issues/2529. Providing Python 3 packages might help with this, and I guess would also be handy for the migrations as Python 2 becomes increasingly deprecated in various distros.

[dgard1981]

@djyou I've installed 'virtualenv' as suggested in this issue, but it didn't work; even in the virtualenv with Python 3.6 installed, the CLI still tried to use Python 2.7.

However, now that I'm out of the virtualenv I notice that installing it seems to have completely broken the CLI. Thus, I would recommend that the linked issue is either updated or deleted so that others don't run in to issues.

[dgard@localhost ~]$ az --version
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/azure/cli/__main__.py", line 32, in <module>
    az_cli = get_default_cli()
  File "/usr/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 515, in get_default_cli
    help_cls=AzCliHelp)
  File "/usr/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 60, in __init__
    register_ids_argument(self)  # global subscription must be registered first!
  File "/usr/lib/python2.7/site-packages/azure/cli/core/commands/arm.py", line 180, in register_ids_argument
    from msrestazure.tools import parse_resource_id, is_valid_resource_id
  File "/usr/lib/python2.7/site-packages/msrestazure/__init__.py", line 28, in <module>
    from .azure_configuration import AzureConfiguration
  File "/usr/lib/python2.7/site-packages/msrestazure/azure_configuration.py", line 34, in <module>
    from msrest import Configuration
  File "/usr/lib/python2.7/site-packages/msrest/__init__.py", line 28, in <module>
    from .configuration import Configuration
  File "/usr/lib/python2.7/site-packages/msrest/configuration.py", line 35, in <module>
    from typing import TYPE_CHECKING, Optional, Dict, List, Any, Callable, Union  # pylint: disable=unused-import
ImportError: No module named typing

I tried to reinstall, but saw Requirement already satisfied for all packages. I had to uninstall and then reinstall the CLI to get it to work again.

[djyou]

@tjprescott Can we get some assistance from the CLI team regarding this issue? Basically certain combinations of OS and Python 2 have this default encoding problem, which potentially affects many command modules and the code base is outside of acr.

[tjprescott]

@djyou I will take a look, though @dgard1981 reports it is only affecting the ACR commands.

[tjprescott]

Running on with CentOS 7.6.1810 (Red Hat 4.8.5-36) and Python 2.7.5, I do not encounter any issue with the ACR commands.

Virtual environments only apply if you are doing a pip install, not yum installation. Do you have existing registries in your subscription (ie. can you see them from portal)?

[tjprescott]

@marstr do you have any ideas on what might be causing this? Both users are on CentOS and Python 2.7. My only thought is that maybe some metadata in an existing registry has Unicode characters, but I'm not able to repro.

[ixxie]

Its worth noting the similarity to https://github.com/Azure/azure-cli/issues/6408 - perhaps the solution can be borrowed from there.

[djyou]

In the other issue acr has a format that's causing the problem and the workaround was to not format the string. There wasn't a workaround to print the string with Python 2 in some OSs. We ended up asking users to upgrade to Python 3 as well.

This specific case, as mentioned in https://github.com/Azure/azure-cli/issues/8312#issuecomment-458332856, the static logger is causing the problem so none of the commands in the file would work.

[dgard1981]

@tjprescott Yes I do already have a Container Registry, created via the Portal.

Additionally, I know I said this was only affecting acr commands originally, but I'm starting to find it with other commands now too - az functionapp list, az aks list, az storage account list are three such examples. Note that I obviously haven't tried every single command.

I can't speak for functionapp and aks commands, but storage account commands have been working in the past. I will attempt to downgrade the CLI when I get the opportunity, to see if I can tie the issue to a specific release.

[tjprescott]

Thanks for the update @dgard1981. I will remove the ACR label since this is affecting other commands.

[marstr]

I'm having trouble reproing this with 2.0.57.

When I do the following, everything seems to work as expected:

docker run -it centos:centos7 /bin/bash
rpm --import https://packages.microsoft.com/keys/microsoft.asc
sh -c 'echo -e "[azure-cli]\nname=Azure CLI\nbaseurl=https://packages.microsoft.com/yumrepos/azure-cli\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/azure-cli.repo'
yum install -y azure-cli
az login
# Follow instructions to authenticate
az acr list
python --version
rpm -q centos-release

Note: The last two commands tell me that the container is using Python 2.7.5 and centos-release-7-6.1810.2.el7.centos.x86_64.

[marstr]

I'm going to close this out for now. If you can help us get a reliable repro, we'll be happy to open it back up. :)

[bartoostlander]

Hi Team, I get the error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 172: ordinal not in range(128)
Logged from file util.py, line 28

when I execute "az storage blob exists". This is on Red Hat Enterprise Linux Server release 7.6 (Maipo) running Python 2.7.5 on a server in Azure SouthCentral and an identical one in Azure West. What can I do to fix it?

[medved-nsk]

Hi,
We have the same error on CentOS 7.6.1810, default Python 2.7.5 and _azure-cli 2.0.60-1.el7_ installed by yum

$ az acr login --name RegistryName
'ascii' codec can't encode character u'\xa0' in position 172: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py", line 206, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 328, in execute
    raise ex
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 172: ordinal not in range(128)

Any chances it'll be fixed or the only way is to use Python 3? Thanks.

[martynakolaczek]

I have exactly the same issue, when running following command:
az appservice plan create

on:
Centos: 7.2.1511 (Core)
Python: 2.7.5
azure-cli 2.0.64

I tried above solutions, with no luck - as mentioned above this broke my existing azure-cli installation.
What is the solution for this issue (as I can see that it has been closed)?

UPDATE: I was able to install everything on fresh machine and this time azure-cli worked fine. Working configuration is:
Centos: 7.7 (fresh)
Python: 2.7.5
azure-cli 2.0.75