Azure-cli: Getting Unicode

[bartoostlander]

Hi Team, I get the error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 172: ordinal not in range(128)
Logged from file util.py, line 28

when I execute "az storage blob exists".

To reproduce

export AZURE_STORAGE_ACCOUNT="xxxxxxx"
export AZURE_STORAGE_ACCESS_KEY="yyyyyyy"
az storage blob exists --container-name mycontainer --name node_1.lock

Expected behavior
In the past, this returned an object or string with _true_ or _false_ in it. Since early this year it throws a UnicodeEncodeError error in util.py.

Environment summary
azure-cli 2.0.59, bash shell, Red Hat Enterprise Linux Server release 7.6 (Maipo) running Python 2.7.5 [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on a server in Azure SouthCentral, and on an identical one in Azure West.

Additional context
The script running the 'offending' command is an essential step in our Disaster Recovery procedure and runs nightly.

[bartoostlander]

As it turns out, running the following command before running any Azure CLI command solves the problem:

export LANG=en_US.UTF-8

[bartoostlander]

Running the following command before running any Azure CLI commands avoids the problem:

export LANG=en_US.UTF-8

[bartoostlander]

I spoke too soon. Issue still exists... :-(

[PatMyron]

related: https://github.com/Azure/azure-cli/issues/8312, https://github.com/Azure/azure-cli/issues/6408, https://github.com/Azure/azure-cli/issues/4418, https://github.com/Azure/azure-cli/issues/1277

Pats-MacBook-Pro  ~ $ az resource list > /dev/null 
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 868, in emit
    msg = self.format(record)
  File "/usr/local/lib/python2.7/site-packages/knack/log.py", line 78, in format
    msg = logging.StreamHandler.format(self, record)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 741, in format
    return fmt.format(record)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 469, in format
    s = self._fmt % record.__dict__
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 172: ordinal not in range(128)
Logged from file util.py, line 26

Pats-MacBook-Pro  ~ $ which python pip az
/usr/local/bin/python
/usr/local/bin/pip
/usr/local/bin/az

Pats-MacBook-Pro  ~ $ python --version
Python 2.7.16

Pats-MacBook-Pro  ~ $ pip --version
pip 19.0.3 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)

Pats-MacBook-Pro  ~ $ uname -a
Darwin Pats-MacBook-Pro.local 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64

Pats-MacBook-Pro  ~ $ az --version
azure-cli (2.0.41)

acr (2.0.28)
acs (2.1.3)
advisor (0.5.1)
ams (0.1.1)
appservice (0.1.36)
backup (1.1.2)
batch (3.2.4)
batchai (0.3.1)
billing (0.1.8)
cdn (0.0.14)
cloud (2.0.15)
cognitiveservices (0.1.13)
command-modules-nspkg (2.0.1)
configure (2.0.17)
consumption (0.3.1)
container (0.2.1)
core (2.0.41)
cosmosdb (0.1.21)
dla (0.1.0)
dls (0.0.22)
dms (0.0.1)
eventgrid (0.1.12)
eventhubs (0.1.3)
extension (0.1.0)
feedback (2.1.3)
find (0.2.11)
interactive (0.3.25)
iot (0.1.21)
keyvault (2.0.23)
lab (0.0.22)
monitor (0.1.8)
network (2.1.5)
nspkg (3.0.2)
profile (2.0.27)
rdbms (0.2.5)
redis (0.2.14)
reservations (0.2.1)
resource (2.0.32)
role (2.0.26)
servicebus (0.1.2)
servicefabric (0.0.12)
sql (2.0.27)
storage (2.0.35)
vm (2.0.35)

Python location '/usr/local/opt/python@2/bin/python2.7'
Extensions directory '/Users/patmy/.azure/cliextensions'

Python (Darwin) 2.7.16 (default, Mar  4 2019, 09:01:38) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)]

Legal docs and information: aka.ms/AzureCliLegal

[PatMyron]

Worked around this on my Mac by forcing the use of azure-cli from Homebrew instead of Pip:

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-macos

brew update && brew install azure-cli && brew link --overwrite azure-cli

[marstr]

@bartoostlander, seems like the workaround is effectively "use Python3 for the Azure CLI". This is easy for @PatMyron, who is installing with brew because Python3 is the default in that environment.

However, since you're coming from Red Hat, I'll assume you install from the RPM that we publish. That makes it trickier, as my understanding is that Python3 won't be available through first party repositories until EL8.

Do you have access to a python3 package through third party repositories?

[bartoostlander]

@marstr , I installed Python 3.6 from the RedHat SCL. I can then run "scl enable rh-python36 bash" and run my script (that does "az storage blob exists"). Echoing "which python" in that script confirms that Python 3.6 is active. However, the "az storage" command on the next line still gives the same error, referring to Python 2.7!

Apparently, I need to do something to convince the Azure CLI to use the currently active Python version. What could that be?

[marstr]

Yep, that's expected! We try to isolate ourselves from system-python by running in a virtualenv. Which in general is a great thing, but in this particular case it makes hard to break-open your install to change things.

(While not what we usually recommend) In this case the easiest thing will be to install directly from PyPI using pip by doing the following:

# remove the old azure-cli install for cleanliness
yum erase -y azure-cli

# get python3 if you don't already have it (@bartoostlander can skip this)

## your SCL may vary if you're not using CentOS
yum install -y centos-release-scl
yum install -y rh-python36
scl enable rh-python36 bash

# Create a virtualenv where you would like to install the azure-cli
## Make sure that python invokes python3 before executing this step. If it doesn't, run the scl enable from above; this is because scl enable isn't durable between bash sessions.
cd ${yourDesiredLocation}
python -m virtualenv azure-cli

# Install the Azure-CLI from PyPI into this virtualenv
./azure-cli/bin/python -m pip install azure-cli

# Create a launcher for the azure-cli, and make it executable
cat > /usr/bin/az << EOM
#!/usr/bin/env bash
${yourDesiredLocation}/azure-cli/bin/python -m azure.cli "\$@"
EOM

chmod +x /usr/bin/az

To remove this installation, all you'll have to do is delete ${yourDesiredLocation}/azure-cli and /usr/bin/az.

[bartoostlander]

Aha! You just saved me from going doing a rabbit hole. I'll follow your instructions. Your responsiveness is highly appreciated!

[bartoostlander]

Preliminary observation: it works... I'll close this issue after further validation. Thank you!

[bartoostlander]

Success. Thank you, @marstr !

[sdkks]

Worked around this on my Mac by forcing the use of azure-cli from Homebrew instead of Pip:

docs.microsoft.com/en-us/cli/azure/install-azure-cli-macos

brew update && brew install azure-cli && brew link --overwrite azure-cli

brew install azure-cli exits non zero because symlink can't be created. Maybe:
brew update && brew install azure-cli || brew link --overwrite azure-cli ?