Azure-cli: az acr build throws UnicodeEncodeError: 'ascii' codec can't encode character

Created on 22 May 2018 · 16Comments · Source: Azure/azure-cli

Describe the bug
We're running az acr build ... from our jenkins slaves that also run tests.
If the docker build process returns unicode characters, az throws an UnicodeDecodeError (full traceback below).
When running the build process interactively on my local machine i see the line that is causing the error: npm WARN deprecated [email protected]: 🙌 Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update!
I suspect the emoji is causing the error because the library colorama used in az handles the output different for a non-tty environment.

To Reproduce
trigger a az acr build with a Dockerfile that causes unicode output in e.g. jenkins (non-tty environment).

Expected behavior
i'd expect the build to succeed and the unicode characters returned as well

Environment summary
rpm installation, bash, non interactive.
version:

azure-cli (2.0.32)

acr (2.0.24)
acs (2.0.32)
advisor (0.5.1)
ams (0.1.0)
appservice (0.1.32)
backup (1.1.1)
batch (3.2.2)
batchai (0.2.2)
billing (0.1.8)
cdn (0.0.14)
cloud (2.0.13)
cognitiveservices (0.1.13)
command-modules-nspkg (2.0.1)
configure (2.0.15)
consumption (0.3.1)
container (0.1.23)
core (2.0.32)
cosmosdb (0.1.21)
dla (0.0.19)
dls (0.0.21)
dms (0.0.1)
eventgrid (0.1.12)
eventhubs (0.1.3)
extension (0.0.13)
feedback (2.1.1)
find (0.2.9)
interactive (0.3.20)
iot (0.1.20)
keyvault (2.0.21)
lab (0.0.22)
monitor (0.1.6)
network (2.1.0)
nspkg (3.0.2)
profile (2.0.24)
rdbms (0.2.2)
redis (0.2.13)
reservations (0.1.2)
resource (2.0.28)
role (2.0.23)
servicebus (0.1.2)
servicefabric (0.0.12)
sql (2.0.25)
storage (2.0.32)
vm (2.0.31)

Python location '/usr/lib64/az/bin/python'
Extensions directory '/var/lib/jenkins/.azure/cliextensions'

Python (Linux) 2.7.5 (default, Apr 11 2018, 07:36:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context
this seems to be related: https://github.com/tartley/colorama/issues/36

full output:

az acr build -r myreg --no-push false -t myreg.azurecr.io/repo/myapp:master -f ./Dockerfile .
Sending build context (43.398 KiB) to ACR.
Queued a build with build ID: aah
Waiting for a build agent...
time="2018-05-22T21:11:59Z" level=info msg="Running command docker login -u 00000000-0000-0000-0000-000000000000 --password-stdin myreg.azurecr.io"
Login Succeeded
time="2018-05-22T21:12:01Z" level=info msg="Untarring to /root/acr-builder/src"
time="2018-05-22T21:12:01Z" level=info msg="Running command docker build --pull -f ./Dockerfile -t myreg.azurecr.io/repo/myapp:master ."
Sending build context to Docker daemon  220.2kB

Step 1/8 : FROM node:carbon
carbon: Pulling from library/node
3d77ce4481b1: Already exists
534514c83d69: Already exists
d562b1c3ac3f: Already exists
4b85e68dc01d: Already exists
f6a66c5de9db: Already exists
7a4e7d9a081d: Already exists
49f75af57b99: Pulling fs layer
454abe4fb374: Pulling fs layer
454abe4fb374: Verifying Checksum
454abe4fb374: Download complete
49f75af57b99: Verifying Checksum
49f75af57b99: Download complete
49f75af57b99: Pull complete
454abe4fb374: Pull complete
Digest: sha256:19efc0ee886057870334b4b4b4a505ea8a158d135f592cc367ab8840b684f6c0
Status: Downloaded newer image for node:carbon
 ---> 037acc6a00bc
Step 2/8 : WORKDIR /usr/src/app
Removing intermediate container ade8e4293757
 ---> 5fa54edc3ae3
Step 3/8 : COPY package*.json ./
 ---> 295c9eeaf7fc
Step 4/8 : RUN npm install
 ---> Running in 6846edad6a09
npm WARN deprecatedERROR: 'ascii' codec can't encode character u'\U0001f64c' in position 29: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 332, in execute
    six.reraise(*sys.exc_info())
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 306, in execute
    result = cmd(params)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 167, in __call__
    return super(AzCliCommand, self).__call__(*args, **kwargs)
  File "/usr/lib64/az/lib/python2.7/site-packages/knack/commands.py", line 109, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 420, in default_command_handler
    result = op(**command_args)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/build.py", line 330, in acr_build
    return acr_build_show_logs(client, build_id, registry_name, resource_group_name, True)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/build.py", line 71, in acr_build_show_logs
    raise_error_on_failure=raise_error_on_failure)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/build.py", line 132, in _stream_logs
    print(flush.decode('utf-8', errors='ignore'))
  File "/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py", line 40, in write
    self.__convertor.write(text)
  File "/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py", line 141, in write
    self.write_and_convert(text)
  File "/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py", line 166, in write_and_convert
    self.write_plain_text(text, cursor, start)
  File "/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py", line 174, in write_plain_text
    self.wrapped.write(text[start:end])
UnicodeEncodeError: 'ascii' codec can't encode character u'\U0001f64c' in position 29: ordinal not in range(128)

Container Registry Service Attention bug

Source

jodok

Most helpful comment

i created a fix upstream, but unfortunately this isn't merged yet: https://github.com/tartley/colorama/pull/167 - meanwhile we ended up patching the library in our own deployments - which works so far.

jodok on 20 Jul 2018

👍2

All 16 comments

I had this as well:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 38: ordinal not in range(128)

Has the fix been released yet?

DaveKlassen on 18 Jul 2018

jodok on 20 Jul 2018

👍2

Is there a work around we can do?
Any published released version?

We have an angular app that is failing to build because of this

timja on 5 Oct 2018

Use option --no-format

troydai on 5 Oct 2018

aah thanks I see, it was released in 2.0.46

timja on 5 Oct 2018

You're welcome.

troydai on 5 Oct 2018

Using az v2.0.47 with --no-format didn't solve this for me

DaveKlassen on 16 Oct 2018

@DaveKlassen if this isn't fixing the specific case then might be worth considering reopening the issue.
Do you have a Dockerfile that is hitting this for a repro?

sajayantony on 16 Oct 2018

Its a really simple Dockerfile. However there is a line where I run pipenv like:

RUN pipenv clean
RUN pipenv sync --dev

It seems to be failing when we run pipenv to prepare the container. docker build doesn't cause this error... but I would prefer to use az acr build to have a tighter integration with the registry.

Also it seems that the error isn't always consistent. I have seen the error in three different forms:

UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2026' in position 38: ordinal not in range(128)
UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2026' in position 73: ordinal not in range(128)
UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2026' in position 1037: ordinal not in range(128)

DaveKlassen on 16 Oct 2018

@DaveKlassen what is the base image used your Dockerfile?

northtyphoon on 17 Oct 2018

The base image contains the following:

FROM centos:latest

ENV LANG en_US.utf8

# Install tools
RUN yum -y install epel-release
RUN yum -y install python-pip
RUN yum -y install python-devel
RUN pip install --upgrade pip
RUN pip install --upgrade setuptools

RUN pip install pipenv==11.10.4

DaveKlassen on 17 Oct 2018

@DaveKlassen, so far I am unable to repro the issue in python 2.7.15 (alpine3.7). RUN pipenv sync --dev does output the ellipsis … (u+2026) properly. What is your python and os version?
Also may I suggest you try on Azure Cloud Shell which has python 3.5.2 and latest az tool installed?

northtyphoon on 18 Oct 2018

I have a specific Dev environment that cannot be changed, and cannot be disclosed (ie. v3.5 is not an option). When I have time I will reproduce the errata for you.

One hint here, did you place anything in the PipFile? I would try a bunch of random packages, since I am fairly sure this issue occurs during pipenv install/sync

DaveKlassen on 18 Oct 2018

@DaveKlassen I did try a bunch of packages in PipFile. The ellipsis character just worked fine on my machine. I noticed you reached Microsoft customer service and they should start a thread with you. If possible, can you send your Dev environment and PipFile to the customer service representative? It will help us narrow down the issue quickly. Thanks in advance.

northtyphoon on 20 Oct 2018

I have sent this data. Likely the only different part is the PipFile

DaveKlassen on 20 Oct 2018

I'm getting this error as well. When I run the az acr build command during a Jenkins build (on a server with azure-cli 2.0.47 and acr 2.1.6) the last line of output I get is Length: 22897802 (22M) [application/octet-stream] before failing with this error UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2018' in position 11: ordinal not in range(128).

However, I can run it on my local development environment using azure-cli 2.0.41 and acr 2.0.28. Here is the same section of output from my local build:

Step 5/11 : RUN cd /usr/src &&     wget https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz &&     tar xzf Python-3.7.2.tgz &&     rm -f Python-3.7.2.tgz
 ---> Running in 5f6aace0d8a4
--2019-02-20 20:28:51--  https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz
Resolving www.python.org (www.python.org)... 151.101.52.223, 2a04:4e42:d::223
Connecting to www.python.org (www.python.org)|151.101.52.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22897802 (22M) [application/octet-stream]
Saving to: ‘Python-3.7.2.tgz’

It looks like there is some problem encoding the ‘ character in the last line. Here is the full error from the Jenkins build:


        "ERROR: 'ascii' codec can't encode character u'\\u2018' in position 11: ordinal not in range(128)", 

        "Traceback (most recent call last):", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py\", line 197, in invoke", 

        "    cmd_result = self.invocation.execute(args)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 369, in execute", 

        "    six.reraise(*sys.exc_info())", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 343, in execute", 

        "    result = cmd(params)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 182, in __call__", 

        "    return self.handler(*args, **kwargs)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py\", line 436, in default_command_handler", 

        "    result = op(**command_args)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/build.py\", line 126, in acr_build", 

        "    return stream_logs(client, run_id, registry_name, resource_group_name, no_format, True)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/_stream_utils.py\", line 63, in stream_logs", 

        "    raise_error_on_failure)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/_stream_utils.py\", line 127, in _stream_logs", 

        "    print(flush.decode('utf-8', errors='ignore'))", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py\", line 40, in write", 

        "    self.__convertor.write(text)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py\", line 141, in write", 

        "    self.write_and_convert(text)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py\", line 166, in write_and_convert", 

        "    self.write_plain_text(text, cursor, start)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py\", line 174, in write_plain_text", 

        "    self.wrapped.write(text[start:end])", 

        "UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2018' in position 11: ordinal not in range(128)