Tox: Unicode issue : UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 46133: ordinal not in range(128)

[VedaAnnayappa]

Getting below error when we run tox:

File "/usr/local/lib/python2.7/dist-packages/tox/reporter.py", line 74, in logline_if
message = str(msg) if key is None else "{}{}".format(key, msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 46133: ordinal not in range(128)

[gaborbernat]

Can you please post full stack trace and version information?

[VedaAnnayappa]

Hi,

Here is the data requested:

tox --version

3.13.2 imported from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc

Below is the data printed for key and value, value contain a large message.

(u'###### key :', None)

'''(u'###### msg :', u'Ignoring contextlib2: markers \'python_version == "2.7"\' don\'t match your environment\nIgnoring sphinx: markers \'python_version == "2.7"\' don\'t match your environment\nIgnoring sphinx: markers \'python_version == "2.7"\' don\'t match your environment\nIgnoring contextlib2: markers \'python_version < "3.0"\' don\'t match your environment\nCollecting Babel==2.3.4 (from -c /home/lower-constraints.txt (line 7))\n  Using cached https://files.pythonhosted.org/packages/b4/ec/acd307eac2e23f9cab1c8bdbe29b3b1d43215e31c32f8aa91b3a97925b5b/Babel-2.3.4-py2.py3-none-any.whl\nCollecting bandit==1.1.0 (from -c /home/lower-constraints.txt (line 8))\n  Using cached https://files.pythonhosted.org/packages/58/97/836e0adc60e379f15a6679586b90e615caf46664b5f8052af3dd2c670b43/bandit-1.1.0-py2.py3-none-any.whl\nCollecting coverage==4.0 (from -c /home/lower-constraints.txt (line 16))\nCollecting ddt==1.0.1 (from -c /home/lower-constraints.txt (line 19))\n  Using cached https://files.pythonhosted.org/packages/71/f1/ccb89dbb20a9ca5b16bc37e7472d21f30f3d848c2799c6ea56e253b63047/ddt-1.0.1-py2.py3-none-any.whl\nCollecting fixtures==3.0.0 (from -c /home/lower-constraints.txt (line 29))\n  Using cached https://files.pythonhosted.org/packages/a8/28/7eed6bf76792f418029a18d5b2ace87ce7562927cdd00f1cefe481cd148f/fixtures-3.0.0-py2.py3-none-any.whl\nCollecting gabbi==1.35.0 (from -c /home/lower-constraints.txt (line 33))\n  Using cached https://files.pythonhosted.org/packages/fe/19/f459838eb9805eae5dc9a77e5a147d02592892234fbc6ae3fc5bd5bf3b32/gabbi-1.35.0-py2.py3-none-any.whl\nCollecting GitPython==2.1.8 (from -c /home/lower-constraints.txt (line 35))\n  Using cached https://files.pythonhosted.org/packages/5b/38/0433c06feebbfbb51d644129dbe334031c33d55af0524326266f847ae907/GitPython-2.1.8-py2.py3-none-any.whl\nCollecting hacking==0.12.0 (from -c /home/lower-constraints.txt (line 37))\n  Using cached https://files.pythonhosted.org/packages/91/ba/ef758cd0de7660fbf741fe067f3d82a708b7996a82b0569013419bd360ec/hacking-0.12.0-py2.py3-none-any.whl\nCollecting jsonpath-rw-ext==1.1.3 (from -c /home/lower-constraints.txt (line 44))\n  Using cached https://files.pythonhosted.org/packages/9c/51/05fa4a86e50e0c7d551bcef3f2900bf6c5654819ca3d89c97838e9411411/jsonpath_rw_ext-1.1.3-py2.py3-none-any.whl\nCollecting mccabe==0.2.1 (from -c /home/lower-constraints.txt (line 54))\nCollecting mock==3.0.0 (from -c /home/lower-constraints.txt (line 56))\n  Using cached https://files.pythonhosted.org/packages/59/83/8fcdcf4babcc55290e50cbd54fcc9fcdd81c2e62eff27c1e6a2f23d79381/mock-3.0.0-py2.py3-none-any.whl\nCollecting mox3==0.20.0 (from -c /home/lower-constraints.txt (line 58))\n  Using cached https://files.pythonhosted.org/packages/ee/26/09b414c7bb664917a52f3aa39042641e29c0e3a5cbfd2f97ce2acf6d45a3/mox3-0.20.0-py2.py3-none-any.whl\nCollecting os-resource-classes==0.1.0 (from -c /home/lower-constraints.txt (line 69))\n  Using cached https://files.pythonhosted.org/packages/b8/87/fddacc61eb3a95517dcf56d5264b10719fbe4d22bc8f41108db0d72b5d4c/os_resource_classes-0.1.0-py2.py3-none-any.whl\nCollecting oslo.config==6.1.0 (from -c /home/lower-constraints.txt (line 78))\n  Using cached https://files.pythonhosted.org/packages/68/6f/55ecede7bc929cf7450de1d1fe365347c36c215d74f6350736744f92a30c/oslo.config-6.1.0-py2.py3-none-any.whl\nCollecting '''

error seen is

File "/usr/local/lib/python2.7/dist-packages/tox/reporter.py", line 74, in logline_if
    message = str(msg) if key is None else "{}{}".format(key, msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 46133: ordinal not in range(128)

Stack trace

Traceback (most recent call last):
  File "/usr/local/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/usr/local/lib/python2.7/dist-packages/tox/session/__init__.py", line 44, in cmdline
    main(args)
  File "/usr/local/lib/python2.7/dist-packages/tox/session/__init__.py", line 68, in main
    exit_code = session.runcommand()
  File "/usr/local/lib/python2.7/dist-packages/tox/session/__init__.py", line 192, in runcommand
    return self.subcommand_test()
  File "/usr/local/lib/python2.7/dist-packages/tox/session/__init__.py", line 220, in subcommand_test
    run_sequential(self.config, self.venv_dict)
  File "/usr/local/lib/python2.7/dist-packages/tox/session/commands/run/sequential.py", line 9, in run_sequential
    if venv.setupenv():
  File "/usr/local/lib/python2.7/dist-packages/tox/venv.py", line 594, in setupenv
    status = self.update(action=action)
  File "/usr/local/lib/python2.7/dist-packages/tox/venv.py", line 257, in update
    self.hook.tox_testenv_install_deps(action=action, venv=self)
  File "/usr/local/lib/python2.7/dist-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pluggy/manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 81, in get_result
    _reraise(*ex)  # noqa
  File "/usr/local/lib/python2.7/dist-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python2.7/dist-packages/tox/venv.py", line 740, in tox_testenv_install_deps
    venv._install(deps, action=action)
  File "/usr/local/lib/python2.7/dist-packages/tox/venv.py", line 457, in _install
    self.run_install_command(packages=packages, options=options, action=action)
  File "/usr/local/lib/python2.7/dist-packages/tox/venv.py", line 407, in run_install_command
    env=env,
  File "/usr/local/lib/python2.7/dist-packages/tox/venv.py", line 574, in _pcall
    report_fail=not is_test_command,
  File "/usr/local/lib/python2.7/dist-packages/tox/action.py", line 121, in popen
    reporter.quiet(output)
  File "/usr/local/lib/python2.7/dist-packages/tox/reporter.py", line 123, in quiet
    self.logline_if(Verbosity.QUIET, "quiet", msg)
File "/usr/local/lib/python2.7/dist-packages/tox/reporter.py", line 74, in logline_if
    message = str(msg) if key is None else "{}{}".format(key, msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 46133: ordinal not in range(128)

[lubomir-brindza]

If anyone is still hitting this issue, it's probably caused by an error messages emitted by pip during preparation of the testing environment (in my case, it was a failing python-mysql build).

In my particular case I was able to get to the actual error message by exporting LANG=en_US.UTF-8

ref: https://stackoverflow.com/a/43385201/574084

[VedaAnnayappa]

its not a right idea to set complete environment en_US.UTF-8, what in case if we are using a different locales in another applications running on the same OS

[lubomir-brindza]

Sure, it's just a temporary workaround - one still needs to resolve the errors thrown (by pip or otherwise).

But the gist is, as I understand it, that tox tries to convert the subprocess output to str(), which will be a problem on PY2 if the output contains Unicode characters, as str() still refers to an ASCII string.

Maybe a false assumption, going barely by the traceback here, haven't looked at the source in depth.

[ewhauser]

I'm running into the same issue; however setting LANG doesn't resolve the problem.

[ewhauser]

Figured it out. You might also have to set LC_ALL if it is not set:

export LC_ALL= en_US.UTF-8

[allanj]

Figured it out. You might also have to set LC_ALL if it is not set:

export LC_ALL= en_US.UTF-8

This one works perfectly for me