Tox: Code can't detect encoding on OSX

[butla]

I've seen a strange problem today on my coworker's Mac. He doesn't have the time to submit a bug and I don't have a Mac, so the description might not be the best, but if someone else could reproduce it, it'd be cool.

Environment:

• 10.10.5 OSX Yosemite
• Python 2.7.13
• English US UTF-8 locale

Reproduction:
Have a Tox configuration running tests on python 2.7. The sequence we have is

coverage erase
coverage run -m py.test -vv tests/
coverage combine
coverage report -m

Version of coverage is 4.4.1.

Run the environment with Tox, e.g. tox -e py27.

coverage report crashes with LookupError: unknown encoding: on line 34 of summary.py

Explanation:
The function that's called there tries to get the encoding like this:

encoding = (
    getattr(outfile, "encoding", None) or
    getattr(sys.__stdout__, "encoding", None) or
    locale.getpreferredencoding()
)

Normally, when running the code with py27's virtualenv (the one created by Tox) activated, all of these methods were returning UTF-8, but when Tox is running the commands, they all return None.

I solved the issue by setting PYTHONIOENCODING in Tox:

[testenv:py27]
setenv =
    PYTHONIOENCODING = UTF-8

P.S. We're only using 2.7 because guys who started the project didn't know much about Python, and I didn't have enogh time to migrate it.

[obestwalter]

Hi @butla thanks for reporting this. This might be one of those environment variables that need passing through to avoid problems. tox changed its default in 2.0 to not pass through the environment wholesale to the testrun. Since then a lot of hidden dependencies on env vars have popped up and the list is growing:

https://github.com/tox-dev/tox/blob/e8891a5bc8349b76e1692e3236e36c1fc8beecbe/tox/config.py#L471-L496

Maybe this needs to be added for Mac on py27 - if anyone else with a Mac could verify that ...

[butla]

Yeah, it'd be good if someone who feels that they have a properly configured Python on Mac checked that. There's also the possibility that the Python we were working with wasn't installed properly.

[asottile]

Perhaps the problem here is your python2.7 processes are writing to a pipe / file instead of directly to the terminal? When stdout is not a tty in python2.7, it will use the ASCII encoding (ignoring LANG, etc.).

Are you perhaps using tox --resultjson ..., envlogdir (in tox.ini), or tox ... > ...?

[butla]

@asottile Nope, sorry, nothing like that. Tox is run without any options. The project isn't open source, but I can share the tox.ini with you:

[tox]
envlist = pylint,pylint-tests,py27

[testenv]
deps =
    -rrequirements.txt
    -rrequirements-test.txt
    pipdeptree

passenv =
    FIVE9_USER
    FIVE9_PASSWORD

[testenv:pylint]
basepython = python2.7
envdir = {toxworkdir}/py27

commands =
    pylint the_name_of_the_project --rcfile=.pylintrc

[testenv:pylint-tests]
basepython = python2.7
envdir = {toxworkdir}/py27

commands =
    pylint tests --rcfile=.pylintrc-tests

[testenv:py27]
setenv =
    COVERAGE_PROCESS_START = {toxinidir}/.coveragerc
    # this is needed so that decoding strings in coverage on OSX doesn't crash
    PYTHONIOENCODING = UTF-8

commands =
    # so that we properly detect lines that lost coverage
    coverage erase
    coverage run -m py.test -vv tests/
    coverage combine
    coverage report -m

As you see, there's nothing too fancy in there.

[asottile]

@butla can you produce the output of the following command: env | grep -Ei '(lang|lc)'? And just to confirm, this is being run from a terminal?

This looks vaguely similar to a bug I fixed in visual studio code (and subsequently got fixed in cpython 2.7 itself):

• https://github.com/pre-commit/pre-commit/issues/519
• https://github.com/Microsoft/vscode/pull/26412
• https://bugs.python.org/issue30338
• https://github.com/python/cpython/commit/94a3694c3dda97e3bcb51264bf47d948c5424d84

[butla]

@asottile Yep, it was a terminal. This is the output:

$ env | grep -Ei '(lang|lc)'
LC_ALL=en_US.UTF-8
LANG=en_US

I'll take a look at the bugs later.

[asottile]

Ahah! Yep this is exactly the other issue. Your LANG variable is malformed -- it should look the same as your LC_ALL variable.

[butla]

@asottile Allright. The message got passed to the aforementioned Mac owner :) I probably need to dig deeper into the links you've posted, but do you know why LANG can be malformed? Something had to set it.

[asottile]

Probably a typo in bashrc? Hard to say, my case above was some errant application code.

[cryvate]

I am unable to reproduce on my MacBook, no matter what I set those environment variables to.

[asottile]

@cryvate here's a simple reproducer:

$ LANG=en_US /usr/bin/python -c 'import io; io.open("/dev/null")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
LookupError: unknown encoding: 

Note that this is fixed in the version that brew ships (due to the cpython bug I posted above being fixed):

$ LANG=en_US /usr/local/bin/python2 -c 'import io; io.open("/dev/null")'
$

$ /usr/bin/python --version
Python 2.7.10
$ /usr/local/bin/python2 --version
Python 2.7.14