Python: Set the locale to C.UTF-8 for Python 3

[rspeer]

Python 3 uses Unicode everywhere, which makes it inconvenient to use in an environment that claims not to support Unicode, such as these Docker images. The system I/O encoding ends up being "ascii", which means (for example) that non-ASCII strings simply can't be used with the print() function.

Usually, on Debian, the system's default locale would have set the encoding to UTF-8, but locales aren't installed.

The issue is easily fixed by adding one line to a Dockerfile, but I think it should be there in the Python 3 image itself so it doesn't catch people by surprise.

The line to add is:

ENV LANG C.UTF-8

[yosifkit]

See: https://github.com/docker-library/python/pull/5#issuecomment-50556812

[rspeer]

That doesn't convince me that it's reasonable to run Python 3 in an ASCII locale.

Sometimes scripts print things. The print function uses _actual_ Unicode in py3. It _is_ one of the layers of the Unicode sandwich. But it relies on the locale to be reasonable.

I see no downside to setting a locale as simple as C.UTF-8.

[rspeer]

Here's a minimal example to show you that the Unicode problem is not in my code.

FROM python:3.4
RUN python -c "print(chr(0x5555))"

Currently, this gives:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\u5555' in position 0: ordinal not in range(128)

But on a correctly configured system, it gives:

Here is a thread from the Python mailing list about this issue, where many of the Python developers describe this as a bug in the OS distribution: http://bugs.python.org/issue19846

This state of things is kind of unfortunate, because other systems run Python 3 in an insufficient locale as well, and it makes Python 3 look broken. But it's probably not going to change anytime soon, as the developers are unwilling to override what the system locale says when the system locale is wrong. That's why I'm reporting it as a bug here. Docker is the "OS distribution" here, and it's the only place the issue can be fixed.

Nick Coghlan: "At the moment, setting "LANG=C" on a Linux system _fundamentally breaks Python 3_, and that's not OK."

STINNER Victor: "The solution is to fix the locale, not to fix Python. For example, don't set LANG to C."