Pip: [Windows] Could not install packages due to an EnvironmentError: [Errno 42] Illegal byte sequence

[segevfiner]

Environment

• pip version: 18.0
• Python version: 2.7.15 64 bit
• OS: Windows 10.0.17134.165 x64
• The console locale is set to cp872.
• The console font is consolas.

Description
pip throws an exception when trying to display a progress bar for the download of any package on Windows.

Expected behavior
pip should work and install the package correctly.

How to Reproduce

Note that this is environment dependent.

1. pip install -I --no-cache-dir --verbose wheel

Using --progress-bar ascii doesn't help. --progress-bar off works, but is sub-optimal as you get no progress.

Analysis
The following seems to succeed, while later attempts to actually write the characters to the console fail. 😖
https://github.com/pypa/pip/blob/3e81d8cde142687deda8fb5a07ca3fb38906b007/src/pip/_internal/utils/ui.py#L52-L60

It will work on Python 3 since it has unicode support for the console. There is win-unicode-console which could be used to get similar support for Python 2. But the check should probably be fixed anyhow.

Output

>pip install -I --no-cache-dir --verbose wheel
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Config variable 'Py_UNICODE_SIZE' is unset, Python ABI tag may be incorrect
Created temporary directory: c:\users\user\appdata\local\temp\pip-ephem-wheel-cache-5axu12
Created temporary directory: c:\users\user\appdata\local\temp\pip-req-tracker-r7iyls
Created requirements tracker 'c:\\users\\user\\appdata\\local\\temp\\pip-req-tracker-r7iyls'
Created temporary directory: c:\users\user\appdata\local\temp\pip-install-5priwk
Collecting wheel
  1 location(s) to search for versions of wheel:
  * https://pypi.org/simple/wheel/
  Getting page https://pypi.org/simple/wheel/
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/wheel/ HTTP/1.1" 200 7077
  Analyzing links from page https://pypi.org/simple/wheel/
    Found link https://files.pythonhosted.org/packages/51/15/d31364732deb6e7d1490f8a08f011997b52b4b106d592d36e917404be613/wheel-0.1.tar.gz#sha256=2cc0dd3d9465ae323ad6e85fc9dc54ab3664da2c60ce42189b9ae4c44de7d8bd (from https://pypi.org/simple/wheel/), version: 0.1
    Found link https://files.pythonhosted.org/packages/e2/1e/aedf6a9c6c0a588f96e9fe6e953de0c9f9084baad05f3ed0bd4a9b4a469f/wheel-0.2.tar.gz#sha256=82026a421ca379affefa9a0cb85807047e7184574a92f406670b2dcc3384da36 (from https://pypi.org/simple/wheel/), version: 0.2
    <snip>
    Found link https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl#sha256=80044e51ec5bbf6c894ba0bc48d26a8c20a9ba629f4ca19ea26ecfcf87685f5f (from https://pypi.org/simple/wheel/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*), version: 0.31.1
    Found link https://files.pythonhosted.org/packages/2a/fb/aefe5d5dbc3f4fe1e815bcdb05cbaab19744d201bbc9b59cfa06ec7fc789/wheel-0.31.1.tar.gz#sha256=0a2e54558a0628f2145d2fc822137e322412115173e8a2ddbe1c9024338ae83c (from https://pypi.org/simple/wheel/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*), version: 0.31.1
  Using version 0.31.1 (newest of versions: 0.1, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.5, 0.6, 0.7, 0.8, 0.9, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.19.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.0, 0.29.0, 0.30.0, 0.31.0, 0.31.1)
  Created temporary directory: c:\users\user\appdata\local\temp\pip-unpack-euk6au
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl HTTP/1.1" 200 41567
  Downloading https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl (41kB)
  Downloading from URL https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl#sha256=80044e51ec5bbf6c894ba0bc48d26a8c20a9ba629f4ca19ea26ecfcf87685f5f (from https://pypi.org/simple/wheel/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*)
    24% |Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\_internal\commands\install.py", line 299, in run
    resolver.resolve(requirement_set)
  File "c:\python27\lib\site-packages\pip\_internal\resolve.py", line 102, in resolve
    self._resolve_one(requirement_set, req)
  File "c:\python27\lib\site-packages\pip\_internal\resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\python27\lib\site-packages\pip\_internal\resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "c:\python27\lib\site-packages\pip\_internal\operations\prepare.py", line 283, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 836, in unpack_url
    progress_bar=progress_bar
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 673, in unpack_http_url
    progress_bar)
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 897, in _download_http_url
    _download_url(resp, link, content_file, hashes, progress_bar)
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 617, in _download_url
    hashes.check_against_chunks(downloaded_chunks)
  File "c:\python27\lib\site-packages\pip\_internal\utils\hashes.py", line 48, in check_against_chunks
    for chunk in chunks:
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 585, in written_chunks
    for chunk in chunks:
  File "c:\python27\lib\site-packages\pip\_internal\utils\ui.py", line 161, in iter
    self.next(n)
  File "c:\python27\lib\site-packages\pip\_vendor\progress\__init__.py", line 73, in next
    self.update()
  File "c:\python27\lib\site-packages\pip\_vendor\progress\bar.py", line 86, in update
    self.writeln(line)
  File "c:\python27\lib\site-packages\pip\_vendor\progress\helpers.py", line 68, in writeln
    print(line, end='', file=self.file)
  File "c:\python27\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 141, in write
    self.write_and_convert(text)
  File "c:\python27\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 169, in write_and_convert
    self.write_plain_text(text, cursor, len(text))
  File "c:\python27\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 174, in write_plain_text
    self.wrapped.write(text[start:end])
IOError: [Errno 42] Illegal byte sequence
Cleaning up...
Removed build tracker 'c:\\users\\user\\appdata\\local\\temp\\pip-req-tracker-r7iyls'
There was an error checking the latest version of pip
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\_internal\utils\outdated.py", line 90, in pip_version_check
    state = SelfCheckState(cache_dir=options.cache_dir)
  File "c:\python27\lib\site-packages\pip\_internal\utils\outdated.py", line 25, in __init__
    self.statefile_path = os.path.join(cache_dir, "selfcheck.json")
  File "c:\python27\lib\ntpath.py", line 65, in join
    result_drive, result_path = splitdrive(path)
  File "c:\python27\lib\ntpath.py", line 115, in splitdrive
    if len(p) > 1:
TypeError: object of type 'bool' has no len()

P.S. The second exception is an entirely different bug... This happens when you use --no-cache-dir. Not sure if reported, if not, then a separate issue needs to be opened for that.

[segevfiner]

https://github.com/pypa/pip/blob/e818039dde54eae19391c829e4ffd227a988d76c/src/pip/_internal/utils/ui.py#L272
So "ascii" is the Unicode IncrementalBar... 😔 That's another bug, no?

Anyhow, this doesn't happen when you just print the Unicode character, in fact it works in the Python interpreter. It seems to be triggered by calling locale.setlocale(locale.LC_ALL, ''), here is an mwe:

#!/usr/bin/env python2
from __future__ import print_function
import locale

locale.setlocale(locale.LC_ALL, '')
print(u' |\u2588')

WAT

[segevfiner]

It seems I have stumbled upon "FUN"... 😢

Apparently, msvcrt does charset conversion when writing to its file descriptors based on the set locale! and it's even special cased to handle the OEM console code page (You can see this in crt/src/write.c:_write_nolock if you have MSVC 2008).

When the "C" locale is set, no conversion is done. Python encodes to the OEM code page, and it passes through to the console unscathed. But once you do setlocale than the CRT expects you to use the ANSI code page, but Python will be encoding to the OEM code page which will result in this error from fwrite.

file.encoding in Python 2 is also not settable directly from Python (C API only), it's only used for stdio and set internally on startup: Python/pythonrun.c:349-378.

Later I found this describing this: Why printf can display non-ASCII characters when “C” locale is used?.

[segevfiner]

https://bugs.python.org/issue34283

[segevfiner]

Some choice possible workarounds (There may be other/better ways), if any of them looks acceptable I can try to make a PR:

import ctypes

# Not accessible you say? 😈 
if sys.platform == "win32" and six.PY2:
    ctypes.pythonapi.PyFile_SetEncoding(ctypes.py_object(sys.stdin), locale.getpreferredencoding(False))
    ctypes.pythonapi.PyFile_SetEncoding(ctypes.py_object(sys.stdout), locale.getpreferredencoding(False))
    ctypes.pythonapi.PyFile_SetEncoding(ctypes.py_object(sys.stderr), locale.getpreferredencoding(False))

# In _select_progress_class. This makes the check for the characters use the correct
# encoding. Actually writing anything will still use the wrong encoding of course
# We can also just skip the check and use ascii always if this condition holds
if sys.platform == "win32" and six.PY2:
    encoding = locale.getpreferredencoding(False)

# This should actually make the Unicode chars work, an external library that would need to be vendored and initialized before colorama.
if sys.version_info < (3, 6):
    import win_unicode_console
    win_unicode_console.enable()

[pfmoore]

So you're saying this is a core Python issue that the Python core devs aren't willing to fix? (FWIW, as a Python core dev myself, I'm in agreement with Steve on the matter). If so, I don't see that there's any point in pip trying to work around it.

Does #5671 fix the issue (at least in cases where the user hasn't overridden the default style)? If so, that seems to be a sufficient workaround.

I'm -1 on the various workarounds you suggest. Of them all, the only one I'd remotely consider is the use of win_unicode_console, and I'm not particularly keen on that (I'd be concerned about compatibility issues, and I doubt our test suite or CI is complete enough to give much reassurance that something hasn't broken).

[segevfiner]

Essentially yes.

In my opinion, It's a core issue in the way Python 2.7 (and below, most likely) handle the console code page if you call setlocale. I kinda opened the issue for documentation purposes (So that people can find it and get linked to workarounds), I doubted anyone would want to make any changes to encodings in Python 2.7 at this point. Let alone a potentially non-backwards compatible fix. Python 3 and all... :grin:

5671 doesn't fix this. It only fixes the option for the user to override pip's automatic choice and use an ASCII only progress bar. It will still be a required to use it manually, and placing it in a config file will have it take effect for Python 3 too, which the user likely doesn't want.

Without implementing a workaround, pip will be broken by default for many new users on Python 2.7 on Windows (Ones who use code pages which trigger this, I'm not sure it happens with all of them), which I don't think is particularly nice. Things should really work out of the box. Users shouldn't need to apply workarounds themselves to get things to work with supported Python versions. Python 2.7 is still supported. For now... :wink:

[undera]

One workaround that worked for me: use pip install <package> -q. This made pip to not display progressbar which were triggering issue. I had no output displayed, but at least it has installed the package.

[segevfiner]

--progress-bar off also works.

[eranzim]

I also get this error on an updated Windows 10 with pip 18.0 and 18.1, didn't have it in 9.0.1 or 10.0.1.. I use --progress-bar=off, but as it was stated before - this is sub-optimal, there should be a fix rather than a workaround..

[noamzilo]

pip install -q

works around it for me.
but the problem doesn't always appear.
I just stumbled upon it with python 2.7.14 on windows 10 in pycharm terminal

[pradyunsg]

Thanks to @segevfiner for fixing --progress-bar ascii to actually being ASCII (in #5671). With the next pip release (i.e. later this month), using --progress-bar ascii will be a better alternative to using -q/--quiet, --progress-bar off etc.

Other than that, I'm not sure it'd be worth anyone's time to fix the unicode bug/problem that's Python 2 only. A better option might just be to detect "invalid byte sequence" and tell the user in the error message, that they might want to use --progress-bar ascii.

[pradyunsg]

pip 19.2 has released with the fix for the progress-bar issue -- ascii is now really ascii. :)

We've also documented how Python 2 support will be maintained going forward at: https://pip.pypa.io/en/latest/development/release-process/#python-2-support