Pipenv: UnicodeDecodeError happens on mbcs windows environment (e.g. Japanese) when a package is installed from git+https source

Created on 1 Sep 2019 · 4Comments · Source: pypa/pipenv

Issue description

UnicodeDecodeError happens on mbcs windows environment (e.g. Japanese) when a package is installed from git+https source

Expected result

pipenv install -e git+https://github.com/issuer/some-package

some-package is installed without error.

Actual result

UnicodeDecodeError: 'cp932' codec can't decode byte 0x82 in position 68: illegal multibyte sequence

Environments

windows 10 version 1809

python 3.7.3

pip 19.0.3

pipenv 2018.11.26

Steps to replicate

Use windows which uses mbcs locale as a system locale (e.g. Japanese)
Otherwise changing system locale of windows to mbcs codepage will help.
Create a new directory for project

> mkdir pipenvtest

> cd pipenvtest

Make sure to clean pipenv cache

> rmdir /s /q %LOCALAPPDATA%\pipenv\pipenv\Cache

Initialize venv

> pipenv --python=3.7

Install a package from github source.

> pipenv --verbose install -e git+https://github.com/psf/requests#egg=requests

this leads to following error.

Installing -e git+https://github.com/psf/requests#egg=requests…
[    ] Installing...Installing 'requests'
$ ['D:\\src\\python\\pipenvtest2\\.venv\\Scripts\\pip.exe', 'install', '--src', 'D:\\src\\python\\pipenvtest2\\.venv\\src', '--verbose', '--upgrade', '-e', 'git+https://github.com/psf/requests#egg=requests', '-i', 'https://pypi.org/simple']

Adding requests to Pipfile's [packages]…
Installation Succeeded
Pipfile.lock (e8493f) out of date, updating to (e0e925)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
[    ] Pinning VCS Packages...INFO:pipenv.patched.notpip._internal.vcs.git:Cloning https://github.com/psf/requests to c:\users\user1\appdata\local\temp\requirementslibwxdusymj\requests
Traceback (most recent call last):
  File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python37\Scripts\pipenv.exe\__main__.py", line 9, in <module>
  File "c:\python37\lib\site-packages\pipenv\vendor\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\python37\lib\site-packages\pipenv\vendor\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\python37\lib\site-packages\pipenv\vendor\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\python37\lib\site-packages\pipenv\vendor\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "c:\python37\lib\site-packages\pipenv\vendor\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\python37\lib\site-packages\pipenv\cli\command.py", line 254, in install
    editable_packages=state.installstate.editables,
  File "c:\python37\lib\site-packages\pipenv\core.py", line 1992, in do_install
    skip_lock=skip_lock,
  File "c:\python37\lib\site-packages\pipenv\core.py", line 1221, in do_init
    pypi_mirror=pypi_mirror,
  File "c:\python37\lib\site-packages\pipenv\core.py", line 1068, in do_lock
    lockfile=lockfile
  File "c:\python37\lib\site-packages\pipenv\utils.py", line 650, in venv_resolve_deps
    results = c.out
  File "c:\python37\lib\site-packages\pipenv\vendor\delegator.py", line 133, in out
    self.__out = self._pexpect_out
  File "c:\python37\lib\site-packages\pipenv\vendor\delegator.py", line 121, in _pexpect_out
    result += self.subprocess.read()
  File "c:\python37\lib\site-packages\pipenv\vendor\pexpect\spawnbase.py", line 441, in read
    self.expect(self.delimiter)
  File "c:\python37\lib\site-packages\pipenv\vendor\pexpect\spawnbase.py", line 341, in expect
    timeout, searchwindowsize, async_)
  File "c:\python37\lib\site-packages\pipenv\vendor\pexpect\spawnbase.py", line 369, in expect_list
    return exp.expect_loop(timeout)
  File "c:\python37\lib\site-packages\pipenv\vendor\pexpect\expect.py", line 111, in expect_loop
    incoming = spawn.read_nonblocking(spawn.maxread, timeout)
  File "c:\python37\lib\site-packages\pipenv\vendor\pexpect\popen_spawn.py", line 93, in read_nonblocking
    buf += self._decoder.decode(incoming, final=False)
UnicodeDecodeError: 'cp932' codec can't decode byte 0x82 in position 164: illegal multibyte sequence

result of pipenv --support

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: 'c:\\python37\\lib\\site-packages\\pipenv'

Python location: 'c:\\python37\\python.exe'

Python installations found:

3.7.3: C:\Python37\python.exe
3.6.8: C:\Python36\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.3',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.17763',
 'python_full_version': '3.7.3',
 'python_version': '3.7',
 'sys_platform': 'win32'}

System environment variables:

ALIASES
ALLUSERSPROFILE
ANSICON
ANSICON_DEF
APPDATA
ARCHITECTURE
ARCHITECTURE_BITS
ASL.LOG
CCALL
CEXEC
CMDER_ALIASES
CMDER_CLINK
CMDER_CONFIGURED
CMDER_INIT_END
CMDER_INIT_START
CMDER_ROOT
CMDER_SHELL
CMDER_USER_FLAGS
COMMONPROGRAMFILES
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
COMPUTERNAME
COMSPEC
CONEMUANSI
CONEMUANSILOG
CONEMUARGS
CONEMUARGS2
CONEMUBACKHWND
CONEMUBASEDIR
CONEMUBASEDIRSHORT
CONEMUBUILD
CONEMUCFGDIR
CONEMUCONFIG
CONEMUDIR
CONEMUDRAWHWND
CONEMUDRIVE
CONEMUHOOKS
CONEMUHWND
CONEMUPALETTE
CONEMUPID
CONEMUSERVERPID
CONEMUTASK
CONEMUWORKDIR
CONEMUWORKDRIVE
DEBUG_OUTPUT
DRIVERDATA
FAST_INIT
FEFLAGNAME
GIT_INSTALL_ROOT
HOME
HOMEDRIVE
HOMEPATH
LANG
LIB_BASE
LIB_CONSOLE
LIB_GIT
LIB_PATH
LIB_PROFILE
LOCALAPPDATA
LOGONSERVER
MAX_DEPTH
NIX_TOOLS
NUMBER_OF_PROCESSORS
NVM_HOME
NVM_SYMLINK
ONEDRIVE
OS
PATH
PATHEXT
PIPENV_VENV_IN_PROJECT
PLINK_PROTOCOL
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMDATA
PROGRAMFILES
PROGRAMFILES(X86)
PROGRAMW6432
PROMPT
PSMODULEPATH
PUBLIC
RUBYOPT
SESSIONNAME
SVN_SSH
SYSTEMDRIVE
SYSTEMROOT
TEMP
TERM
TIME_INIT
TMP
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
USER_ALIASES
VBOX_MSI_INSTALL_PATH
VERBOSE_OUTPUT
WINDIR
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

b'Pipenv\xe2\x80\x93specific environment variables:'

PIPENV_VENV_IN_PROJECT: true

b'Debug\xe2\x80\x93specific environment variables:'

PATH: D:\app\cmder\bin;D:\app\cmder\vendor\bin;D:\app\cmder\vendor\conemu-maximus5\ConEmu\Scripts;D:\app\cmder\vendor\conemu-maximus5;D:\app\cmder\vendor\conemu-maximus5\ConEmu;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\iCLS\;C:\Program Files\Intel\Intel(R) Management Engine Components\iCLS\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\Users\user1\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Ruby26-x64\bin;C:\Python37\Scripts\;C:\Python37\;C:\Users\user1\AppData\Local\Microsoft\WindowsApps;C:\Users\user1\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\user1\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;D:\app\cmder
LANG: ja_JP.UTF-8

Contents of Pipfile ('D:\src\python\pipenvtest2\Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
requests = {editable = true,git = "https://github.com/psf/requests"}

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('D:\src\python\pipenvtest2\Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "c198550deb9d9c74c28c726f6627267cfa4b9ee9ee39f3c9fb2349b72ce0e925"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:046832c04d4e752f37383b628bc601a7ea7211496b4638f6514d0e5b9acc4939",
                "sha256:945e3ba63a0b9f577b1395204e13c3a231f9bc0223888be653286534e5873695"
            ],
            "version": "==2019.6.16"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "idna": {
            "hashes": [
                "sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407",
                "sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c"
            ],
            "version": "==2.8"
        },
        "requests": {
            "hashes": [
                "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
                "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
            ],
            "index": "pypi",
            "version": "==2.22.0"
        },
        "urllib3": {
            "hashes": [
                "sha256:b246607a25ac80bedac05c6f282e3cdaf3afb65420fd024ac94435cabe6e18d1",
                "sha256:dbe59173209418ae49d485b87d1681aefa36252ee85884c31346debd19463232"
            ],
            "version": "==1.25.3"
        }
    },
    "develop": {}
}

Cause of this problem

This error happens because resolver.py emits utf-8 to stdout although the process watcher in delegators.py is still expecting mbcs encoding like cp932.

In this case popen_spawn.py is trying to decode these string
b'\xe3\x82\xa2\xe3\x82\xaf\xe3\x82\xbb\xe3\x82\xb9\xe3\x81\x8c\xe6\x8b\x92\xe5\x90\xa6\xe3\x81\x95\xe3\x82\x8c\xe3\x81\xbe\xe3\x81\x97\xe3\x81\x9f\xe3\x80\x82'
which is encoded in utf-8.

But popen_spawn.py uses cp932 (Japanese mbcs character set) even for this case.

In short, output using utf-8 should be read as utf-8 not as mbcs.

By the way setting these environment variable takes no effect. Same error happens.

set PYTHONIOENCODING=utf-8
set PYTHONUTF8=1

How to fix

I can send a PR for this.

CLI Encoding Error Handling VCS Type

Source

TeeNak

Most helpful comment

Thank you for tracing this issue. It also happens on my Chinese Windows 10 when I install from git (In fact, the installation is done successfully but the lock process emits this error)

EqualMa on 12 Jun 2020

👍2

All 4 comments

Thanks for reporting and fixing this, it's actually really difficult to track this down for me at least.

Before we make additional changes, can you verify that this still happens? We've done some additional work here so I'd be interested

techalchemy on 25 Apr 2020

I just checked latest code as of 05/03/2020 (02288d8f3) and got following results.

The originally reported UnicodeDecodeError does not happen because underlying failure of TemporaryDirectory cleanup was fixed by the latest change.
(The failure was related to this problem https://bugs.python.org/issue26660)
There still remains mismatch of text encoding between resolver.py and delegators.py which may lead to nightmare.

I personally encourage us to fix this mismatch of encoding in some way or another to prevent future problem, but it is still good for me to close this issue without changing anything because eminent problem was solved.

TeeNak on 5 May 2020

👍1

@TeeNak this is perfect. Thank you for the clarity & detail. I would really like to address this (without breaking anything else) but am likely to hold off until a subsequent release until I have time to reason about an adequate test to protect against regressing.

Thanks again for providing enough information to fully appreciate the underlying cause and potential effects.

techalchemy on 7 May 2020

👍1

Thank you for tracing this issue. It also happens on my Chinese Windows 10 when I install from git (In fact, the installation is done successfully but the lock process emits this error)

EqualMa on 12 Jun 2020

👍2

Was this page helpful?

0 / 5 - 0 ratings