Pipenv: UnicodeDecodeError when running pipenv install on Windows

Created on 14 Dec 2018 · 25Comments · Source: pypa/pipenv

Issue description

I'm installing a pipenv virtual env from a virtualenv project that has a requirement.txt, but it doesn't matter, I can't run any pipenv commands, even pipenv --support.

Expected result

Installation should be successful.

Actual result

pipenv install or pipenv --support

Traceback (most recent call last):
File "c:\users\panzi\appdata\local\programs\python\python36\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\users\panzi\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\panzi\AppData\Local\Programs\Python\Python36\Scripts\pipenv.exe__main__.py", line 5, in
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv__init__.py", line 47, in
from .cli import cli
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\cli__init__.py", line 3, in
from .command import cli
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\cli\command.py", line 7, in
import crayons
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\patched\crayons.py", line 49, in
is_powershell = "powershell" in shellingham.detect_shell()[0]
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham__init__.py", line 22, in detect_shell
shell = get_shell(pid, max_depth=max_depth)
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellinghamnt.py", line 100, in get_shell
processes = dict(_iter_process())
File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellinghamnt.py", line 78, in _iter_process
info = {'executable': str(pe.szExeFile.decode('utf-8'))}
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2: invalid continuation byte

Steps to replicate

pipenv install

After some debugging, I found it's a decoding bug in Python36\Lib\site-packages\pipenv\vendor\shellingham\nt.py

I added a try catch to capture the exception and try to decode it in my locale encoding, then it succeeded. Is there a more elegant way to fix it?

def _iter_process():
   ...
        try:
            info = {'executable': str(pe.szExeFile.decode('utf-8'))}
        except UnicodeDecodeError:
            info = {'executable': str(pe.szExeFile.decode('GBK'))}

Vendored Dependencies

Source

pansila

👍1

Most helpful comment

When can we include this fix? There are no releases for almost a year since the last pipenv release. https://github.com/sarugaku/shellingham/commit/381372076a075dd3e5eea9d1cfe6644e48b60aa8

pansila on 29 Oct 2019

👍3

All 25 comments

Thanks for your effort! It may relate with https://github.com/sarugaku/shellingham. @uranusjr do you mind to have a look for this?

jxltom on 15 Dec 2018

Similar to #2820, but on Windows. Thanks for the debugging effort! I think I know what is going on here. Will fix in the next release.

uranusjr on 15 Dec 2018

👍2 ❤1

By the way, @pansila can you provide the value of sys.getfilesystemencoding() of the Python you install Pipenv in?

uranusjr on 15 Dec 2018

@uranusjr It's 'utf-8'. And furthermore, sys.getdefaultencoding() also gives 'utf-8'.

The problem is that there are some processes with Chinese names, so when I tried to decode them with 'GBK', pipenv succeeded to run.

I'm using a Windows 10 with English as the default language, but I installed some Chinese softwares. It might be difficult to guess all encodings that processes use in this case.

pansila on 15 Dec 2018

Thanks! I believe my solution is correct, then.

uranusjr on 15 Dec 2018

It shouldn’t actually be necessary — but going to and from mbcs will be lossy, since utf-16 uses surrogates and if you fail to encode a surrogate using the python mbcs encoder you can only drop the pair.

The correct behavior is to aim for utf-8 where possible but especially on python 3.x where the surrogatepass handler was added explicitly for this purpose. Also, you can use the native Unicode filesystem APIs on windows.

If you want to make this easier I just finished implementing this in full in vistir (not sure if you want to depend on that in shellingham) but the final implementation fixes an edge case that was picked up separately by hypothesis

Source: https://github.com/sarugaku/vistir/pull/54
https://www.python.org/dev/peps/pep-0529/
Also, I wrote the original implementation of the shell handler for windows

techalchemy on 15 Dec 2018

We can afford to be lossy here since all we want to know is whether an executable is a shell. If the name fails to be decoded, it is definitely not a shell.

I’ve released 1.2.8 for Shellingham. This can be closed after the vendor upgrades for the next release.

uranusjr on 15 Dec 2018

thats ok with me then, although possibly not actually necessary? do we have a way to test it?

techalchemy on 15 Dec 2018

@pansila if you add an import:

from ctypes import create_unicode_buffer

and then you make the following changes:

         executable = create_unicode_buffer(pe.szExeFile)

         info = {'executable': executable.value}

I'd be a lot more comfortable letting windows handle this itself than randomly using mbcs for it, since python doesn't actually encode and decode mbcs correctly (which is why they switched to UTF8)

Does this also work? I'd love a test case (i.e. the code points) for the characters that are breaking -- you can get those with ord("character") -- that way we can test for this a bit more thoroughly

techalchemy on 16 Dec 2018

@techalchemy It fails with another exception.

pipenv --support

Traceback (most recent call last):
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\panzi\AppData\Local\Programs\Python\Python36\Scripts\pipenv.exe\__main__.py", line 5, in <module>
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\__init__.py", line 47, in <module>
    from .cli import cli
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\cli\__init__.py", line 3, in <module>
    from .command import cli
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\cli\command.py", line 7, in <module>
    import crayons
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\patched\crayons.py", line 49, in <module>
    is_powershell = "powershell" in shellingham.detect_shell()[0]
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham\__init__.py", line 22, in detect_shell
    shell = get_shell(pid, max_depth=max_depth)
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham\nt.py", line 106, in get_shell
    processes = dict(_iter_process())
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\shellingham\nt.py", line 79, in _iter_process
    executable = create_unicode_buffer(pe.szExeFile)
  File "c:\users\panzi\appdata\local\programs\python\python36\lib\ctypes\__init__.py", line 291, in create_unicode_buffer
    raise TypeError(init)
TypeError: b'[System Process]'

pansila on 17 Dec 2018

thanks a lot, I think you should pull a request to fix this error.

callmexss on 25 Dec 2018

This will be caught in vendored dependency updates as Tzu-ping mentioned

techalchemy on 25 Dec 2018

Still no success, under Windows7 russian, without russian letters in paths...

> mkdir test && cd test
> pipenv install
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 280: invalid continuation byte
Failed to create virtual environment.

Full output

```bash

pipenv install
Creating a virtualenv for this project…
Pipfile: C:UsersswasherPycharmProjectstestPipfile
Using c:usersswasherappdatalocalprogramspythonpython37python.exe (3.7.3) to create virtualenv…
[=== ] Creating virtual environment...Already using interpreter c:usersswasherappdatalocalprogramspythonpython37python.exe
Using base prefix 'c:\users\swasher\appdata\local\programs\python\python37'
New python executable in C:Usersswasher.virtualenvstest-tQmITBdNScriptspython.exe
Installing setuptools, pip, wheel...
done.

Failed creating virtual environment
File "C:UsersswasherAppDataRoamingPythonPython37site-packagesvirtualenv.py", line 903, in call_subprocess
line = line.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 280: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:usersswasherappdatalocalprogramspythonpython37librunpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:usersswasherappdatalocalprogramspythonpython37librunpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:UsersswasherAppDataRoamingPythonPython37site-packagesvirtualenv.py", line 2610, in
main()
File "C:UsersswasherAppDataRoamingPythonPython37site-packagesvirtualenv.py", line 834, in main
symlink=options.symlink,
File "C:UsersswasherAppDataRoamingPythonPython37site-packagesvirtualenv.py", line 1129, in create_environment
install_wheel(to_install, py_executable, search_dirs, download=download)
File "C:UsersswasherAppDataRoamingPythonPython37site-packagesvirtualenv.py", line 976, in install_wheel
_install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
File "C:UsersswasherAppDataRoamingPythonPython37site-packagesvirtualenv.py", line 1066, in _install_wheel_with_search_dir
call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=script)
File "C:UsersswasherAppDataRoamingPythonPython37site-packagesvirtualenv.py", line 905, in call_subprocess
line = line.decode(fs_encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 280: invalid continuation byte

Failed to create virtual environment.
```

> pipenv support

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: 'C:\\Users\\swasher\\AppData\\Roaming\\Python\\Python37\\site-packages\\pipenv'

Python location: 'c:\\users\\swasher\\appdata\\local\\programs\\python\\python37\\python.exe'

Python installations found:

3.7.3: C:\Users\swasher\AppData\Local\Programs\Python\Python37\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.3',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '7',
 'platform_system': 'Windows',
 'platform_version': '6.1.7601',
 'python_full_version': '3.7.3',
 'python_version': '3.7',
 'sys_platform': 'win32'}

System environment variables:

ACLOCAL_PATH
ALLUSERSPROFILE
ANSICON
ANSICON_DEF
APPDATA
CMDER_ROOT
COMMONPROGRAMFILES
COMPUTERNAME
COMSPEC
CONFIG_SITE
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
CONEMUANSI
CONEMUANSILOG
CONEMUARGS2
CONEMUARGS
CONEMUBACKHWND
CONEMUBASEDIR
CONEMUBASEDIRSHORT
CONEMUBUILD
CONEMUCFGDIR
CONEMUCONFIG
CONEMUDIR
CONEMUDRAWHWND
CONEMUDRIVE
CONEMUHWND
CONEMUHOOKS
CONEMUISADMIN
CONEMUPID
CONEMUPALETTE
CONEMUSERVERPID
CONEMUTASK
CONEMUWORKDIR
CONEMUWORKDRIVE
DISPLAY
EXEPATH
FP_NO_HOST_CHECK
HOME
HOMEDRIVE
HOMEPATH
HOSTNAME
INFOPATH
LANG
LOCALAPPDATA
LOGONSERVER
MANPATH
MINGW_CHOST
MINGW_PACKAGE_PREFIX
MINGW_PREFIX
MSYSTEM
MSYSTEM_CARCH
MSYSTEM_CHOST
MSYSTEM_PREFIX
NUMBER_OF_PROCESSORS
OLDPWD
ORIGINAL_PATH
ORIGINAL_TEMP
ORIGINAL_TMP
OS
PATH
PATHEXT
PKG_CONFIG_PATH
PLINK_PROTOCOL
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMFILES
PROMPT
PS1
PSMODULEPATH
PUBLIC
PWD
PROGRAMDATA
PROGRAMFILES(X86)
PROGRAMW6432
SESSIONNAME
SHELL
SHLVL
SSH_ASKPASS
SYSTEMDRIVE
SYSTEMROOT
TEMP
TERM
TMP
TMPDIR
USERDOMAIN
USERNAME
USERPROFILE
VAGRANT_HOME
VBOX_MSI_INSTALL_PATH
WINDIR
_
__COMPAT_LAYER
WINDOWS_TRACING_FLAGS
WINDOWS_TRACING_LOGFILE
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: C:\Users\swasher\bin;C:\cmder\bin;C:\cmder\vendor\bin;C:\Program Files\Git\bin;C:\cmder\vendor\git-for-windows\mingw64\bin;C:\cmder\vendor\git-for-windows\usr\local\bin;C:\cmder\vendor\git-for-windows\usr\bin;C:\cmder\vendor\git-for-windows\usr\bin;C:\cmder\vendor\git-for-windows\mingw64\bin;C:\cmder\vendor\git-for-windows\usr\bin;C:\Users\swasher\bin;C:\cmder\vendor\conemu-maximus5\ConEmu\Scripts;C:\cmder\vendor\conemu-maximus5;C:\cmder\vendor\conemu-maximus5\ConEmu;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\WindowsPowerShell\v1.0;C:\HashiCorp\Vagrant\bin;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\nodejs;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\swasher\AppData\Local\Programs\Python\Python37\Scripts;C:\Users\swasher\AppData\Local\Programs\Python\Python37;C:\cmder\bin;C:\Program Files\heroku\bin;C:\Users\swasher\AppData\Roaming\npm;C:\Users\swasher\AppData\Local\atom\bin;C:\Users\swasher\AppData\Roaming\Python\Python37\Scripts;C:\Users\swasher\AppData\Local\GitHubDesktop\bin;C:\cmder;C:\cmder\vendor\git-for-windows\usr\bin\vendor_perl;C:\cmder\vendor\git-for-windows\usr\bin\core_perl
SHELL: C:\cmder\vendor\git-for-windows\usr\bin\bash.exe
LANG: ru_RU.UTF-8
PWD: C:/Users/swasher/PycharmProjects/test

Contents of Pipfile ('C:\Users\swasher\PycharmProjects\test\Pipfile'):

UPDATE
My issue resolved. The reason is python was blocked by firewall at unusual path -
C:\Users\swasher\.virtualenvs\flask-vue-heroku-o7FqSK_s\Scripts. With new firewall rule is everything okay.

swasher on 5 May 2019

Same error when running pipenv install, on Windows 10 (Chinese). Caused by some network errors in Chinese. Fixed by adding an environment variable: PYTHONLEGACYWINDOWSFSENCODING=mbcs

nexcvon on 17 Aug 2019

👍3

Fixed this by going to - "c:Python38-32libsite-packagesvirtualenv.py" and changing a fs_encoding variable to = 'GBK'

helping083 on 20 Oct 2019

When can we include this fix? There are no releases for almost a year since the last pipenv release. https://github.com/sarugaku/shellingham/commit/381372076a075dd3e5eea9d1cfe6644e48b60aa8

pansila on 29 Oct 2019

👍3

When can we include this fix? There are no releases for almost a year since the last pipenv release. sarugaku/shellingham@3813720

I would also need a real fix for this, not a hack...

LaBlazer on 8 Nov 2019

thanks, the problem has been solved.

H-Madrid on 3 Jan 2020

Every time I got this error reboot system solves, but the error keep repeat after a while.
how did you solve it? @H-Madrid

kopes18 on 18 Feb 2020

error output:
File "c:\users\xxzz\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\shellingham\nt.py", line 78, in _iter_process info = {'executable': str(pe.szExeFile.decode('utf-8'))} UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2: invalid continuation byte
the error is at this line info = {'executable': str(pe.szExeFile.decode('utf-8'))}
I think change 'utf-8' to system encoding will fix the bug?

kopes18 on 18 Feb 2020

Fix this by replacing shellingham/nt.py [L85-L89]
to local C:\Users\LEE\AppData\Local\Programs\Python\Python38\Lib\site-packages\pipenv\vendor\shellingham\nt.py[#L78]