Pre-commit: UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 867: character maps to <undefined>

[robertoea]

I got this following error when trying to do a pre-commit run:

An unexpected error has occurred: CalledProcessError: Command: ('C:\\Users\\rl\\.cache\\pre-commit\\repo16vk48wk\\py_env-python3.6\\Scripts\\pip.exe', 'install', '.')
Return code: 2
Expected return code: 0
Output: 
    Processing c:\users\rl\.cache\pre-commit\repo16vk48wk

Errors: 
    Exception:
    Traceback (most recent call last):
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\basecommand.py", line 228, in main
        status = self.run(options, args)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\commands\install.py", line 291, in run
        resolver.resolve(requirement_set)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\resolve.py", line 103, in resolve
        self._resolve_one(requirement_set, req)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\resolve.py", line 257, in _resolve_one
        abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\resolve.py", line 210, in _get_abstract_dist_for
        self.require_hashes
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\operations\prepare.py", line 324, in prepare_linked_requirement
        abstract_dist.prep_for_dist(finder, self.build_isolation)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\operations\prepare.py", line 126, in prep_for_dist
        build_requirements, isolate = self.req.get_pep_518_info()
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\req\req_install.py", line 448, in get_pep_518_info
        pp_toml = pytoml.load(f)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_vendor\pytoml\parser.py", line 10, in load
        return loads(fin.read(), translate=translate, filename=getattr(fin, 'name', repr(fin)))
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 867: character maps to <undefined>
    You are using pip version 10.0.1, however version 18.0 is available.
    You should consider upgrading via the 'python -m pip install --upgrade pip' command.


Traceback (most recent call last):
  File "c:\users\rl\source\repos\tests\env\lib\site-packages\pre_commit\error_handler.py", line 47, in error_handler
    yield
  File "c:\users\rl\source\repos\tests\env\lib\site-packages\pre_commit\main.py", line 261, in main
    return run(runner, store, args)
  File "c:\users\rl\source\repos\tests\env\lib\site-packages\pre_commit\commands\run.py", line 268, in run
    repo.require_installed()
  File "c:\users\rl\source\repos\tests\env\lib\site-packages\pre_commit\repository.py", line 195, in require_installed
    _install_all(self._venvs(), self.repo_config['repo'], self.store)
  File "c:\users\rl\source\repos\tests\env\lib\site-packages\pre_commit\repository.py", line 102, in _install_all
    language.install_environment(prefix, version, deps)
  File "c:\users\rl\source\repos\tests\env\lib\site-packages\pre_commit\languages\python.py", line 145, in install_environment
    prefix, ('pip', 'install', '.') + additional_dependencies,
  File "c:\users\rl\source\repos\tests\env\lib\site-packages\pre_commit\languages\helpers.py", line 9, in run_setup_cmd
    cmd_output(*cmd, cwd=prefix.prefix_dir, encoding=None)
  File "c:\users\rl\source\repos\tests\env\lib\site-packages\pre_commit\util.py", line 177, in cmd_output
    returncode, cmd, retcode, output=(stdout, stderr),
pre_commit.util.CalledProcessError: Command: ('C:\\Users\\rl\\.cache\\pre-commit\\repo16vk48wk\\py_env-python3.6\\Scripts\\pip.exe', 'install', '.')
Return code: 2
Expected return code: 0
Output: 
    Processing c:\users\rl\.cache\pre-commit\repo16vk48wk

Errors: 
    Exception:
    Traceback (most recent call last):
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\basecommand.py", line 228, in main
        status = self.run(options, args)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\commands\install.py", line 291, in run
        resolver.resolve(requirement_set)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\resolve.py", line 103, in resolve
        self._resolve_one(requirement_set, req)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\resolve.py", line 257, in _resolve_one
        abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\resolve.py", line 210, in _get_abstract_dist_for
        self.require_hashes
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\operations\prepare.py", line 324, in prepare_linked_requirement
        abstract_dist.prep_for_dist(finder, self.build_isolation)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\operations\prepare.py", line 126, in prep_for_dist
        build_requirements, isolate = self.req.get_pep_518_info()
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_internal\req\req_install.py", line 448, in get_pep_518_info
        pp_toml = pytoml.load(f)
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\site-packages\pip\_vendor\pytoml\parser.py", line 10, in load
        return loads(fin.read(), translate=translate, filename=getattr(fin, 'name', repr(fin)))
      File "c:\users\rl\.cache\pre-commit\repo16vk48wk\py_env-python3.6\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 867: character maps to <undefined>
    You are using pip version 10.0.1, however version 18.0 is available.
    You should consider upgrading via the 'python -m pip install --upgrade pip' command.

My .pre-commit-config.yaml:

repos:
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
      language_version: python3.6

Running Python 3.6.6 on Windows 10.

Replacing virtualenv's pip with pip 18.0 did seem to fix the issue, but I'm not sure if this will cause problem somewhere else.

[asottile]

nothing really pre-commit can do about this it seems. It's a bug with pip itself parsing black's pyproject.toml which contains UTF-8 characters (specifically on windows)

Since it appears you're on windows setting LANG=C.UTF-8 probably won't help, you might try using python3.7 instead of python3.6 (which will default to UTF-8 mode!).

[asottile]

more context:

• when virtualenv is released (hopefully later this week) with updated pip this should become magicallyfixed
• a workaround is to use an older version of black which doesn't haven't this character. 18.6b4 appears to be the newest of these: example PR

[madig]

I use pre-commit with Python 3.7.1 on Windows and it doesn't work :( Is there any way of injecting a newer pip myself somehow?

[asottile]

upgrade to virtualenv>=16.1.0 (just released) and this is resolved.