Pip: UnicodeDecodeError When trying to install something with pip2

[Andrej730]

C:\Users\Andrey>pip2 install scrapy
Collecting scrapy
Exception:
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\pip\basecommand.py", line 223, in main
    status = self.run(options, args)
  File "C:\Python27\lib\site-packages\pip\commands\install.py", line 280, in run

    requirement_set.prepare_files(finder)
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 317, in prepare_
files
    functools.partial(self._prepare_file, finder))
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 304, in _walk_re
q_to_install
    more_reqs = handler(req_to_install)
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 439, in _prepare
_file
    req_to_install.populate_link(finder, self.upgrade)
  File "C:\Python27\lib\site-packages\pip\req\req_install.py", line 244, in popu
late_link
    self.link = finder.find_requirement(self, upgrade)
  File "C:\Python27\lib\site-packages\pip\index.py", line 486, in find_requireme
nt
    all_versions = self._find_all_versions(req.name)
  File "C:\Python27\lib\site-packages\pip\index.py", line 404, in _find_all_vers
ions
    index_locations = self._get_index_urls_locations(project_name)
  File "C:\Python27\lib\site-packages\pip\index.py", line 378, in _get_index_url
s_locations
    page = self._get_page(main_index_url)
  File "C:\Python27\lib\site-packages\pip\index.py", line 810, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "C:\Python27\lib\site-packages\pip\index.py", line 920, in get_page
    "Cache-Control": "max-age=600",
  File "C:\Python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 47
7, in get
    return self.request('GET', url, **kwargs)
  File "C:\Python27\lib\site-packages\pip\download.py", line 373, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "C:\Python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 46
5, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 60
5, in send
    r.content
  File "C:\Python27\lib\site-packages\pip\_vendor\requests\models.py", line 750,
 in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes
()
  File "C:\Python27\lib\site-packages\pip\_vendor\requests\models.py", line 673,
 in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "C:\Python27\lib\site-packages\pip\_vendor\requests\packages\urllib3\resp
onse.py", line 307, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "C:\Python27\lib\site-packages\pip\_vendor\requests\packages\urllib3\resp
onse.py", line 243, in read
    data = self._fp.read(amt)
  File "C:\Python27\lib\site-packages\pip\_vendor\cachecontrol\filewrapper.py",
line 54, in read
    self.__callback(self.__buf.getvalue())
  File "C:\Python27\lib\site-packages\pip\_vendor\cachecontrol\controller.py", l
ine 244, in cache_response
    self.serializer.dumps(request, response, body=body),
  File "C:\Python27\lib\site-packages\pip\download.py", line 276, in set
    return super(SafeFileCache, self).set(*args, **kwargs)
  File "C:\Python27\lib\site-packages\pip\_vendor\cachecontrol\caches\file_cache
.py", line 99, in set
    with self.lock_class(name) as lock:
  File "C:\Python27\lib\site-packages\pip\_vendor\lockfile\mkdirlockfile.py", li
ne 18, in __init__
    LockBase.__init__(self, path, threaded, timeout)
  File "C:\Python27\lib\site-packages\pip\_vendor\lockfile\__init__.py", line 18
9, in __init__
    hash(self.path)))
  File "C:\Python27\lib\ntpath.py", line 85, in join
    result_path = result_path + p_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 0: ordinal
not in range(128)

Crash happening on this join at site-packages\pip_vendor\lockfile__init__.py:

self.unique_name = os.path.join(dirname,
                                        "%s%s.%s%s" % (self.hostname,
                                                       self.tname,
                                                       self.pid,
                                                       hash(self.path)))

hostname, tname, pid, path before crash:

('\xcf\xca', '-2b24', 11924, u'c:\\users\\andrey\\appdata\\local\\pip\\cache\\h
tp\\b\\9\\e\\6\\f\\b9e6fc103d2676598ac31ba30018a5110b859143b5cb8105c3666ca2')

[pfmoore]

This looks like a duplicate of #2501. It appears to be a bug in the lockfile module which pip vendors in. Could you raise a bug against the pylockfile project, and we'll vendor in a new version when they fix it.

[Andrej730]

If someone got the same error - i have described way to solve it in #2501

[nedbat]

@Andrej730 should this still be an open issue for pip?

[Choumingzhao]

I got the same error in Python2.7.13 on Windows 10 with my username setting to __Chinese__, and the problem is solved with following coding adding to __site-packages\pip_vendor\lockfile\__init__.py__ right after all other import lines.( code are copied from stackoverflow and some small edit)

import sys
# While sys.setdefaultencoding() return "utf-8" in python3 in my PC
# it return "ascii" in python2, so I changed it to OEM(GBK for me)
reload(sys)  # Reload does the trick!
sys.setdefaultencoding('GBK')  #  you could try it with your OEM encoding.

But some packages depends on Microsoft Visual C++ compilers still failed with encoding problem, so I do the same to the file __site-packagessetuptools\msvc.py__ and it worked !

If problems are not solved with the trick above, maybe you can try this.

[pradyunsg]

@pfmoore Would #4486 have fixed this?

[pfmoore]

@pradyunsg I don't think so. As I noted, it's a problem in the lockfile module that needs to be reported to them. There's no indication that I can see that anyone ever did (although there's even a suggested patch noted in the discussion here :cry:).

[pradyunsg]

@pfmoore lockfile is now deprecated. We should probably shift to fasteners or something.

[pfmoore]

Probably. I've no feel for how easy that will be to do.

[pradyunsg]

4766. :)

[gutsytechster]

Is this still valid as https://github.com/pypa/pip/pull/7023 has already been merged? If not, then I think it can be closed :)

[uranusjr]

Maybe? I’ll add an awaiting response label and let the bot close it automatically if no-one raises any objections. Please feel free to respond if the problem still exists.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.