Pip-tools: Pip compile breaks with scandir==1.4

[milin]

pip-compiles fails for scandir==1.4 dependency

```
scandir==1.4 not in cache, need to check index
Traceback (most recent call last):
File "/www/env/bin/pip-compile", line 11, in
sys.exit(cli())
File "/www/env/lib/python2.7/site-packages/click/core.py", line 716, in __call__
return self.main(args, kwargs)
File "/www/env/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/www/env/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/www/env/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(args, *kwargs)
File "/www/env/lib/python2.7/site-packages/piptools/scripts/compile.py", line 168, in cli
results = resolver.resolve()
File "/www/env/lib/python2.7/site-packages/piptools/resolver.py", line 105, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/www/env/lib/python2.7/site-packages/piptools/resolver.py", line 187, in _resolve_one_round
for best_match in best_matches
File "/www/env/lib/python2.7/site-packages/piptools/resolver.py", line 188, in
for dep in self._iter_dependencies(best_match))
File "/www/env/lib/python2.7/site-packages/piptools/resolver.py", line 261, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
File "/www/env/lib/python2.7/site-packages/piptools/repositories/pypi.py", line 161, in get_dependencies
dependencies = reqset._prepare_file(self.finder, ireq)
File "/www/env/lib/python2.7/site-packages/pip/req/req_set.py", line 587, in _prepare_file
session=self.session, hashes=hashes)
File "/www/env/lib/python2.7/site-packages/pip/download.py", line 810, in unpack_url
hashes=hashes
File "/www/env/lib/python2.7/site-packages/pip/download.py", line 653, in unpack_http_url
unpack_file(from_path, location, content_type, link)
File "/www/env/lib/python2.7/site-packages/pip/utils/__init__.py", line 605, in unpack_file
untar_file(filename, location)
File "/www/env/lib/python2.7/site-packages/pip/utils/__init__.py", line 551, in untar_file
path = os.path.join(location, fn)
File "/www/env/lib/python2.7/posixpath.py", line 80, in join
path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc6 in position 27: ordinal not in range(128)
make: * [install] Error 1

Steps to replicate

1. ...have scandir==1.4 in requirements file
2. ...
3. ...

Expected result

Should work
...

Actual result

Errors out
...

[jochen-ott-by]

I think the problem is the from __future__ import unicode_literals in piptools/_compat/tempfile.py, so this succeeds:

tmpdir = TemporaryDirectory()
assert isinstance(tmpdir.name, unicode)

This unicode directory name causes problems if trying to append a binary str via os.path.join which is not ASCII-encoded.

I propose to either entirely remove the unicode_literals import from the above file or to explicitly use binary strings for the TemporaryDirectory.__init__ default values.

[lefterisnik]

Hi @byjott, I tried both with no success. I get the same error

[jochen-ott-by]

Yes, indeed, that's not enough, as apparently TemporaryDirectory is constructed with non-default arguments in a file also using unicode_literals (specifically from repositories/pypi.py:67).

What finally did work for me was to edit piptools/_compat/tempfile.py and insert the following lines at the top of TemporaryDirectory.__init__:

suffix = suffix.encode()
prefix = prefix.encode()
if dir is not None:
    dir = dir.encode()

This ensures the temporary directory name (self.name in this context) is a binary str, not unicode.

[lefterisnik]

Also, pip-compile works fine when you build pip-tools under python3.

@byjott I will try your solution and I will write back for feedback

[lefterisnik]

@byjott, yes your solution works fine. thanks.

[federicobond]

I can reproduce the problem and confirm that the solution works fine.

[Groxx]

I managed to catch the error:

[localhost] local: pip-compile --no-index --upgrade requirements.in
> /.../env/local/lib/python2.7/site-packages/pip/utils/__init__.py(496)unzip_file()
-> print "caught"
(Pdb) l
491                     fn = split_leading_dir(name)[1]
492                 try:
493                     fn = os.path.join(location, fn)
494                 except:
495                     import pdb; pdb.set_trace()
496  ->                 print "caught"
497                     raise
498                 dir = os.path.dirname(fn)
499                 if fn.endswith('/') or fn.endswith('\\'):
500                     # A directory
501                     ensure_dir(fn)
(Pdb) location
u'/tmp/tmpDj6_ltbuild/scandir'
(Pdb) fn
'test/testdir/subdir/unicod\xc6\x8f.txt'
(Pdb)

Unfortunately, after fixing it once with @byjott's approach, I was unable to reproduce - if you're debugging, watch out! I can reproduce at-will now though, if anyone would like more info.