Pip-tools: pip-compile fails with TypeError on Ubuntu 16.04 + Python 3.5

[tadams42]

Create virtual env and install pip and pip-tools:

$ cd /tmp
$ pyvenv .venv
$ source .venv/bin/activate
$ pip install -U pip wheel                                                                                                              
Requirement already up-to-date: pip in ./.venv/lib/python3.5/site-packages
Collecting wheel
  Using cached wheel-0.29.0-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.29.0
$ pip install -U pip-tools                                                                                                              
Collecting pip-tools
  Using cached pip_tools-1.6.1-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools)
  Using cached click-6.6.tar.gz
Collecting six (from pip-tools)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting first (from pip-tools)
  Using cached first-2.0.1-py2.py3-none-any.whl
Building wheels for collected packages: click
  Running setup.py bdist_wheel for click ... done
  Stored in directory: /home/tomislav/.cache/pip/wheels/b0/6d/8c/cf5ca1146e48bc7914748bfb1dbf3a40a440b8b4f4f0d952dd
Successfully built click
Installing collected packages: click, six, first, pip-tools
Successfully installed click-6.6 first-2.0.1 pip-tools-1.6.1 six-1.10.0

Then compile requirements.in:

$ echo "flask" > requirements.in                                                                                                        
$ pip-compile requirements.in                                                                                                           
Traceback (most recent call last):
  File "/tmp/.venv/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/tmp/.venv/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/.venv/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/tmp/.venv/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/.venv/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/tmp/.venv/lib/python3.5/site-packages/piptools/scripts/compile.py", line 163, in cli
    results = resolver.resolve()
  File "/tmp/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 78, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/tmp/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 154, in _resolve_one_round
    best_matches = set(self.get_best_match(ireq) for ireq in constraints)
  File "/tmp/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 154, in <genexpr>
    best_matches = set(self.get_best_match(ireq) for ireq in constraints)
  File "/tmp/.venv/lib/python3.5/site-packages/piptools/resolver.py", line 201, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/tmp/.venv/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 103, in find_best_match
    prereleases=prereleases)
  File "/tmp/.venv/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/specifiers.py", line 753, in filter
  File "/tmp/.venv/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/version.py", line 31, in parse
  File "/tmp/.venv/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/version.py", line 200, in __init__
TypeError: expected string or bytes-like object

[nvie]

Weird, I cannot replicate this locally on a Mac. I was using the exact same commands you were using here. Must be some platform-specific issue.

[tadams42]

Definitely platform specific, it worked on earlier Ubuntu and Python 3.4. I'll try to find time later, to dig deeper into this one

[taynaud]

It is not specific to python 3.5.1.

In ubuntu 16.04:

[~]$ mkdir test                                                                                                              
[~]$ cd test                                                                                                                 
[~/test]$ echo "wheel" > requirements.in                                                                                     
[~/test]$ pip-compile requirements.in                                                                                        
Traceback (most recent call last):
  File "/usr/local/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/piptools/scripts/compile.py", line 163, in cli
    results = resolver.resolve()
  File "/usr/local/lib/python2.7/dist-packages/piptools/resolver.py", line 78, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/dist-packages/piptools/resolver.py", line 154, in _resolve_one_round
    best_matches = set(self.get_best_match(ireq) for ireq in constraints)
  File "/usr/local/lib/python2.7/dist-packages/piptools/resolver.py", line 154, in <genexpr>
    best_matches = set(self.get_best_match(ireq) for ireq in constraints)
  File "/usr/local/lib/python2.7/dist-packages/piptools/resolver.py", line 201, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python2.7/dist-packages/piptools/repositories/pypi.py", line 103, in find_best_match
    prereleases=prereleases)
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/specifiers.py", line 753, in filter
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/version.py", line 31, in parse
  File "/usr/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/version.py", line 200, in __init__
TypeError: expected string or buffer

[tadams42]

Tried like this (incorporating pull request #350):

pyvenv .venv
source .venv/bin/activate
pip install -U pip
pip install -e git://github.com/taynaud/pip-tools.git@ca3ef2e74cd2058fca903a67656e739e93531023#egg=pip-tools
echo "flask" >> requirements.in
pip-compile requirements.in
pip-sync requirements.txt

pip-compile now passes, pip-sync doesn't:

pip-sync requirements.txt
Uninstalling pkg-resources-0.0.0:
  Successfully uninstalled pkg-resources-0.0.0
Collecting Jinja2==2.8
  Using cached Jinja2-2.8-py2.py3-none-any.whl
Collecting MarkupSafe==0.23
Collecting Werkzeug==0.11.9
  Using cached Werkzeug-0.11.9-py2.py3-none-any.whl
Collecting flask==0.10.1
  Using cached Flask-0.10.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/.venv/lib/python3.5/site-packages/setuptools/__init__.py", line 11, in <module>
        from setuptools.extern.six.moves import filterfalse, map
      File "/tmp/.venv/lib/python3.5/site-packages/setuptools/extern/__init__.py", line 1, in <module>
        from pkg_resources.extern import VendorImporter
    ImportError: No module named 'pkg_resources'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-i3gandlh/flask/
Traceback (most recent call last):
  File "/tmp/.venv/bin/pip-sync", line 9, in <module>
    load_entry_point('pip-tools', 'console_scripts', 'pip-sync')()
  File "/tmp/.venv/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/.venv/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/tmp/.venv/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/.venv/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/tmp/.venv/src/pip-tools/piptools/scripts/sync.py", line 75, in cli
    install_flags=install_flags))
  File "/tmp/.venv/src/pip-tools/piptools/sync.py", line 156, in sync
    check_call([pip, 'install'] + pip_flags + install_flags + sorted(to_install))
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip', 'install', 'Jinja2==2.8', 'MarkupSafe==0.23', 'Werkzeug==0.11.9', 'flask==0.10.1', 'itsdangerous==0.24']' returned non-zero exit status

[tadams42]

After I do the following pip-sync passes:

pip uninstall setuptools 
pip install -U setuptools 
pip-sync requirements.txt 

Why does first pip-sync do the following?

pip-sync requirements.txt
Uninstalling pkg-resources-0.0.0:
  Successfully uninstalled pkg-resources-0.0.0

[juyrjola]

I'm encountering this problem as well with Ubuntu 16.04 and Python 3.5. Apparently specifier.filter expects to have version as class pkg_resources.extern.packaging.version.Version, but it is provided with class pip._vendor.packaging.version.Version. filter then assumes the passed item is a string, and attempts to parse it.

[juyrjola]

Making sure we pass strings to specifier.filter fixes the problem, but I'm not sure if it's the proper fix:

diff --git a/piptools/repositories/pypi.py b/piptools/repositories/pypi.py
index 79c8674..0a3849c 100644
--- a/piptools/repositories/pypi.py
+++ b/piptools/repositories/pypi.py
@@ -95,8 +95,8 @@ class PyPIRepository(BaseRepository):
             return ireq  # return itself as the best match

         all_candidates = self.find_all_candidates(ireq.name)
-        candidates_by_version = lookup_table(all_candidates, key=lambda c: c.version, unique=True)
-        matching_versions = ireq.specifier.filter((candidate.version for candidate in all_candidates),
+        candidates_by_version = lookup_table(all_candidates, key=lambda c: str(c.version), unique=True)
+        matching_versions = ireq.specifier.filter((str(candidate.version) for candidate in all_candidates),
                                                   prereleases=prereleases)

         # Reuses pip's internal candidate sort key to sort

[mkoistinen]

I'm on Mac OS X 10.11.4 with latest command line tools (v7.3) and this exact thing happens to me as well, but only under Python 3.5. @nvie which version of the command line tools are you running?

[sfriesel]

The second part @tadamic mentioned (Uninstalling pkg-resources-0.0.0) could be fixed by adding "pkg-resources" to PACKAGES_TO_IGNORE in sync.py because it should probably not be touched in any case. I don't know if that would also fix the original error as well (I didn't encounter it)

[tadams42]

Latest try on Python 3.5.2, Ubuntu 16.04.1 LTS, pip==9.0.1, pip-tools==1.8.0

• pip-compile passes without problems and without any workaround required
• pip-sync fails:

~
pip-sync requirements.txt
Uninstalling pkg-resources-0.0.0:
Successfully uninstalled pkg-resources-0.0.0
Collecting Jinja2==2.9.4
Using cached Jinja2-2.9.4-py2.py3-none-any.whl
Collecting MarkupSafe==0.23
Using cached MarkupSafe-0.23.tar.gz
Could not import setuptools which is required to install from a source distribution.
Traceback (most recent call last):
File "/tmp/.venv/lib/python3.5/site-packages/pip/req/req_install.py", line 387, in setup_py
import setuptools # noqa
File "/tmp/.venv/lib/python3.5/site-packages/setuptools/__init__.py", line 11, in
from setuptools.extern.six.moves import filterfalse, map
File "/tmp/.venv/lib/python3.5/site-packages/setuptools/extern/__init__.py", line 1, in
from pkg_resources.extern import VendorImporter
ImportError: No module named 'pkg_resources'
~

@nvie, I'd say the remaining problem in this issue is equal to one described in #422 so

• close one of them as duplicate
• apply solution proposed by @sfriesel to close both, this one and #422

[siulkilulki]

For me the same. pip-sync uninstalls pkg-resources-0.0.0

[vphilippon]

Should be fixed: pkg_resources was added to PACKAGES_TO_IGNORE.