Pip-tools: pip-compile fails on ipython package spec

[alflanagan]

Describe the issue briefly here.

pip version: 8.1.2
pip-tools version: 7.0.0
python version: 3.5.1 (virtualenv)

When attempting to pip-compile a file containing the spec "ipython", the pkg_resources package raises an exception.

Steps to replicate

Note: i'm having trouble replicating this. It may be necessary to start with an empty cache.

1.

$ cat > test.in
ipython

2.

$ pip-compile test.in

Expected result

creates test.txt with ipython==5.0.0.

Actual result

Traceback (most recent call last):
  File "/home/aflanagan/Envs/PyBadge/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/piptools/scripts/compile.py", line 195, in cli
    reverse_dependencies = resolver.reverse_dependencies(results)
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/piptools/resolver.py", line 267, in reverse_dependencies
    return self.dependency_cache.reverse_dependencies(non_editable)
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/piptools/cache.py", line 139, in reverse_dependencies
    return self._reverse_dependencies(ireqs_as_cache_values)
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/piptools/cache.py", line 163, in _reverse_dependencies
    for name, version_and_extras in cache_keys
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/piptools/utils.py", line 191, in lookup_table
    for value in values:
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/piptools/cache.py", line 164, in <genexpr>
    for dep_name in self.cache[name][version_and_extras])
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 3047, in parse
    req, = parse_requirements(s)
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2991, in parse_requirements
    "version spec")
  File "/home/aflanagan/Envs/PyBadge/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2956, in scan_list
    raise RequirementParseError(msg, line, "at", line[p:])
pkg_resources.RequirementParseError: Expected version spec in pexpect; sys_platform != "win32" at ; sys_platform != "win32"

[Groxx]

It also seems to be failing on pickleshare 0.7.3. Compiling with verbose includes output like this under "Finding secondary dependencies":
pickleshare==0.7.3 requires pathlib2; python_version in "2.6 2.7 3.2 3.3"
and has a traceback:

Traceback (most recent call last):
  File "/home/myself/irslib/env/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/piptools/scripts/compile.py", line 194, in cli
    reverse_dependencies = resolver.reverse_dependencies(results)
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/piptools/resolver.py", line 242, in reverse_dependencies
    return self.dependency_cache.reverse_dependencies(non_editable)
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/piptools/cache.py", line 139, in reverse_dependencies
    return self._reverse_dependencies(ireqs_as_cache_values)
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/piptools/cache.py", line 163, in _reverse_dependencies
    for name, version_and_extras in cache_keys
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/piptools/utils.py", line 181, in lookup_table
    for value in values:
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/piptools/cache.py", line 164, in <genexpr>
    for dep_name in self.cache[name][version_and_extras])
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3008, in parse
    req, = parse_requirements(s)
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2952, in parse_requirements
    "version spec")
  File "/home/myself/irslib/env/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2917, in scan_list
    raise RequirementParseError(msg, line, "at", line[p:])
pkg_resources.RequirementParseError: Expected version spec in pathlib2; python_version in "2.6 2.7 3.2 3.3" at ; python_version in "2.6 2.7 3.2 3.3"

IPython 5.0.0 fails similarly, probably for the same reason. Output for IPython:
ipython==5.0.0 requires backports.shutil-get-terminal-size; python_version == "2.7", decorator, pathlib2; python_version == "2.7" or python_version == "3.3", pexpect; sys_platform != "wi n32", pickleshare, prompt-toolkit<2.0.0,>=1.0.3, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2

[DanielArndt]

Getting the same thing with notebook (which in turn would require ipython).

[DanielArndt]

Seems this can be fixed by upgrading to a newer version of setuptools. Anything above v20.2.2 seems to work.

I went to add this to install_requires but I guess that is considered unsafe.

Interested to know what the correct way of "permanently" fixing it is.

[scottbelden]

I think I'm having a similar issue with flake8. When I upgraded to setuptools 25.1.0 it works, but here's the output of the run with setuptools 20.0:

$ pip list
click (6.6)
first (2.0.1)
pip (8.1.2)
pip-tools (1.7.0)
setuptools (20.0)
six (1.10.0)
wheel (0.29.0)

$ pip-compile -r -v requirements.in

                          ROUND 1                           
Current constraints:
  flake8

Finding the best candidates:
  found candidate flake8==3.0.2 (constraint was <any>)

Finding secondary dependencies:
  flake8==3.0.2 not in cache, need to check index
  flake8==3.0.2             requires configparser; python_version < "3.2", enum34; python_version < "3.4", mccabe<0.6.0,>=0.5.0, pycodestyle<2.1.0,>=2.0.0, pyflakes!=1.2.0,!=1.2.1,!=1.2.2,<1.3.0,>=0.8.1

New dependencies found in this round:
  adding ['configparser', '', '[]']
  adding ['enum34', '', '[]']
  adding ['mccabe', '<0.6.0,>=0.5.0', '[]']
  adding ['pycodestyle', '<2.1.0,>=2.0.0', '[]']
  adding ['pyflakes', '!=1.2.0,!=1.2.1,!=1.2.2,<1.3.0,>=0.8.1', '[]']
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  configparser
  enum34
  flake8
  mccabe<0.6.0,>=0.5.0
  pycodestyle<2.1.0,>=2.0.0
  pyflakes!=1.2.0,!=1.2.1,!=1.2.2,<1.3.0,>=0.8.1

Finding the best candidates:
  found candidate configparser==3.5.0 (constraint was <any>)
  found candidate enum34==1.1.6 (constraint was <any>)
  found candidate flake8==3.0.2 (constraint was <any>)
  found candidate mccabe==0.5.1 (constraint was >=0.5.0,<0.6.0)
  found candidate pycodestyle==2.0.0 (constraint was >=2.0.0,<2.1.0)
  found candidate pyflakes==1.2.3 (constraint was >=0.8.1,!=1.2.0,!=1.2.1,!=1.2.2,<1.3.0)

Finding secondary dependencies:
  mccabe==0.5.1 not in cache, need to check index
  mccabe==0.5.1             requires -
  configparser==3.5.0 not in cache, need to check index
  configparser==3.5.0       requires -
  enum34==1.1.6 not in cache, need to check index
  enum34==1.1.6             requires -
  pycodestyle==2.0.0 not in cache, need to check index
  pycodestyle==2.0.0        requires -
  flake8==3.0.2             requires configparser; python_version < "3.2", enum34; python_version < "3.4", mccabe<0.6.0,>=0.5.0, pycodestyle<2.1.0,>=2.0.0, pyflakes!=1.2.0,!=1.2.1,!=1.2.2,<1.3.0,>=0.8.1
  pyflakes==1.2.3 not in cache, need to check index
  pyflakes==1.2.3           requires -
------------------------------------------------------------
Result of round 2: stable, done

Traceback (most recent call last):
  File "/Users/scott/.virtualenvs/piptools/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/piptools/scripts/compile.py", line 195, in cli
    reverse_dependencies = resolver.reverse_dependencies(results)
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/piptools/resolver.py", line 267, in reverse_dependencies
    return self.dependency_cache.reverse_dependencies(non_editable)
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/piptools/cache.py", line 139, in reverse_dependencies
    return self._reverse_dependencies(ireqs_as_cache_values)
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/piptools/cache.py", line 163, in _reverse_dependencies
    for name, version_and_extras in cache_keys
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/piptools/utils.py", line 191, in lookup_table
    for value in values:
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/piptools/cache.py", line 164, in <genexpr>
    for dep_name in self.cache[name][version_and_extras])
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3047, in parse
    req, = parse_requirements(s)
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2991, in parse_requirements
    "version spec")
  File "/Users/scott/.virtualenvs/piptools/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2956, in scan_list
    raise RequirementParseError(msg, line, "at", line[p:])
pkg_resources.RequirementParseError: Expected version spec in configparser; python_version < "3.2" at ; python_version < "3.2"

[veatch]

The setuptools docs recommend using their ez_setup.py script to download and install setuptools if the required version isn't present. So maybe it's as simple as adding ez_setup.py to pip-tools and adding this to the top of setup.py:

import ez_setup
ez_setup.use_setuptools(version='20.2.2')

But I'm not confident about whether that's the right solution here.

[nkuttler]

I seem to have the same problem with pylint.

ValueError: ('Expected version spec in', u'backports.functools-lru-cache; python_version == "2.7"', 'at', u'; python_version == "2.7"')

Upgrading setuptools indeed fixes this problem, but it would be good to give users a more useful error message.

[k0nG]

Also getting similar issue with Faker. It seems the 'extras_require' is where different python versions are specified is the problem:
https://github.com/joke2k/faker/blob/v0.7.2/setup.py#L70-L80

I've updated to pip-tools 1.8.0 and updated setuptools to 32.0.0 and that's seem to have done the trick.

[vphilippon]

This was a bug with setuptools likely having trouble parsing environment markers (python_version and extra). Like many noted, this was fixed in newer setuptools version (28.7.1 and higher).

@nkuttler It's going to be hard to get a nicer error message in that kind of case, as this is an unexpected bug from setuptools, the kind that "should never happen again".

I'll close this as it's fixed, feel free to reopen if needed.
Thanks!

[peteratticusberg]

Just posted that I had an issue here with a newer version of setuptools but then deleted it as it turns out I misundertood the OPs issue, apologies for the confusion

[bepetersn]

I'm having a similar sounding issue presently with setuptools 38.x and pip-tools 1.11, though the final exception is ValueError not RequirementParseError. The error happens after I add ipython to my requirements.in: ipython<6.0. Perhaps this version req just can't work? Using a 2.7 python.

Traceback (most recent call last):
  File "/var/www/matin/app/pyenv/bin/pip-compile", line 11, in <module>
    sys.exit(cli())
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/piptools/scripts/compile.py", line 225, in cli
    reverse_dependencies = resolver.reverse_dependencies(results)
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/piptools/resolver.py", line 297, in reverse_dependencies
    return self.dependency_cache.reverse_dependencies(non_editable)
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/piptools/cache.py", line 139, in reverse_dependencies
    return self._reverse_dependencies(ireqs_as_cache_values)
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/piptools/cache.py", line 163, in _reverse_dependencies
    for name, version_and_extras in cache_keys
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/piptools/utils.py", line 200, in lookup_table
    for value in values:
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/piptools/cache.py", line 164, in <genexpr>
    for dep_name in self.cache[name][version_and_extras])
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 2582, in parse
    reqs = list(parse_requirements(s))
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 2507, in parse_requirements
    line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
  File "/var/www/matin/app/pyenv/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 2475, in scan_list
    raise ValueError("Expected "+item_name+" in",line,"at",line[p:])
ValueError: ('Expected version spec in', u'pathlib2; python_version in "2.6 2.7 3.2 3.3"', 'at', u'; python_version in "2.6 2.7 3.2 3.3"')

[vphilippon]

The distribute version (0.6.24) seems pretty old, although I can't compare with my setup at the moment (on my phone).
You can try creating a virtualenv using virtualenv 15.1.0, confirm that the pip-compile works, and check the version of distribute installed

[bepetersn]

Upgrading distribute did it for me! Thanks.