Pytest: more-itertools 6.0.0 release breaks pytest 4.2

[msnook]

We are seeing issues with new installations of pytest-4.2, as it pulls in more-itertools-6.0.0 (which was released about 2 hours ago at the time of writing this issue).

Is there any way we can lock in pytest-4.2 to use more-itertools-5.0.0 to maintain the python 2.7 compatability?

Here are the errors we are seeing with pytest-4.2 and more-itertools-6.0.0:

12-Feb-2019 02:48:58 | File "/usr/local/lib/python2.7/dist-packages/more_itertools/more.py", line 329
-- | --
12-Feb-2019 02:48:58 | def _collate(*iterables, key=lambda a: a, reverse=False):
12-Feb-2019 02:48:58 | ^
12-Feb-2019 02:48:58 | SyntaxError: invalid syntax

And here is a link to more-itertools release history, indicating that python 2.7 is not supported with more-itertools-6.0.0:
https://pypi.org/project/more-itertools/6.0.0/

[joshuahendinata]

I am using pytest==3.6.3 and it is still breaking this version

[RonnyPfannschmidt]

this is a release bug with more.itertools

[RonnyPfannschmidt]

i take it back, it has all the righ metadata

[RonnyPfannschmidt]

local testing indicates its supposed to work correctly, please provide more system information

[tayalakansh]

facing same issue with pytest 3.10.1. Error is: more-itertools requires Python '>=3.4' but the running Python is 2.7.5

[Zac-HD]

Please update your pip! more-itertools==6.0.0 does set the requires_python metadata, so up-to-date versions of pip will not attempt to install it on old versions of Python.

More generally, this isn't actually a pytest issue, and our only "fix" would be to prevent the upgrade for Python 3 users. Remember, Python 2 is end-of-life at the end of 2019, and Pytest will drop support at that time.

I'm leaving this open for now so others find the issue, but the only solution is pip install --upgrade pip.

[satepna]

I get the same error as @tayalakansh: pip detects the python version mismatch and then fails, instead of installing the last compatible version. So it's a cleaner error than OP, but the result (you can't use pytest with python2 currently) is the same:

$ virtualenv testenv
New python executable in /tmp/testenv/bin/python
Installing setuptools, pip, wheel...done.
$ . testenv/bin/activate
(testenv) $ python --version
Python 2.7.12
(testenv) $ pip install --upgrade pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: pip in ./testenv/lib/python2.7/site-packages (19.0.2)
(testenv) $ pip install pytest    
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting pytest
Collecting atomicwrites>=1.0 (from pytest)
Collecting py>=1.5.0 (from pytest)
Collecting pluggy>=0.7 (from pytest)
Requirement already satisfied: setuptools in ./testenv/lib/python2.7/site-packages (from pytest) (40.8.0)
Collecting six>=1.10.0 (from pytest)
Collecting pathlib2>=2.2.0; python_version < "3.6" (from pytest)
Collecting more-itertools>=4.0.0 (from pytest)
more-itertools requires Python '>=3.4' but the running Python is 2.7.12
(testenv) [1]$ 

[RonnyPfannschmidt]

what are the versions of pip and setuptools there? (i tested on python 2.7 and it installed correctly on pip as old as pip 9.x

[satepna]

what are the versions of pip and setuptools there? (i tested on python 2.7 and it installed correctly on pip as old as pip 9.x

pip 19.0.2 and setuptools 40.8.0 per this:

$ pip install --upgrade pip
Requirement already up-to-date: pip in ./testenv/lib/python2.7/site-packages (19.0.2)
$ pip install --upgrade setuptools
Requirement already up-to-date: setuptools in ./testenv/lib/python2.7/site-packages (40.8.0)

[fallenwood]

I locked more-itertools==5.0.0 in my project's requirements.txt :)
We are using pip 8.x, facing the same issue

[RonnyPfannschmidt]

i believe we will have to bring this up with setuptools/pip, personally i cant replicate the issue

[ThreePinkApples]

If you are running your own devpi-server you need to upgrade to 4.7 or newer to support 'requires_python'
Then it will give you the 5.0.0 package without having to ask for it specifically.
Or you can switch to go directly to pypi, then it should also work

https://pypi.org/project/devpi-server/

[RonnyPfannschmidt]

@ThreePinkApples thanks for that detail, this may be key

i wonder if any of the other users that see this particular issue use devpi < 4.7

[thoha]

Same issue here on Mac OS X 10.8.5, with the error message seen by the OP. Pip 18.0 and setuptools 40.4.3.

[yfried]

this happens for pip<9:

$ pip install -U "pip<9"
...
Successfully installed pip-8.1.2
$ pip install pytest
...
Collecting more-itertools>=4.0.0 (from pytest)
  Downloading https://files.pythonhosted.org/packages/dd/f4/cdfbb6f07f767e0cd8a11b16adfecc735b6eb87dbebda026ce8898e1cc22/more-itertools-6.0.0.tar.gz (68kB)

[nicoddemus]

@msnook can you confirm if you are using the latest pip version and if you are using devpi? Thanks!

[sambrightman]

I can make a PR from the mentioning branch if you like. I'm not sure if setuptools is supposed to respect transient environment markers? In my case the project using pytest is running python setup.py test, which seems to trigger the issue even with latest setuptools (40.8.0).

[nicoddemus]

Oh for python setup.py test indeed is a separate issue... I think the OP was about installing pytest using pip.

@sambrightman, just to confirm, if you install pytest using pip, do you get more-itertools 5 or 6?

[sambrightman]

If I just make a virtualenv for Python 2.7 and pip install pytest I get 5.0.0.

[nicoddemus]

Oh OK, so indeed there are two problems IIUC:

1. Users installing pytest using pip should make sure they are using the latest pip version. This should be enough to fix the problem.
2. Users running python setup.py test using pytest are getting more-itertools 6.0, so it seems like a setuptools problem. Users can fix this by pinning to more-itertools in their own project.

If people can confirm that 1) solves their problem it would be great.

About 2), there's already an issue open about it: https://github.com/pypa/setuptools/issues/1633.

[thoha]

As fallenwood already mentioned, pinning the version of more-itertools to 5.0.0 in your project requirements can serve as a temporary fix.

[nicoddemus]

As fallenwood already mentioned, pinning the version of more-itertools to 5.0.0 in your project requirements can serve as a temporary fix.

Oh sure, I will update my answer.

[thoha]

Perhaps I should also mention that we're getting the error message mentioned by the OP when attempting to run pytest for a buildout project, not when attempting to install pytest.

[nicoddemus]

Perhaps I should also mention that we're getting the error message mentioned by the OP when attempting to run pytest for a buildout project, not when attempting to install pytest.

That's expected I guess, as the package probably installs fine (specially from a wheel) but would get an error at runtime.

In your case, do you know if buildout uses pip to install your project dependencies? Are you executing pytest directly or via python setup.py test?

[sambrightman]

I think a substantial number of downstream projects will be using python setup.py test and it would be good to apply a workaround in pytest instead of requiring it from all of those projects individually. pypa/setuptools#1633 sounds unlikely to be fixed, whilst pytest claims 2.7 support for now.

[nicoddemus]

@sambrightman sounds reasonable.

I wonder if this would work:

INSTALL_REQUIRES = [
    ...
    "more-itertools>=4.0.0",
    'more-itertools<6;python_version<"3.4"',
    ...
]

🤔

@RonnyPfannschmidt do you know from the top of your head if the above would work and if setuptools would honor that when installing dependencies?

(Requiring <6 for all Python versions is not something we should do as it would limit Python 3 users to use more-itertools 6 in their own projects).

[sambrightman]

I've confirmed my branch - referenced above - fixes it in my context. I'll submit as a PR for discussion there.

[thoha]

I've upgraded my pip to the latest version (19.0.2). Buildout does indeed use pip, and I've now seen that similar errors show up during the buildout process:

File "build/bdist.macosx-10.8-x86_64/egg/more_itertools/more.py", line 329
    def _collate(*iterables, key=lambda a: a, reverse=False):
                               ^
SyntaxError: invalid syntax

  File "build/bdist.macosx-10.8-x86_64/egg/more_itertools/tests/test_more.py", line 892
    print(item, file=f)
                    ^
SyntaxError: invalid syntax

zip_safe flag not set; analyzing archive contents...
  File "/Users/thomashamann/PycharmProjects/hittech-api-layer/eggs/tmpLOQDGh/more_itertools-6.0.0-py2.7.egg/more_itertools/more.py", line 329
    def _collate(*iterables, key=lambda a: a, reverse=False):
                               ^
SyntaxError: invalid syntax

  File "/Users/thomashamann/PycharmProjects/hittech-api-layer/eggs/tmpLOQDGh/more_itertools-6.0.0-py2.7.egg/more_itertools/tests/test_more.py", line 892
    print(item, file=f)
                    ^
SyntaxError: invalid syntax

We run pytest using bin/py.test

[James-Crean]

Might I ask why pytest doesn't pin specific versions of requirements? Having installations randomly picking up new versions of dependencies makes for some annoying issues.

[nicoddemus]

Hi @James-Crean,

Because pytest is a tool/library which runs in the same environment as the code being tested, so pinning exact versions of its dependencies would force the pinned requirements on all users.

[nicoddemus]

Fixed, a workaround was implemented in #4774 and will be released in 4.2.1. 👍