Pytest: 3.7.0: Hookspec method signature regression

[jperras]

When upgrading from pytest 3.6.x to 3.7, it seems as though the signature for hookspec has changed:

Traceback (most recent call last):
  File "<redacted>/python3.6/site-packages/_pytest/hookspec.py", line 27, in <module>
    @hookspec(historic=True, warn_on_impl=PYTEST_NAMESPACE)
TypeError: __call__() got an unexpected keyword argument 'warn_on_impl'
ERROR: InvocationError for command '<redacted>/bin/pytest -x --pdb' (exited with code 1)

Reverting back to 3.6.4 fixes the issue.

Plugins being used:

    pytest-mock
    pytest-asyncio
    pytest_profiling

[nicoddemus]

Hi @jperras,

The hook signature did not change, but we are passing the warn_on_impl option to pluggy which is featured in pluggy 0.7. Which pluggy version do you have installed?

[jperras]

I believe the version that was running that caused the error was pluggy==0.6.0; I've performed a few permutations of the package versions when debugging the issue, so it's possible that I am also reporting the incorrect version.

The error was encountered during an automated test run that installed pytest and the various dependencies; perhaps there is an order-dependent installation issue that I had not noticed until now.

Our tox.ini segment that prepares the testing environment dependencies:

deps=
    freezegun
    pytest-mock
    pytest
    pyserial
    pytest-asyncio
    websockets
    pytest_profiling
    -rrequirements.txt

[nicoddemus]

It might be pip dependency resolver playing tricks then, because pytest 3.7 does require pluggy>=0.7:

https://github.com/pytest-dev/pytest/blob/253419316ce09b952862811db229a9e9383a332b/setup.py#L72

I suggest to try recreating the environment with -r to see if the problem goes away.

[nedbat]

This is happening to me on Travis, even after I clear the caches, but only on some of the jobs? https://travis-ci.com/nedbat/coveragepy/builds/81086869 The failures are on the jobs measuring coverage, which might reuse the environment (I'm unclear on the Travis details).

[RonnyPfannschmidt]

@nedbat looks like travis is broken

[nedbat]

The problem seems to be that tox has this requirement: pluggy<1,>=0.3.0 and pip gets confused. I've solved it by adding "pluggy>=0.7" to my own requirements file.

[RonnyPfannschmidt]

@nedbat neat find, thanks for sharing, it might need a fix in tox

[nicoddemus]

Closing as this is not really related to pytest. Thanks to all participants!

[kevindawson]

Sorry to be the bearer of bad news, but

Installing collected packages: pluggy, pytest
  Found existing installation: pluggy 0.6.0
    Uninstalling pluggy-0.6.0:
      Successfully uninstalled pluggy-0.6.0
  Found existing installation: pytest 3.6.4
    Uninstalling pytest-3.6.4:
      Successfully uninstalled pytest-3.6.4
Successfully installed pluggy-0.7.1 pytest-3.7.4
kevin@dual:~/git/pylsusb$ pytest --help
Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 7, in <module>
    from pytest import main
  File "/usr/local/lib/python3.6/dist-packages/pytest.py", line 9, in <module>
    from _pytest.config import main, UsageError, cmdline, hookspec, hookimpl
  File "/usr/local/lib/python3.6/dist-packages/_pytest/config/__init__.py", line 19, in <module>
    import _pytest.hookspec  # the extension point definitions
  File "/usr/local/lib/python3.6/dist-packages/_pytest/hookspec.py", line 27, in <module>
    @hookspec(historic=True, warn_on_impl=PYTEST_NAMESPACE)
TypeError: __call__() got an unexpected keyword argument 'warn_on_impl'

still not fixed, rolling back to pytest-3.6.4
Linux dual 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[RonnyPfannschmidt]

@kevindawson still a pip issue - pytest has correctly listed that it needs a newer pluggy version

[RonnyPfannschmidt]

@kevindawson on second thought - it seems to more of a system issue

[kevindawson]

@RonnyPfannschmidt thanks for the prompt baff ;)

• if it's a local system issue on my side
• please give me a kick :)
• do I need to stop using pip and use git clone && setup instead?

[RonnyPfannschmidt]

@kevindawson as starting point, ask pip what version of pluggy it has installed, also check the pythonpath

its possible that a system level install is shadowing the newer version (it looks like you are on a debian or derivative and did a global install)

[kevindawson]

@RonnyPfannschmidt

• thanks for the kick I will go and look n hack about

many thanks

[kevindawson]

@RonnyPfannschmidt

you were quite correct

• system level install is shadowing the newer version

again many thanks

[kevindawson]

@RonnyPfannschmidt

this snippet might be of some use

$ pipdeptree
pytest-cov==2.5.1
  - coverage [required: >=3.7.1, installed: 4.5.1]
  - pytest [required: >=2.6.0, installed: 3.7.4]
    - atomicwrites [required: >=1.0, installed: 1.2.1]
    - attrs [required: >=17.4.0, installed: 18.2.0]
    - more-itertools [required: >=4.0.0, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pluggy [required: >=0.7, installed: 0.7.1]
    - py [required: >=1.5.0, installed: 1.6.0]
    - setuptools [required: Any, installed: 40.2.0]
    - six [required: >=1.10.0, installed: 1.11.0]

alive n kicking now ;)

[RonnyPfannschmidt]

@kevindawson thanks for the update on the working package-set, please include a reference to the hicup you faced for future reference

i'm happy its alright now, but me and future victims of the problem might want to know whats wrong

[kevindawson]

@RonnyPfannschmidt ok, I trust this is what you were looking for ;)

My Hiccup

It's possible that a system level install is shadowing the newer version ? YES

I am new to Python but not to nix, so decided I would use 3.6.

As all apps are easy to install, I only need ONE copy even thought I do have numerous user accounts.

• sudo -H pip3 install --upgrade --force-reinstall pytest-cov
  
  
  
  • pytest-cov==2.5.1
  
  
  • pytest==3.7.4
  
  
  • pluggy==0.7.1
  
  

Should have been all that I needed, But

• pip[3] list |grep pluggy OR pip[3] list |grep pytest
  
  
  
  • I assumed that pip is using the same search rules as python, thus we see some pluggy==0.6.0 lying around from disrto install or deb.
  
  
  • showed remnants of the following
    
    
    
    
    
    • pytest==3.6.4
    
    
    
    • pluggy==0.6.0
    
    
    
  
  

• Looked for pytest & pluggy in /usr/local/lib/python[x.x]/dist-packages/
  
  
  
  • rm -rf to _ files dirs of pluggy & pytest
  
  

CLEAN

• sudo -H pip3 install --upgrade --force-reinstall pytest-cov

All Good

[RonnyPfannschmidt]

@kevindawson thanks for following up with the steps and the problem that happened

the issue you faced is the main reason why we strongly recommend using virtualenvs instead of the system python (since it is entirely possible that you inadvertently broke something that's part of your system by using pip on the system python using sudo)

[kevindawson]

@RonnyPfannschmidt

Thanks for the info. I was trying to avoid virtualenvs per project

• as for (since it is entirely possible that you inadvertently broke something that's part of your system by using pip on the system python using sudo)
• that is why I was only using python3.6 as ubuntu18.04 uses python2.x
• I thought it would provide separation by using pip3

Any pointers on Git & virtualenvs best practices, please

[RonnyPfannschmidt]

hmm, personally i just use virtualenvs and tox __often__
pip3 still installs into the system, personally i strongly recommend to just get used to virtualenv as part of the work-flow

[kevindawson]

@RonnyPfannschmidt

• after some RTFM
• watching Kenneth Reitz - Pipenv: The Future of Python Dependency Management - PyCon 2018
• I have decided to hack around with pipenv
  
  
  
  • no more pip plus virtualenvs all in one box;)
  
  

many thanks

[RonnyPfannschmidt]

@kevindawson fabulous, i'm happy you found a setup that works for you and shared it with us

[The-Compiler]

This still seems like an issue in some way or another? I get the same when I try doing tox -e docs with the current pytest master, with tox 3.12.1:

reading sources... [ 93%] reference

Warning, treated as error:
autodoc: failed to import function 'hookspec.pytest_assertrepr_compare' from module '_pytest'; the following exception was raised:
Traceback (most recent call last):
  File "/home/florian/proj/pytest/.tox/docs/lib/python3.7/site-packages/sphinx/ext/autodoc/importer.py", line 232, in import_module
    __import__(modname)
  File "/home/florian/proj/pytest/src/_pytest/hookspec.py", line 578, in <module>
    @hookspec(historic=True, warn_on_impl=PYTEST_LOGWARNING)
TypeError: __call__() got an unexpected keyword argument 'warn_on_impl'

Indeed pip installed pluggy==0.6.0, I wonder why though?!

edit: It also doesn't install more-itertools, but seems to do so if I upgrade the pip installed in the virtualenv (from 9.0.1 which virtualenv installs to 19.1.1)

[nicoddemus]

Weird, it works for me with tox 3.12.1 and current master... 🤔