Pytest: Breaking change 5.3 -> 5.4

[ofek]

From: https://github.com/Teemu/pytest-sugar/issues/194

pytest-sugar 0.9.3 crashes when running pytest with the -k flag. It works if I downgrade pytest back to 5.3. (note: for me it crashes without -k using pytest-cov)

Command used to run pytest: pytest -k test

Test file

def test_example():
    assert False

plugins: sugar-0.9.3, apitestkit-3.18.0, bbm-testkit-3.9.0, responses-0.4.0
collecting ... INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/main.py", line 191, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/main.py", line 246, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/main.py", line 257, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/main.py", line 455, in perform_collect
INTERNALERROR>     session=self, config=self.config, items=items
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/mark/__init__.py", line 150, in pytest_collection_modifyitems
INTERNALERROR>     deselect_by_keyword(items, config)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/mark/__init__.py", line 127, in deselect_by_keyword
INTERNALERROR>     config.hook.pytest_deselected(items=deselected)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/terminal.py", line 431, in pytest_deselected
INTERNALERROR>     self._add_stats("deselected", items)
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/terminal.py", line 402, in _add_stats
INTERNALERROR>     self._set_main_color()
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/terminal.py", line 1090, in _set_main_color
INTERNALERROR>     self._main_color = self._determine_main_color(bool(unknown_types))
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/terminal.py", line 1077, in _determine_main_color
INTERNALERROR>     elif "passed" in stats or not self._is_last_item:
INTERNALERROR>   File "/Users/craig/src/boughtbymany/s-subscription/.tox/app/lib/python3.6/site-packages/_pytest/terminal.py", line 496, in _is_last_item
INTERNALERROR>     return len(self._progress_nodeids_reported) == self._session.testscollected
INTERNALERROR> AttributeError: 'NoneType' object has no attribute 'testscollected'

@asottile debugging: https://github.com/Teemu/pytest-sugar/issues/194#issuecomment-636432701

from a git bisect, this is the first bad commit:

$ git bisect bad
57512aa997320cc8769666bb9de8a14ea35dc180 is the first bad commit
commit 57512aa997320cc8769666bb9de8a14ea35dc180
Author: Daniel Hahler <[email protected]>
Date:   Mon Jan 6 15:09:09 2020 +0100

    _get_main_color: no yellow ("boring") for non-last item

    - refactor _get_main_color/build_summary_stats_line
    - factor out property _is_last_item; test_summary_stats: tr._is_last_item
    - _write_progress_information_filling_space: remove color arg
    - use setter for stats, handling main color
    - _get_main_color: skip cache for last item
    - Handle random order in test for py35.

 changelog/6409.bugfix.rst |   1 +
 src/_pytest/terminal.py   | 135 +++++++++++++++++++++++++---------------------
 testing/test_pdb.py       |   2 +-
 testing/test_terminal.py  |  62 +++++++++++++++++----
 4 files changed, 127 insertions(+), 73 deletions(-)
 create mode 100644 changelog/6409.bugfix.rst

I had to bisect manually since I could only reproduce it in the foreground -- the example I was using to reproduce it "consistently" was:

bash -c 'cd .. && rm -rf .pytest_cache/ && python -m pytest --lf -k test t.py;python -m pytest --lf -k test t.py'

[nicoddemus]

PR open: https://github.com/Teemu/pytest-sugar/pull/197

[ofek]

Awesome!

[bluetech]

Closing as this seems settled.