Pytest: OutcomeExceptions should be handled during collection (not causing INTERNALERROR)

Created on 22 Nov 2019 · 9Comments · Source: pytest-dev/pytest

When trying to use this from pytest-cov's source:

% p -m 'not slow'
===================================== test session starts ======================================
platform linux -- Python 3.8.0, pytest-5.3.0, py-1.8.0, pluggy-0.13.1
rootdir: …/Vcs/pytest-cov, inifile: setup.cfg, testpaths: tests
plugins: forked-1.1.3, xdist-1.30.0, cov-2.8.1
collected 124 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/main.py", line 196, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/main.py", line 239, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/main.py", line 249, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/main.py", line 450, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 182, in _multicall
INTERNALERROR>     next(gen)  # first yield
INTERNALERROR>   File "…/Vcs/pytest-cov/tests/conftest.py", line 34, in pytest_collection_modifyitems
INTERNALERROR>     item.add_marker(pytest.mark.slow)
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/mark/structures.py", line 313, in __getattr__
INTERNALERROR>     fail(
INTERNALERROR>   File "…/Vcs/pytest-cov/.venv/lib/python3.8/site-packages/_pytest/outcomes.py", line 128, in fail
INTERNALERROR>     raise Failed(msg=msg, pytrace=pytrace)
INTERNALERROR> Failed: 'slow' not found in `markers` configuration option

bug

Source

blueyed

All 9 comments

INTERNALERROR>   File "…/Vcs/pytest-cov/tests/conftest.py", line 34, in pytest_collection_modifyitems
INTERNALERROR>     item.add_marker(pytest.mark.slow)

pytest-cov should register it's slow marker, but I agree that we should have a better UX when this hasn't happened (including suggesting how to fix it from the error message).

Zac-HD on 25 Nov 2019

@Zac-HD
There's no "slow" marker in pytest-cov.
This issue is about the UX in case of the error/warning in that case.

blueyed on 25 Nov 2019

There's no registered slow marker, but pytest-cov/tests/conftest.py tries to add it anyway.

It's definitely a UX problem for pytest, but it's reporting a real problem for pytest-cov.

Zac-HD on 25 Nov 2019

👍1

Ok.
Just to be clear: it was me adding item.add_marker(pytest.mark.slow) to its conftest (locally), and this issue is about improving the UX / error in that case (for when it is not registered).
IIRC this error is better in general, so is likely about fail.Exception not being handled here properly.

blueyed on 25 Nov 2019

👍1

originally this was an attribute error, what happens in that particular case, if we add that back?

RonnyPfannschmidt on 25 Nov 2019

@RonnyPfannschmidt
It was changed in 5436e4299.
(btw: in https://github.com/pytest-dev/pytest/pull/6232 I've changed it to not use pytrace=False already, because I found the traceback/location to be useful.)

The issue is that "fail.Exception" is not handled here. This also applies to AttributeErrors (just tried it directly from pytest_collection_modifyitems).

It might make more sense for non-outcome-errors (altough not really "internal"), but outcome-exceptions should be handled probably.

blueyed on 26 Nov 2019

@blueyed i think we should change all those fails to actual UsageError

RonnyPfannschmidt on 26 Nov 2019

@RonnyPfannschmidt
sounds good in general, but is unclear to me: do you mean all failures (pytest.fail and/or Outcome exceptions in general) during collection then?

Likely you've meant that the initial reported one ("'X' not found in markers configuration option") should become an UsageError by itself, which makes sense.
This leaves us with what should happen with pytest.fail etc.

blueyed on 26 Nov 2019

outcome exceptions should be handled where we have actual outcomes, else they should be a internalerror

the modify-items hook is something where we do not expect outcomes

RonnyPfannschmidt on 26 Nov 2019

Was this page helpful?

0 / 5 - 0 ratings