Pytest: References to tests in === FAILURES === report should be suitable as input to select tests.

[TauPan]

Hi!

I'd like to be able to quickly retest a certain failing test after running the whole suite. --last-failed provides a similar functionality, but not the same if multiple (or many) tests fail.

Whenever I have failures, the "FAILED/PASSED" lines in the session report lists them in a way that is suitable for input for selecting them for a subsequent run (as outlined in https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests).

project/package/tests/test_or.py::TestSomeClass::test_some_behaviour FAILED  [  1%]

However when I'm investigating a failure I'm often looking at the backtrace and variables in the === FAILURES === section of pytest's output:

============================================== FAILURES ==============================================
___________________________ TestSomeClass.test_some_behaviour ____________________________

self = <project.package.tests.test_or.TestSomeClass testMethod=test_some_behaviour>

    def test_some_behaviour(self):
[...]
self       = <project.package.tests.test_or.TestSomeClass testMethod=test_some_behaviour>

project/package/tests/test_or.py:158: AssertionError

And to get the node id of the test in question, I either have to scroll up (quite far, if my testsuite contains over 500 tests) or piece the information together from the output.

I can use the method name test_some_behaviour together with -k but this might execute multiple tests with the same name.

In the concrete example it would be convenient for me if project/package/tests/test_or.py::TestSomeClass::test_some_behaviour would appear somewhere in the FAILURES output.

Thanks!

[pytestbot]

GitMate.io thinks possibly related issues are https://github.com/pytest-dev/pytest/issues/50 (Select tests according to their mark), https://github.com/pytest-dev/pytest/issues/2579 (Test report text issue), https://github.com/pytest-dev/pytest/issues/2156 (py.test reports IndexError/KeyError as an Failure instead of Error), https://github.com/pytest-dev/pytest/issues/2549 (Report "pytestmark = skip" only once), and https://github.com/pytest-dev/pytest/issues/157 (--verbose should report why a test case was skipped).

[TauPan]

I just checked the issues @pytestbot referenced and I don't think any of them is (closely) related.

[nicoddemus]

Hi @TauPan,

The -ra flag can be used to produce a summary at the end of the test run that can be used to copy/paste test node ids:

====================================================== FAILURES =======================================================
________________________________________________________ test _________________________________________________________

    def test():
>       assert 0
E       assert 0

test_foo.py:2: AssertionError
=============================================== short test summary info ===============================================
FAIL test_foo.py::test
============================================== 1 failed in 0.27 seconds ===============================================

You think that's enough?

[TauPan]

Yes. Thanks. In fact -rf would be enough for me.

I failed to find this in any documentation, although I now see that it's mentioned in the --help output.

Is this mentioned anywhere else? https://docs.pytest.org/en/latest/contents.html doesn't seem to have a section describing the reporting flags in more detail.

[nicoddemus]

Hi,

It is shown in some places:

• https://docs.pytest.org/en/latest/skipping.html?highlight=-rx
• https://docs.pytest.org/en/latest/example/simple.html?highlight=-rx#incremental-testing-test-steps

But you are correct that it should be displayed more prominently somewhere.

[nicoddemus]

Opened #3566 to track adding a blurb about -r to the docs. Thanks for the report @TauPan!