Pytest: Feature idea: Find root failing test by examining module imports

Created on 11 Mar 2019 · 3Comments · Source: pytest-dev/pytest

Currently if you're testing a large, layered project with many tests and you break something then you will get many failed tests, and they will be displayed in order pytest finds them. However, it's likely that some of these failures are closer to the root of the problem with your code, while others are merely downstream consequences.

Might it be possible to use the internal imports in a project to suggest which test failures are closer to the root of the problem? Perhaps by creating a tree of internal imports and highlighting the test failure which calls the fewest?

Does something like this already exist? Is this even possible? Does python's importing logic prohibit this somehow? Is this unnecessary because there's a better way to approach this kind of problem in practice?

question

Source

TomNicholas

All 3 comments

Hmm, I don't know of anything currently like this but it sounds great! I'd guess that imports are a bit coarser than idea here, but there's a better option: Coverage can be tracked per-test-function.

So for an initial implementation, I'd try querying the SQLite coverage database for the lines with the highest ratio of (failing tests covering this line)/(all tests covering this line). That would - roughly - tell you where problems might be. Then maybe select a few failing tests with small but diverse coverage?