Pytest: Can I collect `Items` without running `pytest.main()`?

[Zac-HD]

I'm currently working on a fuzzing mode for Hypothesis, and would really like to have a smooth integration with existing test suites. The basic idea is for fuzz <user args here> to pick up each test that would be collected by pytest --collect-only -m hypothesis <user args here> and then do some longer-running magic with them.

Hacking around via the CLI to get test ids kinda works, but is (a) aesthetically displeasing, and (b) incorrect with respect to plugins - including Hypothesis' own pytest plugin - which modify collected items. My best attempt so far, which doesn't work at all, was:

config = _prepareconfig(args=None, plugins=None)
try:
    session = Session.from_config(config)
    config._do_configure()
    config.hook.pytest_collection(session=session)
    items = session.perform_collect()  # sadly always empty
finally:
    config._ensure_unconfigure()

(asking here rather than stackoverflow because it's a rather niche question about internals)

[bluetech]

• Missing pytest_sessionstart/pytest_sessionfinish
• Calling collection twice (the pytest_collection hook just calls perform_collect)
• Relying on sys.argv

So:

import pytest
from _pytest.config import _prepareconfig

config = _prepareconfig(args=[], plugins=None)
try:
    session = pytest.Session.from_config(config)
    config._do_configure()
    config.hook.pytest_sessionstart(session=session)
    config.hook.pytest_collection(session=session)
finally:
    config.hook.pytest_sessionfinish(session=session, exitstatus=session.exitstatus)
    config._ensure_unconfigure()

print(session.items)

If you want a bit more isolation you can also try replacing _prepareconfig with get_config.

[Zac-HD]

You're a legend, that works perfectly. Thanks!

[nicoddemus]

@Zac-HD any reason why not run pytest.main()? I think using a custom plugin and calling pytest.main is more future proof:

class ItemsCollector:

    def pytest_collection_finish(self, session):
        self.node_ids = [x.nodeid for x in session.items]

collector = ItemsCollector()
pytest.main(["", "-pno:terminal", "--collect-only"], plugins=collector)

But it is a bummer that this is not simple to do programatically without using private APIs.

[nicoddemus]

This would be nice to have:

import pytest

config = pytest.Config.from_args(args=[], plugins=None)
try:
    session = pytest.Session.from_config(config)
    config.run_configure()
    config.hook.pytest_sessionstart(session=session)
    config.hook.pytest_collection(session=session)
finally:
    config.hook.pytest_sessionfinish(session=session, exitstatus=session.exitstatus)
    config.run_unconfigure()

print(session.items)

[Zac-HD]

Using pytest.main() is probably a better solution! Thanks @nicoddemus :smile:

Unfortunately -pno:terminal errors out because of the unrecognised --tb=short in addopts, which seems to be leaking back out to my CLI wrapper. Eh, I'd need to swallow stdout with the internals approach anyway...