Pytest: Run Tests For N Number of Hours

[ImXron]

Hi!

Is there a way to run / repeat tests for N number of hours?

I know it's possible to run tests N number of times with xdist and pytest-repeat.

My use case is time though. I could do some math and divide the desired time by how long each test run takes (to calculate how many times it should repeat), but that's very hacky and would not be true across all computers and networks.

Thanks!

[RonnyPfannschmidt]

Please outline the use case that makes this necessary

[ImXron]

Hi @RonnyPfannschmidt, of course.

Use case:
A device needs to be stress tested for a user defined amount of hours (e.g. 8 hours) using an existing pytest test suite. The test suite would repeat for 8 hours straight.

This is a common use case for testing hardware.

Let me know if you require additional info.

Thanks!

[blueyed]

Use timeout (from GNU coreutils)?

[ImXron]

@blueyed Thanks for the suggestion!

Unfortunately, there's no easy way for our infrastructure to support running external commands like this.

Is there a hook (or multiple hooks) I can implement that would allow me throw a test run in a loop (while still allowing pytest to run session setups and tear downs) ?

Thanks!

[blueyed]

Hmm, I've assumed you could just wrap the pytest command with it?
I guess you could use hooks around each test run, and check if the duration is over.

[ImXron]

Yeah wrapping the pytest command would work in normal cases. It's just that my team's infrastructure doesn't allow us to do that. We put some constraints so devs can't do anything naughty, so the entrypoint must always be pytest <options>.

The flow I'm looking for would be something like:

1. Start test session.
2. Run tests.
3. Execute tear down fixtures (all scopes)
4. Check if duration is < X hours.
5. Repeat.

You think I would need to implement an initialization hook or a test running hook (or both)?

Also, happy belated World Vegan Day @blueyed 🌱.

[blueyed]

Maybe you can use any session wrapping hooks to run pytest itself via subprocess, wrapped with timeout then? (or use the timeout functionality in subprocess)

But I'd rather question the infrastructure then - since you can execute any code from Python, why wouldn't you be allowed to wrap it in timeout itself?
FWIW you would need extra handling with timeout then with regard to exit code likely, i.e. I would wrap it in a shell script then likely.

As for your flow: do you want to run the whole session always?
You could also have the looping / stress test in a single test maybe even, which only stops when the time is over - or do you expect different kind of errors, that should be reported, but not stop the run before 8h?

[ImXron]

Calling subprocess in a hook might work, good point.

To clarify, the infrastructure calls pytest specifically (with some pytest options), not just python. Either way, it might be worth making changes to the infrastructure in the long run.

For the flow, yes, trying to loop the session. There are tear down fixtures that handle a bunch of stuff I need (process results, etc) with each loop. The session would continue upon test failures. I could stuff all the tests into a single test yes, but if the tear down fixtures don't execute, it wouldn't work for my use case. So ideally, each loop would look like a typical pytest run.

I'm trying to keep maintenance as low as possible, so using the test suites as is, would be amazing.

[nicoddemus]

Hi @ImXron,

Just to answer your original question, in case you want to go with that direction, you can reimplement the pytest_runtestloop:

https://github.com/pytest-dev/pytest/blob/35800a2f73c4233285a4a8f403ca94bc9c6e998e/src/_pytest/main.py#L249-L263

In your case, you would keep looping over the tests until the desired duration ends.

[ImXron]

@nicoddemus Awesome! I'll experiment with this hook.

Thanks!

Also, I can't tell you how many times I've searched online for pytest questions, and you're the one with the answer. Thanks for your work on this project!

[ImXron]

Seems to be working great!

I'll definitely publish a plugin for this.

Thanks again guys!

[ImXron]

Took a stab at a plugin for this. Put up a first draft PR for feedback purposes. Any feedback would be much appreciated. Nit pickers are welcome as well!
Pytest-stress

Thanks!