Elixir: Make mix test stop on nth failure

[ibizaman]

Is it possible to add a flag asking mix test to stop on nth failure?

This is part of my test repairing workflow when using pytest, a python test executor.
I run it like this: pytest --maxfail=1, it stops on first error, I repair that error (or try at least), re-run the pytest command and repeat until there are errors.
I would use this only for manually repairing tests, not when running tests in a continuous integration fashion.

It would be awesome to have something similar in mix test task.

[whatyouhide]

Is there value to making the maximum number of tests allowed to fail configurable instead of just using something like "stop on the first failing test"?

[ibizaman]

I never set maxfail to something >1 so not for me at least. And I never found usage examples of using it with more than one.

[eksperimental]

I guess >1 could be useful when introducing a breaking change in a project, that requires to modifiy a hell of a lot of tests.. so you want to see/fix them by chunks of n's

[josevalim]

Why would such feature be required given we already print failures as they come? I would understand the importance of this feature if you had to wait for the whole suite to finish but you can already start acting as soon as the first failures show up.

[eksperimental]

Because tests can be long, specially if you are on an slow computer, and you may want to be doing something else (concurrency in real life :-) rather than looking at the computer screen waiting for it to fail.
So maybe do something like:
mix test --max-fail 3 && play_sound_success || play_sound_error

I'm not saying it is required feature, I'm just giving a possible case scenario

[josevalim]

I see. If someone would like to submit a pull request, we would likely accept it, but for us it is low priority. :)

[ibizaman]

@josevalim to be honest, I wanted this because I didn't know at the time about the log capture option. I wanted this because in my case I have a 7000 line test debug output (all SQL requests are logged) when running all tests and I spend a stupid amount of time finding the printed failures. But as I said, log capture solves this!

[stevegraham]

+1

Why would such feature be required given we already print failures as they come?

So a CI build can immediately stop on a failed test.