Behat: Option to re-run scenarios after failure

[ice9js]

Hi,

I think this might be a cool feature, as in some cases like mine, it's not possible to completely isolate the test environment. This sometimes results in a few cases failing while the features actually work, but someone else messed up the test.
Of course I could take the fact that an outsider can mess up a test result into consideration when writing the test, but then I wouldn't exactly be testing what I want to be testing anyway.

What do you think ?

[stof]

There is a --rerun option to run only the scenarios which have failed during the previous run. Is it what you need ?

[ice9js]

Won't that mess up my test results ?
Right now I'm using the almost finished version of the JUnit formatter... I suppose that the results from the second run would replace the results from the first one.

[everzet]

They would. But why would you rerun on CI?

[ice9js]

Well, in case you're still wondering:

On that project, the tests were actually run in production in addition to the regular CI runs, so we could make sure everything kept on running smoothly. Because of that, occasionally a user would interfere and the test would fail. It was of use to us then, to be able to repeat the particular test, so we didn't have to go through all the faulty results each time.

I'm not part of that project anymore, so I don't see any reason to go on with this issue anyway, thanks for closing it.

[jakzal]

Don't run your tests on production people :/

[ciaranmcnulty]

:+1:

[shrikeh]

Actually, running behavioural testing against production with tools such as
Ghost Inspector plugged into PagerDuty if they fail is great. Metric-based
monitoring cannot hope to have the depth of understanding of your business
value that behavioural testing allows. I don't really care if my server is
at 95% of CPU utilisation: that's a smell, sure, but what I really care
about is if my users can't transact.

Just split your tests with a tag called @production and have only those
ones run.