Flank: Restore repeatTests in Flank

[bootstraponline]

rpappalax

the use case is determining if a new test is going to fail intermittently. we often run a new test locally and it works fine, but then once merged, it begins failing on Firebase. We usually re-try it multiple times manually by retriggering, hoping to flush out any intermittents that way, but it is time-consuming. It would be faster if we could do multiple attempts all at a go

repeatTests

The number of times to repeat each test. Uses Flanks default value when not specified.

[rpappalax]

Would it be possible (instead of having these run back-to-back), to have an asynchronous way of doing this? One of the great features of flank is it's ability to run multiple (albeit different) tests in parallel to save time. The same applies to this situation where we want to do multiple test runs, but just for one single test. If we could run them across multiple shards that would save an enormous amount of time for soak-testing a new test to ensure that it isn't vulnerable to intermittent fails

[bootstraponline]

agreed, we'll make sure repeat tests is async and shardable.

[rpappalax]

Thanks, @bootstraponline do you by chance have this feature work scheduled? Having this in place would help us a great deal debugging/reproducing intermittent failures

[bootstraponline]

I have the team entirely focused on these two blocking issues:
https://github.com/Flank/flank/issues/818
https://github.com/Flank/flank/issues/809

after these are resolved, we'll pick up feature work and this will be high priority.

[bootstraponline]

I think num-test-runs does what you want, we renamed this feature recently. If that doesn't do what you want, let me know and I'll reopen.

[rpappalax]

I think num-test-runs does what you want, we renamed this feature recently. If that doesn't do what you want, let me know and I'll reopen.

If I'm not mistaken, num-test-runs is used primarily to make flaky tests more usable as-s. So, if I set that value to 10, it would keep trying one after another, but only if there's a fail. We're trying to determine frequency of intermittent fails. We have some that are elusive and some that happen quite often, but for new tests we'd like to see if a test is flaky ASAP so we can stabilize it before it goes into production. Hence, we'd like to run a batch of tests in parallel.

So, for example, if we could run 30 tests in parallel, did it fail 0/30, 1/30 times or 15/30 times? It's hard to get this kind of data without pushing a new test into production and crossing our fingers. Usually then, it's our devs that quietly encounter issues and perform re-tries on their own.

I hope this clarifies what we're trying to do

[bootstraponline]

There's num-flaky-test-attempts and num-test-runs 🙂

num-test-runs does what you want

[rpappalax]

@bootstraponline I never thanked you for this. It's helped immensely 🙇‍♂️