Add a retry mode to flank that retries only failed flaky tests. FTL retry will retry entire shards, including successful tests.
This feature is not currently on the FTL roadmap.
bootstraponline
All 6 comments
Would love to have this feature!
narenkmanoharan
on 26 May 2020
Oh it would be really very useful!
kbazleva
on 8 Jun 2020
hi guys, do you have plans to implement this feature?
It would be nice to re-run only failed tests instead of entire shard.
mariuszmachniakzendesk
on 25 Aug 2020
we do plan to support this feature! there are other tasks (bug fixes, stability) that have taken priority over feature work.
bootstraponline
on 25 Aug 2020
One thing that would be worth to keep in mind, since FTL does not support such logic it would require from flank creating new matrix (matrices) for failed tests. It's not something extremely difficult but there is another overhead. FTL will set up
new device(s) and that takes some time. I think it's not a blocker but let's don't forget about it.
pawelpasterz
on 21 Sep 2020
please also make sure that the total test count tally is not impacted by the retry. If we have 100 tests that execute, and 2 are flaky and get re-ran twice, we should still report 100 tests. I saw this issue in FTL where they actually incremented the total test count, which threw off our jenkins test analyzer.
Joseph-Labonte
on 2 Oct 2020
Related issues
shawnshaligram
路
3Comments
adamfilipow92
路
7Comments
jakob-grabner
路
7Comments
leinardi
路
4Comments
miguelslemos
路
3Comments
Most helpful comment
please also make sure that the total test count tally is not impacted by the retry. If we have 100 tests that execute, and 2 are flaky and get re-ran twice, we should still report 100 tests. I saw this issue in FTL where they actually incremented the total test count, which threw off our jenkins test analyzer.