Flank: If one test fails due to an app crash, all shards report failure

[shawnshaligram]

Describe the bug
Say if a test suite has 2 tests which run on 2 shards and if 1 test fails with an app crash the other test shard automatically fails as well even though it should pass

To Reproduce

1. Upgrade from Flank v4 to v21
2. Create two tests in Espresso and set it up to run on 2 shards
3. Ensure one of the tests fail with an application crash
4. Observe the other test shard also fails automatically

Expected behavior
Entire Test Suite shouldn't be affected if 1 test fails with an application crash

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

Yes

Post the output of flank --version.

v21

Additional context

Add any other context about the problem here.

Works fine with v4 but not with v21

[bootstraponline]

I think this actually might be a bug in the FTL server. Please post in #test-lab on Slack.

For background context, Flank v4 always used one matrix per shard. In the modern version of flank, we switched to using FTL's server side sharding. That means the outcomes and results are now controlled by the FTL implementation of sharding. Sometimes they have weird bugs related to outcomes.

[pawelpasterz]

And small addition from my side. Currently, all shards are run within one matrix (for single test apk pair). So if a single test in one shard fails -- all matrix is considered as failed. That is different behavior compared to v4 when each shard was a separate matrix, so they (matrices) were independent of each other (as @bootstraponline wrote above).

Flank does not create/assume results, it only fetches them from FTL result API.

I know flank's documentation is not in the best shape, but we are working on it :)

[jan-gogo]

@shawnshaligram Tell us if you have any additional questions or thoughts about this issue. If not, we will close it after few days.