Describe the bug
A clear and concise description of what the bug is.
[task 2020-07-23T20:29:15.152Z] โโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
[task 2020-07-23T20:29:15.153Z] โ OUTCOME โ MATRIX ID โ TEST DETAILS โ
[task 2020-07-23T20:29:15.153Z] โโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
[task 2020-07-23T20:29:15.153Z] โ flaky โ matrix-1vj2pt9wih182 โ 1 test cases failed, 106 passed, 2 flaky โ
[task 2020-07-23T20:29:15.153Z] โโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
FTL confirmed the matrix outcome is actually failure. Flank reported flaky.
Let's diagram the Flank outcome setting logic with PlantUML in addition to fixing this bug.
Expected behavior
A clear and concise description of what you expected to happen.
I expected the outcome to match the FTL outcome (failure).
Details (please complete the following information):
Have you tested on the latest Flank snapshot?
Post the output of
flank --version.
Flank v20.07.0
Additional context
Add any other context about the problem here.
https://firebase-community.slack.com/archives/C72V6UW8M/p1595539705001900
The logic that flank needs to reflect:
| Case | Outcome | Exit code |
| ------------- | :-------------: | :----------: |
| at least one test failed | failed | 1 |
| no failed test but at least 1 flaky | flaky | 0 |
| no failed and no flaky | success | 0 |
gcloud output


flank output



What needs to be fixed:
0 but status is success)failure -> failed) cc @jan-gogo
- do we want to change the naming of the outcome?
for naming, let's match gcloud.
no failed test but at least one flaky EDIT: below outcome is Corner Case
Above is a corner case that needs to be resolved in a separate issue (#918). It requires to check flank API calls with gcloud's.
Originally described bug to be resolved in #917
@rpappalax
Fix in review #917. Please let me know immediately if you will encounter a similar issue in the future. Please, keep in mind there is another issue found (thanks to your case ๐ ) we still need to investigate #918
Most helpful comment
for naming, let's match gcloud.