Flank: Outcome incorrectly set by Flank

Created on 24 Jul 2020  ยท  6Comments  ยท  Source: Flank/flank

Describe the bug

A clear and concise description of what the bug is.

[task 2020-07-23T20:29:15.152Z] โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
[task 2020-07-23T20:29:15.153Z] โ”‚ OUTCOME โ”‚      MATRIX ID       โ”‚               TEST DETAILS               โ”‚
[task 2020-07-23T20:29:15.153Z] โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
[task 2020-07-23T20:29:15.153Z] โ”‚  flaky  โ”‚ matrix-1vj2pt9wih182 โ”‚ 1 test cases failed, 106 passed, 2 flaky โ”‚
[task 2020-07-23T20:29:15.153Z] โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

FTL confirmed the matrix outcome is actually failure. Flank reported flaky.

Let's diagram the Flank outcome setting logic with PlantUML in addition to fixing this bug.

Expected behavior

A clear and concise description of what you expected to happen.

I expected the outcome to match the FTL outcome (failure).

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

Post the output of flank --version.

Flank v20.07.0

Additional context

Add any other context about the problem here.

https://firebase-community.slack.com/archives/C72V6UW8M/p1595539705001900

Bug

Most helpful comment

  • do we want to change the naming of the outcome?

for naming, let's match gcloud.

All 6 comments

The logic that flank needs to reflect:

| Case | Outcome | Exit code |
| ------------- | :-------------: | :----------: |
| at least one test failed | failed | 1 |
| no failed test but at least 1 flaky | flaky | 0 |
| no failed and no flaky | success | 0 |

gcloud output

at least one test failed

Screenshot 2020-07-28 at 09 01 41

no failed test but at least 1 flaky

Screenshot 2020-07-28 at 09 01 56

no failed and no flaky

Screenshot 2020-07-28 at 09 02 09

flank output

at least one test failed

Screenshot 2020-07-28 at 09 33 40
Screenshot 2020-07-28 at 09 18 09

no failed test but at least one flaky EDIT: below outcome is Corner Case

Screenshot 2020-07-28 at 09 13 15

no failed and no flaky

Screenshot 2020-07-28 at 09 23 23

What needs to be fixed:

  • the failed test always means failed outcome, regardless of others being flaky (currently flaky tests obscure failed ones)
  • flaky tests mean flaky outcome but with success exit code (currently exit code is 0 but status is success)
  • optional: do we want to change the naming of the outcome? (ex. failure -> failed) cc @jan-gogo
  • do we want to change the naming of the outcome?

for naming, let's match gcloud.

no failed test but at least one flaky EDIT: below outcome is Corner Case
Screenshot 2020-07-28 at 09 13 15

Above is a corner case that needs to be resolved in a separate issue (#918). It requires to check flank API calls with gcloud's.
Originally described bug to be resolved in #917

@rpappalax
Fix in review #917. Please let me know immediately if you will encounter a similar issue in the future. Please, keep in mind there is another issue found (thanks to your case ๐Ÿ˜„ ) we still need to investigate #918

Was this page helpful?
0 / 5 - 0 ratings