Flank: Improve MatrixResultsReport to handle internal errors

[bootstraponline]

MatrixResultsReport is confusingly reporting 100% success when errors are encountered that don't produce junit.xml files.

22:38:36   7m 42s matrix-dsyuq3rdz80va Error: Internal error during validation
22:38:36   7m 42s matrix-dsyuq3rdz80va ERROR

22:39:51 MatrixResultsReport
22:39:51   200 / 200 (100.00%)

In this case the exit code is correctly specified as 2, however the results report should include the failure. I think the bug is failures without XML are silently ignored.

[bootstraponline]

PollMatrices
  0m  0s matrix-21w98hkff16b2 VALIDATING
  0m 15s matrix-21w98hkff16b2 Error: Internal error during validation
  0m 15s matrix-21w98hkff16b2 ERROR

FetchArtifacts

CostReport
  No cost. 0m

MatrixResultsReport
  1 / 1 (100.00%)

[runningcode]

This seems quite dangerous for users of flank which rely solely on the junit output. This is a typical scenario in a CI setup.

[bootstraponline]

This seems quite dangerous for users of flank which rely solely on the junit output. This is a typical scenario in a CI setup.

what specifically seems dangerous? The proposed solution? Or the bug?

The idea is to get this information in the junit xml so that failures are reported as failures -- both in the exit code and junit xml.

JUnit XML processing is being moved into the FTL server. Once that happens, Flank will be able to fetch this file from Gcloud storage instead of trying to recreate it manually.

[runningcode]

The bug _could_ be dangerous if someone creates a CI task that only looks at the junit output to determine success.

Moving the junit xml processing to the server sounds nice. Is there a ticket to track that work?

[bootstraponline]

I agree re: bug being dangerous.

re: ticket, yes but it's only accessible to Google employees.

Most of the server improvements planned are NDA so I can't list them on GitHub. The tl;dr is that firebase is adding support for test sharding and that will fix this specific issue and many others.

[bootstraponline]

It turns out this bug is also impacting me at work so I expect to have some cycles soon to resolve.

[bootstraponline]

I've spent some cycles investigating this. The root issue is Firebase Test Lab will not generate JUnit XML for many reasons. As a result, JUnit XML alone is not sufficient to determine if a run succeeded or not.

I've updated the Flank exit code logic to print detailed information on matrix failures. This will make it obvious when something weird has happened in the matrix. The fixes have landed in Flank v8.0.1.

The recommendation is to always check the exit code.