Bazel: Sort test result by status

[meteorcloudy]

Currently Bazel doesn't group the test result by their status.

For example, the failed tests are scattered among the passed tests. It's hard to figure out what tests are failing.

//tools/android:instrumentation_test_check_test                          PASSED in 0.8s
//tools/android:junction_test                                            PASSED in 5.4s
//tools/android:resource_extractor_test                                  FAILED in 0.7s
//tools/android:stubify_manifest_test                                    PASSED in 3.1s
//tools/aquery_differ:aquery_differ_test                                 PASSED in 1.9s
//tools/bash/runfiles:runfiles_test                                      PASSED in 2.0s
//tools/build_defs/pkg:path_test                                         FAILED in 0.9s
//tools/java/runfiles/testing:RunfilesTest                               PASSED in 1.4s
//tools/java/runfiles/testing:UtilTest                                   FAILED in 2.1s
//tools/jdk:proguard_whitelister_test                                    PASSED in 0.7s

Instead we should sort the test results by status:

//tools/android:instrumentation_test_check_test                          PASSED in 0.8s
//tools/android:junction_test                                            PASSED in 5.4s
//tools/android:stubify_manifest_test                                    PASSED in 3.1s
//tools/aquery_differ:aquery_differ_test                                 PASSED in 1.9s
//tools/bash/runfiles:runfiles_test                                      PASSED in 2.0s
//tools/java/runfiles/testing:RunfilesTest                               PASSED in 1.4s
//tools/jdk:proguard_whitelister_test                                    PASSED in 0.7s
//tools/android:resource_extractor_test                                  FAILED in 0.7s
//tools/java/runfiles/testing:UtilTest                                   FAILED in 2.1s
//tools/build_defs/pkg:path_test                                         FAILED in 0.9s

[meteorcloudy]

FYI @gunan @philwo

[meteorcloudy]

/cc @aehlig Do you have any idea on this?

[mzeren-vmw]

Or just s/PASSED/passed/ ?

[gunan]

To just add one more thing.
When we run with -k, sometimes we get "NO STATUS".
Having PASSED tests all first, then having anything else would be very helpful.

[Monnoroch]

Feels like too small a feature request to create an issue, but there also should be a test result output mode that doesn't show cached tests. There's one that shows all (summary) and failed only (terse), but nothing in between to show all non-cached tests (both passed and failed).

[meteorcloudy]

@dslomov Can you help triage this issue?

[meteorcloudy]

Looks like Bazel already sort the test summary by a certain order:
https://github.com/bazelbuild/bazel/blob/29981f324e123fd1ff1a4aa3ac65beba4c35f725/src/main/java/com/google/devtools/build/lib/runtime/TestSummary.java#L470

I checked with the latest Bazel release, it does work.

I'll try to add another --test_summary choice to skip cached tests.

[philwo]

This does not seem to reliable work with Bazel 0.27.0. The results are only "mostly" sorted - there's a long list of PASSED tests, then a mix of FAILED and PASSED tests at the bottom.

It's interesting that all of the tests that are in the mixed list seem to be sharded tests, as they all have a line like: "Stats over 3 runs: max = 0.4s, min = 0.3s, avg = 0.3s, dev = 0.1s".

Example: