Flank: JUnitReport.xml only includes 50 tests out of 73

[dmytrodanylyk]

👋 Thank you for this amazing tool.

Describe the bug
There are 2 issues that are probably related:

1. When I run tests via flank with max-test-shards: "5" for some reason it uses only 1 shard, I can't find the reason why.
2. The JUnitReport.xml which is created only contains 50 tests, while matrix_0/Nexus5-23-en-portrait/test_result_1.xml has 73 tests.

Is there any hard-coded value of the number of tests JUnitReport.xml can have?

To Reproduce

Steps to reproduce the behaviour:

Run tests via flank with following configuration:

gcloud:
  app: '/Users/ddanylyk/.../build/testlab/dummy.apk'
  test: '/Users/ddanylyk/Work/.../debug/android-debug-androidTest.apk'
  use-orchestrator: false
  record-video: true
  auto-google-login: false
  environment-variables:
    coverage: false
    clearPackageData: false
    coverageFile: /sdcard/coverage.ec
  device:
    - model: Nexus5
      version: 23
      locale: en
      orientation: portrait
  test-targets:
     - 'notAnnotation QuarantineTest'
flank:
  max-test-shards: '5'
  full-junit-result: true
  shard-time: '120'
  num-repeat-tests: '1'
  disable-sharding: 'false'
  smart-flank-gcs-path: gs://...testlabRunAndroidTest.xml

Expected behaviour

1. Flank should use more than 1 shard if max-test-shards: '5'.
2. JUnitReport.xml should contain all tests, not just 50.

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

No

Post the output of flank --version.

v20.09.3 (tried v21.02.0 same issue)

Additional context

Exanple of matrix_0/Nexus5-23-en-portrait/test_result_1.xml file:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="" tests="73" failures="0" errors="0" skipped="0" time="25.17" timestamp="2021-02-26T03:07:15" hostname="localhost">
  <properties />
  <testcase name="test1" classname="...CustomTagHandlerTest" time="0.306" />
  <testcase name="test2" classname="...CustomTagHandlerTest" time="0.306" />
...71 more entries
</testsuite>

Example of JUnitReport.xml file:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuites>
  <testsuite name="Nexus5-23-en-portrait" tests="50" failures="0" flakes="0" errors="0" skipped="0" time="25.170" timestamp="2021-02-26T03:06:50" hostname="localhost">
    <testcase name="test1" classname="...CustomTagHandlerTest" time="0.489">
      <webLink>https://console.firebase.google.com/project/.../testcases/1</webLink>
    </testcase>
...49 more entries
</testsuites>

[pawelpasterz]

Hey 👋
Thanks for reporting. We will take a look shortly.

Is there any hard-coded value of the number of tests JUnitReport.xml can have?

There is not, JUnitReport should contain all tests

There should be android_shards.json created by the flank (it is done automatically since v20.12.0, with an earlier version you need to add --dump-shards). This file contains shards created by flank (as JSON representation), could you verify if there is something missing or wrong? You can send you obfuscated JSON (add --obfuscate)

Do you have parameterized tests?

Sorry for a bunch of questions, it will help us to track the problem :)

[pawelpasterz]

Ah, I think I've found something...
@dmytrodanylyk could you try with full-junit-result: false ?

[dmytrodanylyk]

@pawelpasterz the full-junit-result: true is something I tried to see if .xml generated by this flag would contain all tests, but unfortunately it did not. My original YAML flank file didn't have this flag.

I am trying to update to v21.02.0 and check the android_shards.json, I will post results shortly.

Thank you for jumping on this issue so quickly.

[pawelpasterz]

NP :)

full-junit-result is used mainly together with num-flaky-tests-attempts: [number]. If set with true Flank will produce FullJUnitTestResult with all runs for specific test case (meaning, it includes failed attempts for flaky tests)

[dmytrodanylyk]

Ok, it may take more time than I expected to run flank with --dump-shards because we are using some custom gradle plugin to tun flank in Android.

I noticed a strange coincidence that the smart flank cache hit is 50 / 73, and that's the number of tests that I see in JUnitReport. Can this be an issue 🤔 ?

RunTests

 Smart Flank cache hit: 68% (50 / 73)
  Shard times: 2790s

  Uploading dummy.apk .
  Uploading androidTest.apk .
  73 tests / 1 shard

[pawelpasterz]

That's definitely valuable info for us. 50 means flank found 50 tests in XML provided in smart flank path. Meaning flank has acquired times for 50 tests and it will be used in shard calculation. For the rest default value will be used (120s)

Do you have parameterized tests?

[dmytrodanylyk]

Do you have parameterized tests?
Nope

[dmytrodanylyk]

The android_shards.json looks perfectly normal, there are 73 tests (as expected).

{
  "matrix-0": {
    "app": ".../build/testlab/dummy.apk",
    "test": ".../debug-androidTest.apk",
    "shards": {
      "shard-0": [
        "class a.a.a.a.a.a.A#a",
        "class a.a.a.a.a.a.A#b",
        "class a.a.a.a.a.a.A#c",
        "class a.a.a.a.a.a.A#d",
        "class a.a.a.a.a.a.A#e",
        "class a.a.a.a.a.a.A#f",
        "class a.a.a.a.a.a.A#g",
        "class a.a.a.a.a.a.A#h",
        "class a.a.a.a.a.a.A#i",
        "class a.a.a.a.a.a.A#j",
        "class a.a.a.a.a.a.A#k",
        "class a.a.a.a.a.a.A#l",
        "class a.a.a.a.a.a.B#a",
        "class a.a.a.a.a.a.B#b",
        "class a.a.a.a.a.a.B#c",
        "class a.a.a.a.a.a.B#d",
        "class a.a.a.a.a.a.B#e",
        "class a.a.a.a.a.a.B#f",
        "class a.a.a.a.a.a.B#g",
        "class a.a.a.a.a.a.B#h",
        "class a.a.a.a.a.a.B#i",
        "class a.a.a.a.a.a.B#j",
        "class a.a.a.a.a.a.B#k",
        "class a.a.a.b.a.a.A#a",
        "class a.a.a.b.a.a.A#b",
        "class a.a.a.a.a.a.C#a",
        "class a.a.a.a.a.b.A#a",
        "class a.a.a.a.a.b.A#b",
        "class a.a.a.b.a.a.A#c",
        "class a.a.a.b.a.a.A#d",
        "class a.a.a.b.a.a.A#e",
        "class a.a.a.b.a.a.A#f",
        "class a.a.a.b.a.a.A#g",
        "class a.a.a.a.a.a.D#a",
        "class a.a.a.a.a.a.E#a",
        "class a.a.a.a.a.A#a",
        "class a.a.a.a.a.a.C#b",
        "class a.a.a.a.a.a.C#c",
        "class a.a.a.a.a.A#b",
        "class a.a.a.a.a.A#c",
        "class a.a.a.a.a.A#d",
        "class a.a.a.a.a.A#e",
        "class a.a.a.a.a.A#f",
        "class a.a.a.a.a.A#g",
        "class a.a.a.a.a.A#h",
        "class a.a.a.a.a.A#i",
        "class a.a.a.a.a.A#j",
        "class a.a.a.a.a.a.D#b",
        "class a.a.a.a.a.a.D#c",
        "class a.a.a.a.a.a.D#d",
        "class a.a.a.a.a.a.D#e",
        "class a.a.a.a.a.a.D#f",
        "class a.a.a.a.a.a.D#g",
        "class a.a.a.a.a.a.D#h",
        "class a.a.a.a.a.a.B#l",
        "class a.a.a.a.a.a.B#m",
        "class a.a.a.a.a.a.B#n",
        "class a.a.a.a.a.a.B#o",
        "class a.a.a.a.a.a.B#p",
        "class a.a.a.a.a.a.E#b",
        "class a.a.a.a.a.a.F#a",
        "class a.a.a.a.a.a.F#b",
        "class a.a.a.a.a.a.F#c",
        "class a.a.a.a.a.a.F#d",
        "class a.a.a.a.a.a.F#e",
        "class a.a.a.a.a.a.F#f",
        "class a.a.a.a.a.a.G#a",
        "class a.a.a.a.a.a.G#b",
        "class a.a.a.a.a.a.G#c",
        "class a.a.a.a.a.a.G#d",
        "class a.a.a.a.a.a.C#d",
        "class a.a.a.a.a.a.C#e",
        "class a.a.a.a.a.a.C#f"
      ]
    },
    "junit-ignored": []
  }
}

[pawelpasterz]

Thanks a lot!

I see we have 2 issues here (could be related though):

• tests are not shared at all
• missing testcases in result file

Ok, it may take more time than I expected to run flank with --dump-shards because we are using some custom gradle plugin to tun flank in Android.

Is it fladle? 🤔

[dmytrodanylyk]

Is it fladle? 🤔

No, we wrote our own testlab-plugin looong time ago to proxy all the work related to testlab, since we have multiple products which are using testlab.

It really looks like it just took 50 first testcase for some reasons. All testcase in test_result_1.xml and JUnitReport.xml are in perfect order.

It doesn't happen with multiple shards, e.g. if there are 4 matrix each contains N<50 number of tests, than JUnitReport.xml will have all 73 tests.

[dmytrodanylyk]

Hey, good news I found what the problem is.

    fun listTestCases(toolResultsStep: ToolResultsStep): ListTestCasesResponse {
        return service
            .projects()
            .histories()
            .executions()
            .steps()
            .testCases()
            .list(
                toolResultsStep.projectId,
                toolResultsStep.historyId,
                toolResultsStep.executionId,
                toolResultsStep.stepId
            ).executeWithRetry()
    }

Basically, this code executes an API request which only returns 50 results.

[dmytrodanylyk]

Here is the patch with a fix.

#1634_-_Fixed_issue_when_JUnitReport_xml_only_contained_50_test_results.patch.zip

[pawelpasterz]

Hey, good news I found what the problem is.

    fun listTestCases(toolResultsStep: ToolResultsStep): ListTestCasesResponse {
        return service
            .projects()
            .histories()
            .executions()
            .steps()
            .testCases()
            .list(
                toolResultsStep.projectId,
                toolResultsStep.historyId,
                toolResultsStep.executionId,
                toolResultsStep.stepId
            ).executeWithRetry()
    }

Basically, this code executes an API request which only returns 50 results.

Awesome! PRs are always welcome 😸
Anyway, we will use your solution once we will start working on this issue.

One puzzle is done, we need to find out why shading is not working in your case 🤔

[dmytrodanylyk]

Awesome! PRs are always welcome 😸

I will try to send PR tomorrow, it has to be from the forked repo right?

One puzzle is done, we need to find out why shading is not working in your case 🤔

I think it just hit the limit, we have over 100 modules and 800 android tests

[bootstraponline]

I will try to send PR tomorrow, it has to be from the forked repo right?

Yes, please send a PR from a forked repo. Thanks for debugging and fixing the issue.