Flank: The repeated field 'google.devtools.testing.v1.TestTargetsForShard.test_targets' should have at least 1 values (found 0)

[Kurt-Bonatz]

The latest snapshot of Flank fails to run due to a 400 error with CTL. Git bisect blames the change #621. Currently, I've tried to remove any test filtering (test-targets in the config) which also produced the same issue. Reproducing with additional apks requires backporting #667 for most revisions. Let me know if I can provide more information.

Flank args output:

AndroidArgs
    gcloud:
      results-bucket: <bucket_name>
      results-dir: null
      record-video: false
      timeout: 30m
      async: false
      results-history-name: null
      # Android gcloud
      app: output/app-dev-debug.apk
      test: output/app1-dev-debug-androidTest.apk
      auto-google-login: false
      use-orchestrator: true
      directories-to-pull:

      performance-metrics: false
      test-runner-class: null
      test-targets:
        - notAnnotation androidx.test.filters#Suppress, org.junit#Ignore
      device:
        - model: Nexus7
          version: 22
          locale: en
          orientation: portrait
      num-flaky-test-attempts: 0

    flank:
      max-test-shards: 50
      shard-time: 45
      num-test-runs: 1
      smart-flank-gcs-path: gs://<bucket_name>/test_results.xml
      smart-flank-disable-upload: true
      files-to-download:

      test-targets-always-run:

      disable-sharding: false
      project: <project-id>
      local-result-dir: results
      # Android Flank Yml
      keep-file-path: false
      additional-app-test-apks:
        - app: null
          test: output/app2-dev-debug-androidTest.apk
        ...
        - app: null
          test: output/app22-dev-debug-androidTest.apk

Error output:

2556 tests / 413 shards

Request failed, retrying 1x com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "The repeated field 'google.devtools.testing.v1.TestTargetsForShard.test_targets' should have at least 1 values (found 0)",
    "reason" : "badRequest"
  } ],
  "message" : "The repeated field 'google.devtools.testing.v1.TestTargetsForShard.test_targets' should have at least 1 values (found 0)",
  "status" : "INVALID_ARGUMENT"
}
...
java.io.IOException: Request failed
    at ftl.http.ExecuteWithRetryKt.executeWithRetry(ExecuteWithRetry.kt:20)
    at ftl.run.AndroidTestRunner$runTests$2$invokeSuspend$$inlined$map$lambda$1.invokeSuspend(AndroidTestRunner.kt:52)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:241)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:740)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "The repeated field 'google.devtools.testing.v1.TestTargetsForShard.test_targets' should have at least 1 values (found 0)",
    "reason" : "badRequest"
  } ],
  "message" : "The repeated field 'google.devtools.testing.v1.TestTargetsForShard.test_targets' should have at least 1 values (found 0)",
  "status" : "INVALID_ARGUMENT"
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:443)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1092)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
    at ftl.http.ExecuteWithRetryKt.executeWithRetry(ExecuteWithRetry.kt:13)
    ... 6 more

[bootstraponline]

Thank you for reporting! We'll take a look tomorrow.

[pawelpasterz]

@Kurt-Bonatz @bootstraponline
I think we've found cause of this issue. I must admit, it wasn't that easy to reproduce it (at least from our perspective) :)
First I'd like to clarify what The latest snapshot of Flank means because logs you printed are from 'older snapshot' version (~month old), ex AndroidTestRunner that appears in stack trace was removed with #661 . Just wanted to make sure we are on the same page.

We reproduced bug by providing flank test apk with all tests annotated with @Ignore and, what is important, without filtering them. There was a request to attach @Ignored test to test matrix so they could be visible in results (#629).

Next all ignored tests have assigned run time 0 to neglect their influence while calculating shards by time. Under the hood Shard#createShardsByShardCount sorts shards by time and since kotlins sortedBy is stable therefor flank eventually references the same shard leaving other(s) without test methods.

This is 100% reproducible with all @Ignored tests in apk and is related with number of ignored methods and shards.

We still wouldn't be aware of this issue if it weren't for incorrect filter
- notAnnotation androidx.test.filters#Suppress, org.junit#Ignore
where should be
- notAnnotation androidx.test.filters.Suppress, org.junit.Ignore
and that would filter Ignored methods, error wouldn't occur but the issue would be still covered :)

That is what we came up with. Please @jan-gogo add something if missing or correct me.

[Kurt-Bonatz]

I appreciate the effort and apologize for some of the confusion! By latest snapshot I was going by the latest release as of yesterday at time of writing. The old logs were just a case of me grabbing the last test run I had executed which was during a git bisect.

The reason for the filters were because we were seeing matrices come back with errors about no tests run. We have very strict suppression of flaky tests so this wasn't too unexpected so I tried to avoid it by removing the addition of the suppressed tests entirely. I realized the filtering mistake shortly after signing off for the day 🤦. Glad it actually helped you though!

Just ran a test run of #704 and everything works as expected with the corrected filters! Thanks again for the quick fix :). Let me know if you need anymore info or another test run.

[pawelpasterz]

@Kurt-Bonatz actually that was quite interesting digging and investigating this issue :)
Please, keep as informed if you will encounter any issue