Flank: java.lang.IllegalStateException: response.testCases must not be null

[AaronMT]

Running Flank Snapshot (f000fe4290c6bf708b83bd84e20f183bd84994e0) and a PR in our project (https://github.com/mozilla-mobile/fenix/pull/10101) hit the following:

[task 2020-04-22T17:59:33.320Z]   5m 50s matrix-1bbsdnipvmr70 Pixel2-28 FINISHED
[task 2020-04-22T17:59:46.865Z]   6m  4s matrix-1bbsdnipvmr70 FINISHED
[task 2020-04-22T17:59:46.865Z] 
[task 2020-04-22T17:59:46.867Z] FetchArtifacts
[task 2020-04-22T17:59:55.485Z]   .................................................
[task 2020-04-22T17:59:55.485Z]   Updating matrix file
[task 2020-04-22T17:59:55.485Z] 
[task 2020-04-22T17:59:56.096Z] java.lang.IllegalStateException: response.testCases must not be null
[task 2020-04-22T17:59:56.096Z]     at ftl.reports.api.CreateTestExecutionDataKt.createTestExecutionData(CreateTestExecutionData.kt:34)
[task 2020-04-22T17:59:56.096Z]     at ftl.reports.api.CreateTestExecutionDataKt$createTestExecutionData$1.invokeSuspend(CreateTestExecutionData.kt)
[task 2020-04-22T17:59:56.096Z]     at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
[task 2020-04-22T17:59:56.096Z]     at kotlinx.coroutines.ResumeModeKt.resumeUninterceptedMode(ResumeMode.kt:45)
[task 2020-04-22T17:59:56.096Z]     at kotlinx.coroutines.internal.ScopeCoroutine.afterCompletionInternal(Scopes.kt:32)
[task 2020-04-22T17:59:56.096Z]     at kotlinx.coroutines.JobSupport.completeStateFinalization(JobSupport.kt:310)
[task 2020-04-22T17:59:56.096Z]     at kotlinx.coroutines.JobSupport.tryFinalizeFinishingState(JobSupport.kt:236)
[task 2020-04-22T17:59:56.096Z]     at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:849)
[task 2020-04-22T17:59:56.096Z]     at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:811)
[task 2020-04-22T17:59:56.096Z]     at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:787)
[task 2020-04-22T17:59:56.096Z]     at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:111)
[task 2020-04-22T17:59:56.097Z]     at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
[task 2020-04-22T17:59:56.097Z]     at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:241)
[task 2020-04-22T17:59:56.097Z]     at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
[task 2020-04-22T17:59:56.097Z]     at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
[task 2020-04-22T17:59:56.097Z]     at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:740)
[task 2020-04-22T17:59:56.117Z] 
[task 2020-04-22T17:59:56.117Z] 
[task 2020-04-22T17:59:56.117Z] FAILURE: UI test run failed, please check above URL

Flank Configuration

[task 2020-04-22T17:53:06.990Z] AndroidArgs
[task 2020-04-22T17:53:06.990Z]     gcloud:
[task 2020-04-22T17:53:06.990Z]       results-bucket: fenix_test_artifacts
[task 2020-04-22T17:53:06.990Z]       results-dir: null
[task 2020-04-22T17:53:06.990Z]       record-video: true
[task 2020-04-22T17:53:06.990Z]       timeout: 30m
[task 2020-04-22T17:53:06.990Z]       async: false
[task 2020-04-22T17:53:06.990Z]       client-details: 
[task 2020-04-22T17:53:06.990Z]       network-profile: null
[task 2020-04-22T17:53:06.990Z]       results-history-name: null
[task 2020-04-22T17:53:06.990Z]       # Android gcloud
[task 2020-04-22T17:53:06.990Z]       app: /builds/worker/checkouts/src/app.apk
[task 2020-04-22T17:53:06.990Z]       test: /builds/worker/checkouts/src/android-test.apk
[task 2020-04-22T17:53:06.990Z]       auto-google-login: true
[task 2020-04-22T17:53:06.990Z]       use-orchestrator: true
[task 2020-04-22T17:53:06.990Z]       directories-to-pull:
[task 2020-04-22T17:53:06.990Z]         - /sdcard/screenshots
[task 2020-04-22T17:53:06.990Z]       performance-metrics: true
[task 2020-04-22T17:53:06.990Z]       test-runner-class: null
[task 2020-04-22T17:53:06.990Z]       test-targets:
[task 2020-04-22T17:53:06.990Z]         - package org.mozilla.fenix.ui
[task 2020-04-22T17:53:06.990Z]       device:
[task 2020-04-22T17:53:06.990Z]         - model: Pixel2
[task 2020-04-22T17:53:06.990Z]           version: 28
[task 2020-04-22T17:53:06.990Z]           locale: en
[task 2020-04-22T17:53:06.990Z]           orientation: portrait
[task 2020-04-22T17:53:06.990Z]       num-flaky-test-attempts: 1
[task 2020-04-22T17:53:06.990Z] 
[task 2020-04-22T17:53:06.990Z]     flank:
[task 2020-04-22T17:53:06.990Z]       max-test-shards: 50
[task 2020-04-22T17:53:06.990Z]       shard-time: -1
[task 2020-04-22T17:53:06.990Z]       num-test-runs: 1
[task 2020-04-22T17:53:06.990Z]       smart-flank-gcs-path: 
[task 2020-04-22T17:53:06.990Z]       smart-flank-disable-upload: false
[task 2020-04-22T17:53:06.990Z]       files-to-download:
[task 2020-04-22T17:53:06.990Z]       test-targets-always-run:
[task 2020-04-22T17:53:06.990Z]       disable-sharding: false
[task 2020-04-22T17:53:06.990Z]       project: moz-fenix
[task 2020-04-22T17:53:06.990Z]       local-result-dir: results
[task 2020-04-22T17:53:06.990Z]       # Android Flank Yml
[task 2020-04-22T17:53:06.991Z]       keep-file-path: false
[task 2020-04-22T17:53:06.991Z]       additional-app-test-apks:
[task 2020-04-22T17:53:06.991Z]       run-timeout: -1
[task 2020-04-22T17:53:06.991Z]       legacy-junit-result: false

[jan-gogo]

Hi @AaronMT, https://github.com/Flank/flank/commit/0d1155db282087ad57fda53c4cb49f1c5e47e62e#diff-f0556ef0c7b1fa5a40e7d8be03562cfb fixes NPE with missing testcases. Please checkout the latest snapshot.

[AaronMT]

I'll report back if I see this happen with 7b518f26d5d06f510901b1ec8b61421239e2edb3 in the next few days (had only seen this once yesterday).

[pawelpasterz]

Hey @AaronMT , any news? Have you encountered this problem?

[AaronMT]

So far so good @pawelpasterz (using snapshot based off 7b518f2). Feel free to close this and I'll re-open if I see it again.