Flank: Shards failed - no test cases inside

[bootstraponline]

Mr.uno

One or more shards failed because there are no test cases inside. Please check your sharding configuration. I've encountered this a couple times with our submodule tests which use a lot of additional test apks. Maybe some error somewhere in that logic - I've only ever seen it with the submodule tests. This is from the latest Flank version.

https://firebase-community.slack.com/archives/C72V6UW8M/p1590537597298100

[inktomi]

Thank you for getting this filed for me! I'm not feeling well today, but I wanted to provide our config as well.

We build a dummy app for the app - it's basically just the app you get from the "New Application" template in Android Studio.

flank:
  additional-app-test-apks:
  - test: feature/1/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/2/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/3/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/4/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/5/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/6/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/7/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/8/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/9/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/10/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/11/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/12/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/13/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/14/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/15/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/16/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/17/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/18/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/19/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/20/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/21/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/22/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/23/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/24/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/25/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/26/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  - test: feature/27/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  legacy-junit-result: 'false'
  local-result-dir: sub-modules-flank-results
  max-test-shards: 5
  project: project-id-12345
  run-timeout: 20m
  smart-flank-gcs-path: gs://project-id-12345-flank/submodules-timing.xml
gcloud:
  app: ./sample-app/build/outputs/apk/debug/sample-app-tester-debug.apk
  async: false
  auto-google-login: false
  device:
  - model: NexusLowRes
    version: 23
  environment-variables:
    clearPackageData: true
  performance-metrics: false
  record-video: false
  results-bucket: sub-modules-flank-results
  test: feature/28/build/outputs/apk/androidTest/debug/android-debug-androidTest.apk
  test-targets:
  - notAnnotation org.junit.Ignore
  use-orchestrator: true

I was getting an error like this before I added the notAnnotation to ensure that Ignored tests were not causing issues. After I added that, it seemed like maybe a bucket consisted entirely of ignored tests or something like that and I started getting the error in the original report here about a shard having no tests. I only noticed One or more shards failed because there are no test cases inside. Please check your sharding configuration. in the Firebase UI after adding the test filter but the error seemed to happen in both cases since the stacktrace is consistent.

 java.lang.ClassNotFoundException: Invalid name: no tests found
at java.lang.Class.classForName(Native Method)
at java.lang.Class.forName(Class.java:324)
at androidx.test.internal.runner.TestLoader.doCreateRunner(TestLoader.java:72)
at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:104)
at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:793)
at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:547)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:390)
at com.work.android.test.view.ViewTestRunner.onStart(ViewTestRunner.kt:25)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1879) 

[bootstraponline]

Thank you for the detailed information. I've added this to the top of our ranked column. It'll be worked on by the team soon.

You can install https://www.zenhub.com/extension if you'd like to see the ZenHub board on the GitHub issues page. 🙂

[jan-gogo]

I wonder if the required fix is already implemented here and is waiting for code review. This implementation filters out empty shards and drops apks without shards. So no empty shards and no apks without shards should be passed to execution.

[inktomi]

I’m happy to build flank off that branch and give it a try. I can probably do that tomorrow.
On May 27, 2020, 3:12 PM -0700, Jan Góral [e-mail], wrote:

I wonder that required fix is already implemented here and is waiting for code review. This implementation filters out empty shards and drops apks without shards. So no empty shards and no apks without shards should be passed to execution.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[inktomi]

I built flank off the android-test-context branch using ./gradlew clean assembleShadowDist and still see the issue.

For example, matrix-29sueztfa3twi reports 5 shards for me. Overall it's failed. Each shard is shown as red in the Firebase UI. Firebase is not showing the warning in the original report about the shards failing because of no test cases, however as you can see, the shard seemed to fail because of no test cases.

Overall matrix..

Clicking into the first shard

Details on the null failure

[inktomi]

Here's a heavily redacted shards output to help. If you need more specific information, I can work to get that for you.

android_shards.json.zip

[inktomi]

This was improperly closed by #817 @bootstraponline @jan-gogo. The issue reproduced for me on that branch.

[bootstraponline]

@inktomi thanks for catching this!

[technoir42]

We have the same issue. In total there are ~200 instrumented tests but we have a custom annotation-based filter and execute only 5-10% of tests at a time. If number of shards > 2, there is a high chance that one of the shards will have no tests and will fail the whole matrix.
Another, related problem is that tests are not evenly distributed among shards after filtering. Is this something that can be improved on Flank side?

[bootstraponline]

Thanks for this feedback. I have raised this with the team.

[jan-gogo]

@inktomi @technoir42 Do you use test-targets-always-run and additional-apks in your config? The answer may help resolve this problem.

[technoir42]

We don't use either of them:

fladle {
    serviceAccountCredentials = file("pathToCredentials")
    useOrchestrator = true
    environmentVariables = [
        "clearPackageData"     : "true"
    ]
    timeoutMin = 30
    recordVideo = true
    performanceMetrics = false
    devices = [
        ["model": "Pixel2", "version": "27", "orientation": "portrait", "locale": "en"]
    ]
    projectId = "id"
    flankVersion = "20.05.2"
    testShards = 5
    flakyTestAttempts = 0
}

[pawelpasterz]

@technoir42
Do your test apks have the same name? (but different directories)

[inktomi]

In our run, we use additional-apks but not test-targets-always-run - you can see our config up a bit in this issue. Our test apks are all named uniquely after the feature module they're built from. Each has a unique path and name.

In the config above I renamed the files just to keep the feature names private.

[pawelpasterz]

In our run, we use additional-apks but not test-targets-always-run - you can see our config up a bit in this issue. Our test apks are all named uniquely after the feature module they're built from. Each has a unique path and name.

In the config above I renamed the files just to keep the feature names private.

Flank has both options additional-apks[1] and additional-app-test-apks[2]. I can see only [2] in config you posted. Just wanted to be sure we are on the same page :)

[inktomi]

Sorry, you're right. Apologies.

[bootstraponline]

Some additional context from Slack: https://firebase-community.slack.com/archives/C1MTSQ5QT/p1591032162198200

Firebase Test Lab considers any shard with all skipped/ignored tests to be failed. I think that's a design defect in FTL and the team is considering making this behavior configurable. In the meantime, Flank should add some intelligence to ensure test shards contain only tests that we expect to run.

[pawelpasterz]

Firebase Test Lab considers any shard with all skipped/ignored tests to be failed. I think that's a design defect in FTL and the team is considering making this behavior configurable. In the meantime, Flank should add some intelligence to ensure test shards contain only tests that we expect to run.

Flank had such feature before -- skip all @Ignore tests. But there was a feature request to attach those to test run.
Also, I don't think it's the case here since ignored tests looks like this in FTL matrix :

but here we have different outcome
https://user-images.githubusercontent.com/74583/83170254-1b976a00-a0c9-11ea-8b7b-d689e9c086c1.png

I managed to receive error when test targets where incompatible with apk (in other words, test targets were from different apk). But that only happened to me when some of test apks had the same name (caching issue I mentioned some time ago). But according to https://github.com/Flank/flank/issues/818#issuecomment-638930346 it's not the case...

[pawelpasterz]

Flank should add some intelligence to ensure test shards contain only tests that we expect to run.

On the second thought, I think we could add warning, throw an error and terminate run (or make it configurable) in case when one shard would consists of ignored tests only, WDYT?

[bootstraponline]

On the second thought, I think we could add warning, throw an error and terminate run (or make it configurable) in case when one shard would consists of ignored tests only, WDYT?

As flank specifies exactly which tests to run on FTL via test targets, I think we should filter out ignored tests before sending them to FTL.

I think the broader issue is Flank currently lacks an end to end test to simulate this large multi-module use case and detect bugs. We should probably invest in this area to be more proactive about finding problems both in Flank and FTL itself.

[inktomi]

I agree, I think it would be best for flank to filter tests that are marked @Ignore. Maybe to future proof that, @Ignore would be the default, but make it configurable somehow. I know some teams use different annotations for flaky tests vs disabled tests etc.

[pawelpasterz]

@bootstraponline @inktomi
Got your point. What I am trying to say is, behavior you both are describing was implemented at very beginning of flank. There was #629 feature request to change it and include ignored tests. That time the only way to achieve it was to include @Ignore tests in matrix. (maybe there is another way now? )

IMHO original behavior (filtering ingored) was better. (now it can be achieved with

test-targets:
  - notAnnotation org.junit.Ignore

I just want explicitly point that it's kind of regression and decide if we are ok with that.

Changing this behavior will be easy so it's all matter of decision.

I think the broader issue is Flank currently lacks an end to end test to simulate this large multi-module use case and detect bugs. We should probably invest in this area to be more proactive about finding problems both in Flank and FTL itself.

Yes, please! I would love to have some time (without new features) to implement it with the team

[bootstraponline]

What I am trying to say is, behavior you both are describing was implemented at very beginning of flank. There was #629 feature request to change it and include ignored tests. That time the only way to achieve it was to include @Ignore tests in matrix. (maybe there is another way now? )

I think there's some missing context, Firebase use to consider ignored tests as passing so there was no cost for Flank to send ignored tests. At some point Firebase decided to treat ignored tests as failing when they make up an entire shard. In general when Firebase changes behavior then Flank often has to follow as they control the logic for matrix outcomes.

If I understand the request in https://github.com/Flank/flank/issues/629, the user wants @Ignore tests represented in the XML. Maybe we could do the following:

• Filter all @Ignore / skipped / etc. tests before sending them to FTL
• Keep track in Flank of all the Ignored tests
• FTL runs the matrix, and at the end Flank adds in the @Ignore tests to the final merged XML report

What do you think?

Yes, please! I would love to have some time (without new features) to implement it with the team

Awesome. Yeah I think we should pause all the new feature work until we invest more in quality.

[pawelpasterz]

I think there's some missing context, Firebase use to consider ignored tests as passing so there was no cost for Flank to send ignored tests. At some point Firebase decided to treat ignored tests as failing when they make up an entire shard. In general when Firebase changes behavior then Flank often has to follow as they control the logic for matrix outcomes.

👍 I was thinking the same as well

If I understand the request in #629, the user wants @Ignore tests represented in the XML. Maybe we could do the following:
Filter all @Ignore / skipped / etc. tests before sending them to FTL
Keep track in Flank of all the Ignored tests
FTL runs the matrix, and at the end Flank adds in the @Ignore tests to the final merged XML report
What do you think?

Agreed, with current flank (after refactor and with our better understanding of flank and FTL) we should be able to achieve it. But still, I am not sure if this should be resolved within this issue, I believe ignored tests are not root cause. But I might be wrong of course. @inktomi do you have some ignored tests within your apk? Is it possible for you to verify if there was shard with only ignored tests inside?

[bootstraponline]

I believe ignored tests are not root cause

Agreed.

[pawelpasterz]

@inktomi @technoir42
We were not able to reproduce this error with the info you provided us (and believe me, we tried dozens of times :) ) but we tracked down few issues that might be potential root cause and fixed it with #853 and #858. Fixes are already merged and available in snapshot version https://github.com/Flank/flank/releases/tag/flank_snapshot. We tested solutions according to our best knowledge of your case but still can't be 100% sure that 'this is it'.

It will be extremely helpful if you could help us and verify if the issue still occurs with snapshot version.
Also, please check https://github.com/Flank/flank/pull/855/files and let us know if you could provide us any information that is missing there.

Thanks in advance!

cc
@bootstraponline

[bootstraponline]

@inktomi Please retest with Flank v20.06.2, then provide matrix ids for the shards that fail so we can investigate the root cause.

[bootstraponline]

I talked with Mr.uno on Slack today. Updating to the latest Flank and AndroidX libraries seems to have fixed the problem. Amex was on AndroidX Test v1.1.0, however firebase is using Orchestrator v1.2.0. My current theory is that this error is related to having an out of date AndroidX library and using sharding with orchestrator.

• https://firebase-community.slack.com/archives/C1MTSQ5QT/p1592526170398200?thread_ts=1592498669.392500&cid=C1MTSQ5QT
• https://github.com/android/android-test/issues/264
• https://github.com/android/android-test/releases/tag/androidx-test-1.3.0-rc01
• https://github.com/Flank/flank/releases/tag/v20.06.2

[jan-gogo]

Updating to the latest Flank and AndroidX libraries seems to have fixed the problem

@bootstraponline So, do you think, can we close this issue?

[bootstraponline]

@bootstraponline So, do you think, can we close this issue?

I think we can close the issue after updating the documentation. Something like, if you see the error "One or more shards failed because there are no test cases inside." then update Flank and all the AndroidX libs to the latest version.

[inktomi]

I agree, after doing that on our project the error seemed to go away. I used the latest Test Orchestrator, 1.3x

[adecker89]

I've started seeing this issue with
Flank: v20.09.3
Test Orchestrator: 1.3.0
AndroidX Test: 1.3.0

adding notAnnotation org.junit.Ignore works as a workaround.

[bootstraponline]

@adecker89 can you raise this on the FTL Slack and post a matrix id? The root cause is FTL does not allow users to specify that no tests should be a successful run instead of a failure.

[adecker89]

Thanks, I've done so.

As an update, I actually think that in my case this error was due to sending a @RequiresDevice annotated test to an emulator and not actually related to @Ignore. I will most likely be able to explicitly filter out that annotation since I'm only targeting emulators in this particular device matrix, but I could see it being troublesome if I needed to target both devices and emulators together.