Flank: Flank dump shards is inconsistent with what flank is running

[MatthewTPage]

Describe the bug
I've been seeing some weird behavior with some tests being ran twice, similar to the issue solved in https://github.com/Flank/flank/pull/1048 but on a much smaller scale than I described there - I wanted to view the shard dump to see if anything obvious stuck out but when I run it, I get a shard dump containing only 3 test cases - I have over 2,000 test cases in my suit and flank finds them and runs those properly when I don't tell it to shard dump, so it seems there's an inconsistency between what shard dump outputs and what Flank actually runs.

This issue seems new - v20.09.1 has this issue for me, but v20.08.3 seems to show the proper shards as I expect.

To Reproduce

1. Run shard dump on a large test suit.
2. See that it only shows a small number of tests.

Expected behavior
It shows the tests in my suit.

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

Yes.

Post the output of flank --version.

I'm using fladle, but I've been working off of the latest snapshot.

Additional context
If it matters I'm using fladle, but because the fladle version can remain constant and I can see it work or not work by changing the flank version, this seems like an issue with flank.

Here are my flank configs (with APK paths removed): https://gist.github.com/MatthewTPage/a356791c742483e211777202403405c4

gcloud:
  device:
  - model: Nexus7
    version: 22
    locale: en_US

  use-orchestrator: true
  auto-google-login: false
  record-video: false
  performance-metrics: false
  timeout: 40m
  results-bucket: # A valid bucket name
  num-flaky-test-attempts: 0

flank:
  max-test-shards: 50
  shard-time: 120
  smart-flank-gcs-path: # a valid gs path
  project: api-project-516222937551
  keep-file-path: false
  run-timeout: 45m
  ignore-failed-tests: false
  disable-sharding: false
  smart-flank-disable-upload: false
  legacy-junit-result: true
  full-junit-result: false
  output-style: single

[bootstraponline]

Thanks for reporting! I've moved this to the top of the Ranked column on ZenHub so the team can investigate.

[MatthewTPage]

@bootstraponline Thanks! If it helps, it looks like the issue is dump shards is only processing one test APK. I pass flank about 20 apk pairs to run tests on.

[pawelpasterz]

@MatthewTPage Hey! Could you share with us dump-shards output?
Also, do you have parameterized tests in your test suite?

[MatthewTPage]

hi @pawelpasterz!

Here's the dump-shards output, with the test case names obfuscated. Again, these are all of the tests from the first APK flank processes, it just fails to process the other ~20 APK pairs for the dump-shards output.

[
  [
    "class com.example.A#a",
    "class com.example.B#b"
  ],
  [
    "class com.example.C#c"
  ]
]

I don't think we have any parameterized tests in our suit. If we do and I just can't find them, there certainly aren't too many of them.

[pawelpasterz]

Thanks! That's a lot of information you gave us and I think we have anything we need

[pawelpasterz]

Hey @MatthewTPage

Here is the summary what I have tried so far:
I launched flank run with 20.08.3 version with config based on the one provided by you. Then I verified if dump shards output was correct. Then I switched to 20.09.1 and 20.09.2 and checked on both. All the results were correct. Then I used fladle to perform the same steps (flank version switched within fladle config). Unfortunately, I was not able to reproduce the error described by you.
I manipulated some values in config (like shardTime, testShards etc) but nothing works for me...
I had 20 test app pairs (but to be fair it was like 200 tests in total) and I used fladle ver 0.12.1 I assume you used the same one.

I reached the wall and cannot jump over it.

Any additional info would be more than welcome.

cc @piotradamczyk5

[MatthewTPage]

@pawelpasterz thanks for investigating! I ran dump shards using the now latest snapshot, and it's working again... super strange. I tested 20.9.1 again and was indeed seeing the issue again there.

No idea what was going on, must have been something weirdly specific to my company's setup, but it seems to be magically fixed now.

Thanks again for the help.

[pawelpasterz]

@MatthewTPage I am glad it is working correctly now. I am super curious why it was happening 🤔 If this issue will occur again feel free to reopen or contact me directly (via slack or email)