Flank: iOS: JUnitReport.xml contains data from previously used device

[andreschab90]

case:

• run test with iphone x on ios 12
• change device to iphone 11 with ios 13
• run tests
  -> JUnitReport.xml contains results from iphone x, which was not used in last execution

[pawelpasterz]

Hey @andreschab90 !
I was unable to reproduce this error with steps you provided.
Is it possible for you to share more info? (yml file would be best, flank version, etc)
Thanks!

[andreschab90]

@bootstraponline @pawelpasterz i had a look into this issue again. The issue with old devices seems to be fixed, but i got a really critical issue when using latest snapshot. I have almost 70 tests, which runs in 13 shards, but on snapshot every shard contains the same 5 tests. The total amount was also 130 tests. So every testcase was executed 26 times. i have this behaviour only on snapshot version.

[pawelpasterz]

@andreschab90 Thanks for answer, could you share with us yml file?

@bootstraponline
I think we can consider this as high priority

[andreschab90]

@pawelpasterz here is my flank.yml

gcloud:
  results-bucket: testlab-ios
  record-video: true
  timeout: 30m
  async: false
  results-history-name: IOS
  num-flaky-test-attempts: 1
  test: $BITRISE_TEST_BUNDLE_ZIP_PATH
  xctestrun-file: $BITRISE_XCTESTRUN_FILE_PATH
  device:
  - model: iphonexs
    version: 12.3
    locale: en
    orientation: portrait
flank:
  max-test-shards: 13
  shard-time: -1
  num-test-runs: 1
  smart-flank-gcs-path: gs://testlab-ios/firebase-test-calculation/results.xml
  smart-flank-disable-upload: true
  disable-sharding: false
  project: testing

[bootstraponline]

I think we can consider this as high priority

Agreed

[bootstraponline]

we probably need to update the nm and swift-demangle binaries.

https://github.com/Flank/binaries

[bootstraponline]

I updated the Flank binaries. In general it seems the new Xcode has caused some problems for Flank's iOS support.

We'll have to test this using the Flank sample app and see if the problems reproduce.

[pawelpasterz]

The cause of this error is flank's internal caching mechanism. For every shard xctestrun file is rewritten but uploaded with the same filename (cache key is filename). Therefor with sharding enabled all shards had the same tests launched.

Fix #759

[pawelpasterz]

Hey @andreschab90 , fix is merged into master. If this is possible, please check if that works in your case, thanks!

[andreschab90]

@pawelpasterz sure we check it next week. Thanks

[andreschab90]

@pawelpasterz every testcase is listed twice in the jnut.xml

[pawelpasterz]

@andreschab90
I've found cause. This was due to fact that flank used non-legacy approach for parsing fetched xml's. Fix can be found #783.
Thanks for reporting those issues, iOS part of flank's code was a bit outdated and now it's refreshed :)

cc
@bootstraponline

[andreschab90]

@pawelpasterz thx will Check it out later

[bootstraponline]

The fix has been released in Flank v20.05.2. https://github.com/Flank/flank/releases/tag/v20.05.2

[andreschab90]

@pawelpasterz @bootstraponline fix works..thx