Fastlane-plugin-test_center: json report files being overwritten and are not collated

[lucasecf]

New Issue Checklist

• [x] Updated fastlane-plugin-test_center to the latest version
• [x] I read the README.md
• [x] I reviewed the example Fastfile

Issue Description

This issue is regarding multi_scan. Before opening the issue, I did try to look for it before and found some similar ones, like this, but couldn't find exactly the same problem, sorry in advance if I missed it.

My problem is, I'm using scan's formatter with "xcpretty-json-formatter", which is a gem that uses xcpretty to output the result in a json format, that our CI (danger) uses to display the result on GH. Since multi_scan runs xcodebuild multiple times (at least once for UI tests and once for Unit tests), this report json file is always overridden and only the last value is reported to GH.

This is how multi_scan was originally being called:

multi_scan(scheme: "Smoke Tests",
                   devices: ["iPhone 8"],
                   clean: true,
                   try_count: 3,
                   formatter: "xcpretty-json-formatter")

The only workaround I've found so far is to use the testrun_completed_block option to change the json file name, so it generate multiple ones:

  Dir.glob("build/reports/*.json").each { |path| File.delete(path) }
  counter = 0
  test_run_block = lambda do |testrun_info|
      counter += 1
      ENV["XCPRETTY_JSON_FILE_OUTPUT"] = "build/reports/errors#{counter}.json"
  end

  test_run_block

  multi_scan(scheme: "Smoke Tests",
                     devices: ["iPhone 8"],
                     clean: true,
                     try_count: 3,
                     testrun_completed_block: test_run_block,
                     formatter: "xcpretty-json-formatter")

(At the same time changing our Dangerfile to 'upload' all json files in the folder)

I haven't tested this thoroughly, but it seems to work I guess. Is there any better/cleaner option than that? Thanks in advance!

[lyndsey-ferguson]

I specifically _collate_ the regular junit xml and html report files into one file with the final results _as if_ the tests had run once for a complete report. It would seem that these json files should _also_ be _collated_. What are your thoughts?

[lucasecf]

HI @lyndsey-ferguson , thanks for the ultra-fast reply.

Yes, _collating_ all files into one seems like a good idea for this case also, and would work fine. That or reporting all json files doesn't change much things on my side, I'm more interested in if there's any cleaner way to "generate" the different files per run besides this counter + testrun_completed_block I posted in the question.

Do you have any lead or light to help me in this part?
Thanks again!

[lyndsey-ferguson]

@lucasecf I don't think that there is a cleaner way, that would be my default suggestion to work around this until I add this feature. Actually, you do get a try_count in the callback that you could use instead of keeping your own counter...try using puts testrun_info to get an idea of what you get in the Hash.

[lucasecf]

@lyndsey-ferguson I just realized I _can't_ keep sending all json files to our CI because it recognizes the whole PR build as failure if any test execution contains a failure. So yeah, collating all files results before "reporting" seems like the only way indeed...

Thanks for the try_count tip, I'll see what I can do meanwhile. Please let me know if I can help with anything for the _bug_ fix.

[lyndsey-ferguson]

I am open to PRs; adding this new functionality could take me a week or two depending on my work load and upcoming vacation (I don't have a lot of time right now).

To see how I added a collate_html_reports action, and called it from multi_scan see [this commit]. You can also review the collate_junit_reports action to see how it works. I think that collating json will probably be the easiest to do.

[lucasecf]

I'd love to contribute if possible, I'm not really a ruby developer but will try to at least have a look when having some free time.

One thing to highlight is that my case is slightly different from html/junit cases - those files are outputted directly by scan (through output_files param).

This json file I need is "outputted" by xcpretty instead, which scan proxies through the formatter parameter (... | xcpretty -f 'xcpretty-json-formatter'...). The output path is not controlled by scan's output_directory , but rather a separate env variable (XCPRETTY_JSON_FILE_OUTPUT) the gem expects (https://github.com/marcelofabri/xcpretty-json-formatter#usage).

In this case, I think it wouldn't be applied to your CorrectingScanHelper (at least not in the same way I guess), I'm not sure what would be the solution...

[lyndsey-ferguson]

Thank you for the tips. Knowing that the code should look for that environment variable will help me determine when the collation needs to be done.

[lyndsey-ferguson]

Note to self: it seems that the cleanest and most obvious would be to add a json output type to multi_scan and then set up the environment/parameters correctly so that scan will generate the files appropriately.

[lyndsey-ferguson]

@lucasecf I am considering simply extending scan's :output_types to include simple json. I would then strip that off in multi_scan and set up the XCPRETTY_JSON_FILE_OUTPUT environment variable to point to the file that is going to be output.

However, that does seem a little confusing because there is an existing option, json-compilation-database. What do you think?

[lucasecf]

I think the json-compilation-database is the confusing one, because it's not usable for CI reports... Not sure what is it's usercase tbh. If scan had a proper json option by default (together with junit and html) it would make a lot of sense imho

[lyndsey-ferguson]

Completed in v3.4.0