Firebase-ios-sdk: Add coverage reporting to CI

[paulb777]

Investigate a solution like https://github.com/SlatherOrg/slather and report coverage for the travis unit test runs.

[wilhuff]

I recently got this partially working for Firestore (not the Travis part)

slather coverage -s
  -i 'Firestore/third_party/abseil-cpp/*'
  -i 'Firestore/Example/*'
  -i 'Firestore/Protos/*'
  --configuration=Debug --scheme=AllTests
  --workspace=Firestore/Example/Firestore.xcworkspace
  --binary-file $HOME/Library/Developer/Xcode/DerivedData/Firestore-byuxnozrvymsezgkpumlkvgmfmtu/Build/Products/Debug-iphonesimulator/Firestore_Example.app/Firestore_Example
  Firestore/Example/Firestore.xcodeproj

Unfortunately there's a problem in the interaction between Xcode 9 and slather that's preventing it from finding the binary based on the configuration and I wasn't able to spend more time figuring out what was going wrong: https://github.com/SlatherOrg/slather/issues/347

[Corrob]

I can successfully collect code coverage locally with the following steps:

• Pod install
• Open Firebase.xcworkspace
• Configure the test scheme to gather coverage
• Run all tests
• Run "slather coverage --html --scheme AllUnitTests_iOS --configuration=Debug --workspace Firebase.xcworkspace/ Firebase.xcodeproj/"
• This produces
  Slathering...
  warning: 14 functions have mismatched data
  warning: 14 functions have mismatched data
  Database/App/iOS/FIRAppDelegate.m: 7 of 21 lines (33.33%)
  ...
  ../Firebase/Database/third_party/Wrap-leveldb/APLevelDB.mm: 170 of 285 lines (59.65%)
  Tested 8935/14687 statements
  Test Coverage: 60.84%
  Slathered

[paulb777]

@Corrob Awesome!

How do you process/visualize the data?

It's probably worth adding the instructions to the Coverage subsection to the Development section on the repo homepage and publicizing it to the team.

[Corrob]

Sorry about the delay, had some other issues pop up.

Slather can also output HTML files that allow you to dig into the coverage.

slather coverage --html --scheme AllUnitTests_iOS --configuration=Debug --workspace Firebase.xcworkspace/ Firebase.xcodeproj/

produces all of these HTML files: https://drive.google.com/open?id=1pIteb1sHho3SeaicbXmSLVcmjRgsxFG0. You can open index.html to start.

I'll send a PR with the necessary changes along with putting it in the README. After that, I'll try to hook it into Travis.

[maksymmalyhin]

@Corrob @paulb777 Here is the tool I mentioned - xcov.
It is not as popular as Slather, but it generated the output divided by modules without extra work.

Here is a simple version I used for testing (run from Example folder):
xcov --workspace Firebase.xcworkspace --scheme AllUnitTests_iOS --output_directory xcov_output

A sample output can be found here

[Corrob]

This is great, Maksym! I really appreciate your fix and input. I think we should go with xcov. I will update my PR.

[maksymmalyhin]

@Corrob I am happy to help!

[paulb777]

CocoaPods 1.9.0 adds pod lib lint support for collecting coverage. See https://github.com/CocoaPods/CocoaPods/pull/9302

[paulb777]

This will get easier after #2022 and we can build the frameworks dynamically since there have been at least historical data collection issues with static frameworks and coverage.

In the meantime, initial work could be done for GoogleUtilities.

[paulb777]

See https://github.com/CocoaPods/CocoaPods/issues/10034#issuecomment-689281086 for a start at adding via the CocoaPods test spec.

[ryanwilson]

Now that we have dynamic frameworks, this should be more straightforward.

[granluo]

xccov, introduced in WWDC19(video at around 0:50:52), could be another way to get the coverage data.
Slather is using profdata (testing binary generated by xcode) over app binary to calculate the coverage, while xccov is actually reading xcresult bundles generated by xcode.
I think xccov is preferred since it is identical to xcode result while slather needs additional calculations of profdata over app binaries.