Collect: Speed up the build timings of CircleCI jobs

[shobhitagarwal1612]

Jobs running on CircleCI:

• build_debug
• test_unit
• test_instrumented (only on master branch
• report_coverage (only on master branch)

There is still scope to speed up the time taken to run each job. Some of the ideas to try out are:

• Try out different parameters available in environment , _JAVA_OPTIONS in config.yml file.
• Cache the android dependencies for pmd, findbugs so that they are not downloaded during every build.
• Optimize the gradle tasks (for eg. check if the dependency testDebugUnitTest is required in jacocoTestReport, if not then remove it)
• Explore other ideas. Read this link as well and many others have posted blogs for the same as well

[andreimarcut]

@opendatakit-bot claim --force

[getodk-bot]

Welcome to Open Data Kit, @andreimarcut! We just sent you an invite to collaborate on this repository at https://github.com/opendatakit/collect/invitations. Please accept this invite in order to claim this issue and begin a fun and rewarding experience contributing to Open Data Kit!

Here are some tips to get you off to a good start:

• Please read the README.md and CONTRIBUTING.md in this repo. Those two documents have much of what you need to get started.
• Join the ODK developer Slack to get help, chat about this issue, and meet the other developers.
• Sign up and introduce yourself on the ODK community forum to meet the broader ODK community.

See you on the other side (that is, the pull request side)!

[andreimarcut]

The strategy I'd like to apply here is to first clean-start the build process (by setting up a private build environment) and then, moving forward, I would like to profile the build and rebuild times in various scenarios. Further on, we should identify actual performance gaps and improve where it's efficiently possible.

[yanokwa]

Hi @andreimarcut! This sounds like a good strategy. The caveat here is that the behavior of the build on CircleCI is going to be different than a local build env. But if you create a fork of Collect and work in a feature branch to optimize build_debug and test_unit, you should be able to get quite far. Have you worked with CircleCI before?

[andreimarcut]

I already forked the repo and set up a private build machine on circleci. Worked on CircleCI since version 1 and also recently adapted/enhanced some scripts to version 2 for other projects.
However, before attempting any optimizations here, I need some data/facts to help figure out and understand the cause and the improvement options..

[yanokwa]

Sounds like we are in great hands! If there is anything I can help with, please ask. And if you need real-time ish responses, most of us are in #collect-code channel on http://slack.opendatakit.org.

[andreimarcut]

I've experimented around 30 runs with different configs, starting from the description of this need.

I've successfully enabled external dependency caching for build_debug and test_unit jobs, which returns an average 10% gain in terms of build time

I've also incremented the machine image from API 26 to API 27, to avoid an unnecessary prerequisite which took from 10 to 20 seconds on the CI set-up

I didn't dive into the instrumentation task as it runs outside the CI env

Also I consider the coverage task impact irrelevant compared to the assembly jobs

When I added the most basic gradle runtime tweaks, I get random build failures. Specifically:
-Xmx 4096m results in out of memory on findbugs
-Dorg.gradle.parallel=true random failures on findbugs task

For now I consider tweaking the runtime depends on tuning the findbugs tasks, as otherwise it the whole build is at risk

Moving forward, other improvement ideas could focus on:

• removing/filtering-out non-sources (libs) from the code quality tasks and also from the code coverage report
• breaking down the project into a modular structure that in return could be partially cached
• enabling gradle build caching, see https://guides.gradle.org/using-build-cache/

[andreimarcut]

I opened a pull request for the current progress, I believe the project could benefit of these improvements while other ideas could emerge

Please check https://github.com/opendatakit/collect/pull/2247