Java: Speed up test suite(s)

[jtigger]

The CI build is approaching 20 minutes. The long pole on the build is (understandably) the journey suite. However, the exercises are guaranteed to be independent: the test suites can be executed in parallel without fear of test pollution. The situation is ripe for parallel execute of tests.

Please update the CI build to minimize both the duration of the total test run _and_ minimize additional added complexity (i.e. readability of the build scripts themselves is paramount).

[jtigger]

As a part of this please include the concerns surfaced in #326.

[Smarticles101]

Is there some way of only running tests on the modified exercise(s)? This would significantly improve the build time as usually only simple things are changed in one or two exercises

[stkent]

Currently the build is executing using Gradle 2.2.1. We should investigate upgrading to 3.5 (or whatever the latest stable version is).

[stkent]

Currently the build is executing using Gradle 2.2.1. We should investigate upgrading to 3.5 (or whatever the latest stable version is).

See https://github.com/exercism/xjava/pull/599

[stkent]

See https://github.com/exercism/xjava/pull/600 for partial parallelization of the build (unit tests).

[stkent]

Things I think we could still investigate here:

• parallelizing journey test execution;
• running unit tests and journey tests in parallel on separate build nodes (I think this is available to Travis OSS repos?);
• installing gems in parallel.

[Smarticles101]

I'm looking into the things you mentioned @stkent :)

[stkent]

I had a play with the journey tests this evening, attempting to run them in parallel. However, I think I hit a wall because they don't share a common root Gradle project. Here's the patch I tried to apply:

diff --git a/bin/journey-test.sh b/bin/journey-test.sh
index b4223f7..6b34670 100755
--- a/bin/journey-test.sh
+++ b/bin/journey-test.sh
@@ -224,35 +224,36 @@ solve_all_exercises() {
   local track_root=$( pwd )
   local exercism_cli="./exercism --config ${exercism_configfile}"
   local exercises=`cat config.json | jq '.exercises[].slug + " "' --join-output`
+  local exercises_array=( $exercises )
   local total_exercises=`cat config.json | jq '.exercises | length'`
-  local current_exercise_number=1
   local tempfile="${TMPDIR:-/tmp}/journey-test.sh-unignore_all_tests.txt"

   pushd ${exercism_exercises_dir}
-  for exercise in $exercises; do
-    echo -e "\n\n"
-    echo "=================================================="
-    echo "${current_exercise_number} of ${total_exercises} -- ${exercise}"
-    echo "=================================================="

+  # Fetch all exercises and copy example implementations into main source set directory.
+  for exercise in $exercises; do
     ${exercism_cli} fetch ${TRACK} $exercise
     cp -R -H ${track_root}/exercises/${exercise}/src/example/${TRACK}/* ${exercism_exercises_dir}/${TRACK}/${exercise}/src/main/${TRACK}/
+  done
+
+  # Check that all exercise tests compile as-is.
+  "$EXECPATH"/gradlew "${exercises_array[@]/%/:compileTestJava}"

+  for exercise in $exercises; do
+    # Remove @Ignore annotations from all exercise tests serially
     pushd ${exercism_exercises_dir}/${TRACK}/${exercise}
-    # Check that tests compile before we strip @Ignore annotations
-    "$EXECPATH"/gradlew compileTestJava
-    # Ensure we run all the tests (as delivered, all but the first is @Ignore'd)
     for testfile in `find . -name "*Test.${TRACK_SRC_EXT}"`; do
       # Strip @Ignore annotations to ensure we run the tests (as delivered, all but the first is @Ignore'd).
       # Note that unit-test.sh also strips @Ignore annotations via the Gradle task copyTestsFilteringIgnores.
       # The stripping implementations here and in copyTestsFilteringIgnores should be kept consistent.
       sed 's/@Ignore\(\(.*\)\)\{0,1\}//' ${testfile} > "${tempfile}" && mv "${tempfile}" "${testfile}"
     done
-    "$EXECPATH"/gradlew test
     popd
-
-    current_exercise_number=$((current_exercise_number + 1))
   done
+
+  # Check that all exercise example implementations pass the associated tests.
+  "$EXECPATH"/gradlew "${exercises_array[@]/%/:test}"
+
   popd
 }

Resulting error: https://travis-ci.org/exercism/java/jobs/252211424

I'd rather not go down the route of inserting a fake root project unless we absolutely need to in the future. Right now, though, I think we're in a good place with the current CI configuration and can close this issue. Thoughts, @exercism/java?

[stkent]

Ah, #326 is still a thing we need to fix.

[kytrinyx]

Is there some way of only running tests on the modified exercise(s)?

@Smarticles101 That's a great idea.

I don't know if travis has a ref of master available. If so we could run:

git diff-tree -r --name-only HEAD master

To get a list of changed files. If none of the files changed are in the exercises directory (or the config.json?) then we wouldn't have to run the journey tests at all.

If we don't have access to the master ref, then we could use the List pull request files endpoint on the GitHub API to get the list of changed paths, and only run the tests if files under ./exercises/ have changed. Then further, figure out which exercises changed and run those.

[stkent]

@kytrinyx this SO post seems to suggest that master should be accessible in a Travis PR build 😄

[Smarticles101]

Awesome to see that this should work. So the idea now is that if no exercises have been modified or added, don't run the journey tests. If some have been modified, only test against those.

Another consideration is to only install ruby and compile jq before the journey tests, because that is the only spot they are used.

[Smarticles101]

326 still needs to be solved, re-opening this

[Smarticles101]

Issues with #734 so far:

1. From #750 travis build #1666, gradle build in journey-test failed, but journey tests completed with exit code 0

[Smarticles101]

The above issue is related to #326. I'll fix it when I find a good solution for error handling.

Information about error handling in bash:

• http://fvue.nl/wiki/Bash:_Error_handling
• http://mywiki.wooledge.org/BashFAQ/105

[Smarticles101]

This has been a lot of fun to work on and get a bit better at bash with. The last way I think we could improve is with being able to fail the build as soon as one of the two jobs fail. Unfortunately that is not possible as of yet (travis-ci#8296). I'm going to close this for now.

[kytrinyx]

Good call—and really great work on this one. It's been fun seeing it progress!