Temurin-build: build Gradle download & compile fails intermittently

[andrew-m-leonard]

https://ci.adoptopenjdk.net/view/Failed%20Builds/job/build-scripts/job/jobs/job/jdk15u/job/jdk15u-windows-x64-hotspot/64/console

21:28:41  Running gradle with /cygdrive/c/openjdk/jdk-11 at /cygdrive/c/workspace/openjdk-build/workspace/.gradle
21:28:43  Downloading https://services.gradle.org/distributions/gradle-6.5-bin.zip
21:29:02  .................................................................................................
21:29:04  
21:29:04  Welcome to Gradle 6.5!
21:29:04  
21:29:04  Here are the highlights of this release:
21:29:04   - Experimental file-system watching
21:29:04   - Improved version ordering
21:29:04   - New samples
21:29:04  
21:29:04  For more details see https://docs.gradle.org/6.5/release-notes.html
21:29:04  
21:29:04  To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/6.5/userguide/gradle_daemon.html.
21:29:23  Daemon will be stopped at the end of the build stopping after processing
21:29:34  > Task :clean UP-TO-DATE
21:29:34  > Task :compileJava NO-SOURCE
21:30:05  > Task :compileGroovy
21:33:51  > Task :compileGroovy FAILED
21:33:51  
21:33:51  FAILURE: Build failed with an exception.
21:33:51  
21:33:51  * What went wrong:
21:33:51  Execution failed for task ':compileGroovy'.
21:33:51  > java.lang.StackOverflowError (no error message)

[M-Davies]

Looks like the next run went through ok https://ci.adoptopenjdk.net/view/Failed%20Builds/job/build-scripts/job/jobs/job/jdk15u/job/jdk15u-windows-x64-hotspot/65/console. It was likely an intermittent network issue (we've seen this before with gradle downloading libraries). Have you seen this anywhere else @andrew-m-leonard ?

[andrew-m-leonard]

As you say, we've seen this intermittently.
I think as it does occur frequently enough, maybe we could cache the gradle project somewhere rather than download each time?

[M-Davies]

We could probably put it into ansible with the playbooks? Tagging @Willsparker in for the infra opinion on that

[Willsparker]

Could someone point me in the direction of where the Gradle stuff is downloaded in the build scripts?

[M-Davies]

@Willsparker https://github.com/AdoptOpenJDK/openjdk-build/blob/master/pipelines/build.gradle and https://github.com/AdoptOpenJDK/openjdk-build/blob/master/pipelines/gradlew do most of the work but I'm not very familiar with Gradle so I can't explain how they're downloaded or where the tasks come from

[andrew-m-leonard]

@Willsparker https://github.com/AdoptOpenJDK/openjdk-build/blob/2f1b2b8894682cce4eecd8fd34f460f564818a67/sbin/build.sh#L472

[andrew-m-leonard]

Essentially it is just building adopt-shared-lib.jar each time. Could we only build it if it is not already available locally?

[aahlenst]

Gradle automatically caches previously downloaded binaries in ~/.gradle/wrapper/dists. They should be reused on successive builds. Is something cleaning that out?

[Willsparker]

I noticed that that Gradle build is happening on one of the new IBMCloud build machines - So there's a good chance that the reason Gradle doesn't reuse the previously downloaded binaries is because there isn't one in there. A quick search of the repo with ~/.gradle/wrapper/dists doesn't show any code that seems to clear it out.

The function that @andrew-m-leonard linked should already handle network issues with downloading Gradle (i.e. it retries 3 times). If it's just building adopt-shared-lib.jar each time, I don't see why this couldn't be done in the Infra playbooks, stored in a common place and copied for each build. It'd be fairly easy to just put something like this at the beginning of the buildSharedLibs function.

if [[ ! -r /build_libs/adopt-shared-lib.jar ]]; then
  ...
else
  cp /build_libs/adopt-shared-lib.jar <Wherever it needs to be>
fi

Theoretically that means we could cut out the Gradle stuff if the library is found

[aahlenst]

Instead of using the Gradle wrapper, we could pre-install Gradle with the playbooks and put it on the path like in the good old days. We could then also remove the code that retries the download. Instead of gradlew we use gradle.

The advantage of building the JAR during the build is we don't have problems with old versions and always get the latest version compiled from source (= easier to debug/reason about).

[andrew-m-leonard]

Looking at the above gradle error it is actually a:

21:33:51  > java.lang.StackOverflowError (no error message)

This is highly likely because the java stack used in gradlew is too small:
https://github.com/AdoptOpenJDK/openjdk-build/blob/5f877ec1540fa0b9bd57f2134757fbb478c2b19c/pipelines/gradlew#L31

This ought to be something like -Xmx512m instead.

[andrew-m-leonard]

@M-Davies @Willsparker I'll create a PR to fix the java stack size and test it out...

[andrew-m-leonard]

Increase gradlew java stack : https://github.com/AdoptOpenJDK/openjdk-build/pull/2158

[sxa]

To be clear, my preference would be to remove the gradle stuff completely as it didn't cause any side effects for Solaris and nobody can explain why it's needed ;-)