Temurin-build: Windows builds look universally broken today (gradle/SSLException)

[sxa]

Looks to be failures on (probably) all windows builds last night:

Downloading https://services.gradle.org/distributions/gradle-5.2.1-bin.zip

Exception in thread "main" javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)

[sxa]

Above is showing on the softlayer build box that's active (sample log: https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk/job/jdk-windows-x64-hotspot/109/console). Azure machines seem to be giving a different error but still in the gradle area:

Downloading https://services.gradle.org/distributions/gradle-5.2.1-bin.zip
...................................................................................

Welcome to Gradle 5.2.1!

Here are the highlights of this release:
 - Define sets of dependencies that work together with Java Platform plugin
 - New C++ plugins with dependency management built-in
 - New C++ project types for gradle init
 - Service injection into plugins and project extensions

For more details see https://docs.gradle.org/5.2.1/release-notes.html

To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/5.2.1/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing
> Task :clean UP-TO-DATE
> Task :compileJava NO-SOURCE
> Task :compileGroovy
> Task :compileGroovy FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileGroovy'.
> java.lang.StackOverflowError (no error message)

e.g. https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk14u/job/jdk14u-windows-x64-openj9-windowsXL/75/console

Some Windows build machines are currently offline which was due to trying to force use of build-azure-win2012r2-x64-2 as much as possible to help reproduce a problem being seen on that system as I recall (Ref: https://github.com/AdoptOpenJDK/openjdk-build/issues/1741) (so the "offline-ness" is unrelated to this, although that does not preclude that the failures are machine specific.

[andrew-m-leonard]

I suspect it's to do with the version of java gradle is running, in the failing build:

Running gradle with C:\Program Files\Java\jdk8u144-b01

in the previous successful builds:

Running gradle with /cygdrive/c/openjdk/jdk11

[andrew-m-leonard]

Failed:
BUILD_CONFIG[USER_SUPPLIED_CONFIGURE_ARGS]=""
Successful:
BUILD_CONFIG[USER_SUPPLIED_CONFIGURE_ARGS]=" --disable-warnings-as-errors --with-freemarker-jar=/cygdrive/c/openjdk/freemarker.jar --with-openssl=/cygdrive/c/openjdk/OpenSSL-1.1.1g-x86_64-VS2017 --enable-openssl-bundling --enable-cuda --with-cuda=C:/PROGRA~1/NVIDIA~2/CUDA/v9.0 --with-toolchain-version=2017"

[andrew-m-leonard]

The node Configuration of the JDKnn_BOOT_DIR is not being set:
JDK10_BOOT_DIR is not set

[sxa]

@Willsparker Could the addition of additional boot JDKs have made a difference to this? I would have assumed that was just an unzip operation so would have no effect on anything else but it's not obvious what else may have impacted this

[Willsparker]

I would imagine so, actually - on the Softlayer machines, I renamed the jdks from C:\openjdk\jdkXX to C:\openjdk\jdk-XX without updating the Jenkins variables (JDKXX_BOOT_DIR) as I didn't realise I had to - They're all updated now. I renamed the JDKs as to keep them consistent with what the playbooks currently do, and what the Azure machines have on them too. Apologies

[Willsparker]

I also just noticed that the Azure machines don't have JDK-11 on them whatsoever... That wasn't due to what I did yesterday, but I've put it on both of them now, and checked those variables are pointing at the right place 👍

[andrew-m-leonard]

So what's really confused me is the Configure->Node->Environment Variables list are NOT actually Environment variables as we know it! but from what google tells me they are injected to the job environment somehow, but not exactly sure I understand how.... but anyhow they are not set in these Windows builds

[Willsparker]

https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-openj9/586/
From the looks of it, changing them to the correct path has fixed it

[sxa]

@andrew-m-leonard What makes you say they are not set in the Windows builds?

[sxa]

If the renamed boot JDK directories were only the SoftLayer machines presumably we still have another problem on the Azure ones to resolve?

[andrew-m-leonard]

Looking at the console log I could tell from the messages it wasn't picking them up, but it may have been because they were pointing to the wrong place...?
We will see tomorrow!

[sxa]

They should show up as environment variables (obviously it's possible that the scripts could have overridden it by the time you see the symptoms).

Since you have access you can always check the current values on any machine by using the Script Console for a machine - the first example on that screen shows how to print an environment variable.

[andrew-m-leonard]

So using Scripting Console does not show these variables as they are not set that way by Jenkins apparently, I found a google reference that they are specially "injected" into the groovy... they are not really Environment Variables as such!!

[Willsparker]

Checked the nightly builds- As you say @sxa , there's something still wrong with the Azure machines. It doesn't seem to like running Gradle with JDK11. Before the builds broke, it was using JDK8 to run Gradle (i.e. https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk14u/job/jdk14u-windows-x64-openj9/71/consoleFull ) - I would imagine this was because it didn't have JDK11 on there to begin with.
@andrew-m-leonard @sxa Should I just rename the C:\openjdk\jdk-11 folder to something else to force it to use JDK8 again. I assume that except Gradle, there's no requirement for it?

[andrew-m-leonard]

@Willsparker the jdk11 build job is succeeding now, it's the Signing job that is now failing...

[andrew-m-leonard]

find: 'basename' terminated by signal 13
find: 'basename' terminated by signal 13
find: 'basename' terminated by signal 13
Signing Windows release
Signing ./bin/fixpath.exe
SignTool Error: File not found: C:\Users\jenkins\windows.p12

WARNING: Failed to sign ./bin/fixpath.exe at 00:14:29: Possible timestamp server error - RC 0 ... Retrying in 10 seconds
SignTool Error: File not found: C:\Users\jenkins\windows.p12

[Willsparker]

I didn't mean the JDK11 build job specifically, I was referring to when JDK11 is being used to run Gradle (i.e. https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk14u/job/jdk14u-windows-x64-hotspot/74/console ) :-)
Seems to only be the azure machines that can't run Gradle with JDK11, the softlayer machines are okay

[Willsparker]

(Though it isn't JDK specific , same issue occurred in JDK11 too : https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-hotspot/580/console )

[Willsparker]

As an update:
The signing issues mentioned above has seemed to be fixed by renaming the C:\openjdk\windows to windows.p12 and then back to windows. Not sure what went on there.

The Azure JDK11 issues have been fixed by renaming the C:\openjdk\jdk-11 repo to C:\openjdk\jdk-11TEST so that the gradleJavaHome var is set to JDK8_BOOT_DIR (https://github.com/AdoptOpenJDK/openjdk-build/blob/1ce2c74aead570f934148b2d69c02f2bd28e069b/sbin/build.sh#L410)
Confirmed it's fixed by running https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x64-hotspot/581/console

I'll do the same on build-azure-win2012r2-x64-1 for when it's reconnected.

[Willsparker]

It seems the windows builds are all okay now- this can probably be closed now :-) @sxa , would you mind? I don't have the perms to close issues :/