Openj9: Add new Windows machines for pull requests and build pipelines

Created on 11 Apr 2018  路  43Comments  路  Source: eclipse/openj9

Four new Windows 2012R2 machines have been created and in need configuring with the ansible scripts from AdoptOpenJDK.

  1. oj9-win2012r2-1; 169.55.170.69
  2. oj9-win2012r2-2; 169.55.150.148
  3. oj9-win2012r2-3; 169.55.170.74
  4. oj9-win2012r2-4; 169.55.150.158

Administrator login passwords available under private communication.

Each one is 8 CPU, 8GB RAM, Windows 2012R2, 100Gb boot disk, 25 Gb data disk, 100 Mbs network link.

  • [x] PR for JenkinsFiles #1778
  • [x] Ansible machines (1/4 done)
  • [x] Connect to Jenkins (1/4 done)
  • [x] Setup Build jobs
  • [x] Setup test jobs
  • [x] Setup Pipeline jobs
  • [x] Setup PR jobs
  • [x] README
build high

All 43 comments

@sxa555 for installation and @AdamBrousseau for adding to jenkins and incorporating into build and pull requests.

Many thanks to IBM for providing Windows machines for OpenJ9's CI pipeline!

Memo to self after a couple of discussions this evening:

  • Windows OpenJ9 machines are set up manually based on the AdoptOpenJDK playbooks
  • There is no ansible tower (AWX) for OpenJ9
  • The jenkins slave cannot be started directly via JNLP since jenkins is behind a proxy

@DanHeidinga I can't seem to assign this work item to me (I'm the only person around and will attempt to begin attacking it tomorrow more - today has been an information consolidation phase) - can you either (1) Add me to whatever team lets me do stuff in this repo (preferred!) or (2) Assign this item to me

One of the annoyances here is that issues can only be assigned to openj9 committers. We've worked around this by having contributors just comment that they are working on a particular item

Oh hadn't realised that even you wouldn't be able to. Grrr ... Maybe I'll go through the signup process just so I can be assigned stuff. Or use it as an excuse not to work on issues here ;-)

In all seriousness I'm now working on the first of the machines - got part way through the install (manually for now) and I'll see if I can do anything more clever for the subsequent ones. It's the first time I've set up an OpenJ9 or AdoptOpenJDK windows box from scratch so I want to make sure I understand what's going on.

You can't just sign up to become an OpenJ9 committer. It requires submitting sufficient change to OpenJ9 to earn the trust of existing committers, and getting voted in https://www.eclipse.org/projects/handbook/#elections-committer

You can sign the ECA to become a contributor.

Windows machine oj9-win2012r2-1 has been set up manually as per the current adoptopenjdk ansible playbook (excluding NTP as yet, and including the SDK)
It does NOT include any of the additional stuff being proposed in https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/300
@AdamBrousseau if you want to try chucking it in jenkins we can see if it works :-) (ping me for the jenkins user password)

Got compiles working
Had to change a few things on the machine

  • Install openssh through Cygwin installer (reference)
  • Uninstall Git for Windows
  • Install Git through Cygwin installer
  • Add Shortname for "Program Files (x86)" directory (reference)

    • fsutil 8dot3name query # returned 2

    • fsutil behavior set disable8dot3 0

    • fsutil file setshortname "Program Files (x86)" PROGRA~2

    • dir /X # confirm shortname created

    • fsutil behavior set disable8dot3 2

  • Install x86_64-w64-mingw32-g++ through Cygwin installer

Current issue is trying to compile the test material (link)

00:04:25.441 clean:
00:04:25.441    [delete] Deleting directory C:\cygwin64\home\jenkins\workspace\Test-Extended-JDK10-win_x86-64_cmprssptrs\openj9\test\functional\VM_Test\bin
00:04:25.441 
00:04:25.441 BUILD FAILED
00:04:25.441 C:\cygwin64\home\jenkins\workspace\Test-Extended-JDK10-win_x86-64_cmprssptrs\openj9\test\TestConfig\scripts\build_test.xml:71: The following error occurred while executing this line:
00:04:25.441 C:\cygwin64\home\jenkins\workspace\Test-Extended-JDK10-win_x86-64_cmprssptrs\openj9\test\functional\build.xml:62: The following error occurred while executing this line:
00:04:25.441 C:\cygwin64\home\jenkins\workspace\Test-Extended-JDK10-win_x86-64_cmprssptrs\openj9\test\functional\VM_Test\build.xml:139: The following error occurred while executing this line:
00:04:25.441 C:\cygwin64\home\jenkins\workspace\Test-Extended-JDK10-win_x86-64_cmprssptrs\openj9\test\functional\VM_Test\build.xml:134: java.nio.file.InvalidPathException: Illegal char <*> at index 113: C:\cygwin64\home\jenkins\workspace\Test-Extended-JDK10-win_x86-64_cmprssptrs\openj9\test\functional\VM_Test\data\*.class

Edit: Also had to install the Cygpath Plugin for Jenkins

Why did you replace the native git client? To my knowledge we haven't used the cygwin one anywhere else so interested to know why we need to use git differently on that machine and I don't want to be running different configurations if at all possible since it could lead to "surprises" when the openj9 stuff is built on the adoptopenjdk build farm.

I see your concern, and generally agree.

All the Windows machines we've ran builds on up until now have been connected to Jenkins using the JNLP method. Which gives Jenkins a "Windows" environment to run jobs. The machines at ci.eclipse have to be connected via ssh which means getting a "Linux" Cygwin environment.

The part that fails is the git -C command to lookup the OpenJ9/OMR SHAs. It fails because the path passed to git is Linux style Cygwin path and Windows Git cannot understand it.

00:06:28.498 + make all
00:06:28.498 fatal: Cannot change to '/cygdrive/c/Users/jenkins/workspace/Build-JDK10-win_x86-64_cmprssptrs/openj9': No such file or directory
00:06:28.498 OpenJ9.gmk:31: *** Could not determine OpenJ9 SHA.  Stop.

It does not make a difference if the Jenkins root is C:\Users\Jenkins or C:\cygwin64\home\jenkins

Uninstalling Windows Git and Installing Git through Cygwin solves the issue.

Edit: I suppose if both Git's were installed then the Cygwin path would find the Cygwin Git first and it would work. But I don't see any value in having the Windows Git installed if we won't be using it.

Note the OpenJ9 build instructions for Windows [1] do not mention installing a Windows git client, they use cygwin.

[1] https://github.com/eclipse/openj9/blob/master/buildenv/Build_Instructions_V8.md

@AdamBrousseau Can you clarify what you're running that's making that error show up. I believe the code is the same for JDK8 but we don't get that error running OpenJ9 JDK8 builds (OpenJ9 JDK9 hasn't run on Windows for a few weeks, but was clean the last time we tried)

Just had a look on one of our existing build machines and git is installed within cygwin there too.

My test failure is resolved by #1803. @sxa555 Can you please go ahead and setup the remaining 3 machines.

9/10 compile fine but we have an issue with JDK8 during configure

checking if closed source is suppressed (openjdk-only)... no
checking which variant of the JDK to build... normal
checking which interpreter of the JVM to build... template
checking which variants of the JVM to build... server
checking which debug level to use... release
./common/autoconf/../../jdk/make/closed/autoconf/generated-configure.sh: line 8280: /cygdrive/c/cygwin64
/home/jenkins/workspace/Build-JDK8-win_x86-64_cmprssptrs/jdk/make/closed/autoconf/openj9ext-version-numbers: No such file or directory
configure exiting with result code 1

I don't understand because that file is clearly there

jenkins@oj9-win2012r2-1 ~/workspace/Build-JDK8-win_x86-64_cmprssptrs
$ ls -al /home/jenkins/workspace/Build-JDK8-win_x86-64_cmprssptrs/jdk/make/closed/autoconf/openj9ext-version-numbers
-rw-r--r-- 1 jenkins None 992 Apr 26 13:58 /home/jenkins/workspace/Build-JDK8-win_x86-64_cmprssptrs/jdk/make/closed/autoconf/openj9ext-version-numbers

generated-configure.sh: line 8280

 # Source the closed version numbers
  . $SRC_ROOT/jdk/make/closed/autoconf/openj9ext-version-numbers

What if you run the configure step manually, on the machine? "." is the same as using "source", maybe cygwin doesn't like the use of ".", try swapping.

Windows machines that will be used to build Java 8 will also need the addition proposed in https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/354 (to register 'msdia*.dll' that are part of the DIA SDK included in VS2010).

Found the Issue with the PATH problem on JDK8.
OpenJDK has a fix, we haven't pulled it in yet
ibmruntimes/openj9-openjdk-jdk8#78

It turns out that builds of all versions of Java will require that machines are updated with the new steps added by https://github.com/AdoptOpenJDK/openjdk-infrastructure/pull/354. For people doing it manually, I recommend omitting the /s option for clear indication of whether it worked or not.

[edit: the /s option was removed from the instructions]

Next issue:

[C:\cygwin64\home\jenkins\workspace\PullRequest-Compile-JDK8-win_x86-64_cmprssptrs-OpenJDK8] Running shell script
+ make all
Building OpenJDK for target 'all' in configuration 'windows-x86_64-normal-server-release'

## Starting langtools
Compiling 2 files for BUILD_TOOLS
Could not open temporary file for writing! c:/cygwin/tmp\atfile_a10924
make[1]: *** No rule to make target 'all', needed by 'default'.  Stop.
make: *** [/home/jenkins/workspace/PullRequest-Compile-JDK8-win_x86-64_cmprssptrs-OpenJDK8//make/Main.gmk:87: langtools-only] Error 2

Not sure why its trying to write to c/cygwin/ without the 64

Does that machine define TMPDIR=C:/cygwin/tmp? That would explain this.

I see Windows system variables TEMP and TMP defined as C:\Windows\TEMP and in Cygwin shell I see TMP and TEMP defines as /tmp

Missing VS2010 SP1 + security update KB2565057 from 169.55.170.69 (oj9-win2012r2-1 ).

Short paths are still not correctly set, missing C:\Program Files PROGRA~1:

5/14/2018  02:38 PM    <DIR>                       Program Files
5/14/2018  02:44 PM    <DIR>          PROGRA~2     Program Files (x86)
5/14/2018  12:09 PM    <DIR>          STRAWB~1     Strawberry
3/22/2018  10:58 AM    <DIR>                       sxs
c:\>fsutil file setshortname "C:\Program Files" PROGRA~1
Error:  Access is denied.

Work-around for this short path:

mklink /D "C:\openjdk\windbgtools\" "C:\Program Files\Debugging Tools for Windows (x64)\"
setx INCLUDE "C:\openjdk\windbgtools\sdk\inc" /M
setx LIB  "C:\openjdk\windbgtools\sdk\lib" /M

OpenJ9 JDK10 builds successfully locally:

jenkins@oj9-win2012r2-2 ~/openj9-openjdk-jdk10
$ build/windows-x86_64-normal-server-release/images/jdk/bin/java -version
openjdk version "10.0.1-internal" 2018-04-17
OpenJDK Runtime Environment (build 10.0.1-internal+0-adhoc.jenkins.openj9-openjdk-jdk10)
Eclipse OpenJ9 VM (build master-59b32449, JRE 10 Windows Server 2012 R2 amd64-64-Bit Compressed References 20180514_000000 (JIT enabled, AOT enabled)
OpenJ9   - 59b32449
OMR      - 31420a86
JCL      - 303bc793d0 based on jdk-10.0.1+10)

Windows JDK8 build failure:

11:21:55 ## Starting langtools
11:21:56 Compiling 2 files for BUILD_TOOLS
11:21:56 Could not open temporary file for writing! c:/cygwin/tmp\atfile_a03868
11:21:56 make[1]: *** No rule to make target 'all', needed by 'default'.  Stop.
11:21:56 make: *** [/cygdrive/c/Users/jenkins/workspace/vsebe_Build-JDK8-win_x86-64_cmprssptrs//make/Main.gmk:87: langtools-only] Error 2

Some of the environment variables are not inherited in cygwin over ssh. Cygwin overrides the windows TMP and TEMP environement variables in /etc/profile: TMP = /tmp, TEMP=/tmp. These are not inherited.
fixpath.h sets TMP = "c:/cygwin/tmp"

tmpdir = getenv("TMP");
  if (tmpdir == NULL) {
    tmpdir = "c:/cygwin/tmp";
  }

This is is cygwin 32 bits path, should have been c:/cygwin64/tmp. It is fixed in OpenJDK9:

  tmpdir = getenv("TEMP");
  if (tmpdir == NULL) {
#if _WIN64
    tmpdir = "c:/cygwin64/tmp";
#else
    tmpdir = "c:/cygwin/tmp";
#endif

Fix available with https://github.com/ibmruntimes/openj9-openjdk-jdk8/pull/87

Work-around for JDK8, set TMP environment variable in the Jenkins node configuration.

12:34:27 + build/windows-x86_64-normal-server-release/images/j2sdk-image/bin/java -version
12:34:27 openjdk version "1.8.0_172-internal"
12:34:27 OpenJDK Runtime Environment (build 1.8.0_172-internal-jenkins_2018_05_18_10_59-b00)
12:34:27 Eclipse OpenJ9 VM (build master-d60a6052, JRE 1.8.0 Windows Server 2012 R2 amd64-64-Bit Compressed References 20180518_5 (JIT enabled, AOT enabled)
12:34:27 OpenJ9   - d60a6052
12:34:27 OMR      - 15290506
12:34:27 JCL      - 9b8da53238 based on jdk8u172-b11)

@vsebe can we not just backport that fix to jdk8 rather then having a special case setup on every machine config?

@AdamBrousseau - I will, the workaround is until the fix makes it into the openj9-openjdk-openjdk8 repo.

The best answer is to define TEMP suitably so as not to depend on the hard-coded path in the source code. It will break trying to build a 32-bit JVM on a machine with only C:/cygwin64 present.

These machines are ready. The OpenJ9 JDK 8, 9 and 10 pipelines pass.

Setting the TMP as environment variable in the Jenkins node configuration ensures the OpenJ9 JDK8 to build successfully (no need to wait on https://github.com/ibmruntimes/openj9-openjdk-jdk8/pull/87).

@vsebe can we delete the TMP/TEMP variables in the machine configs now that your change is merged?
https://ci.eclipse.org/openj9/computer/win2012r2-x86-1/configure

@AdamBrousseau - yes

Done.

Currently investigating the problem with JDK9 PR builds

OpenJ9 compile complete
Creating support/modules_libs/java.base/*/jvm.dll from J9 sources
Copying j9ddr.dat
make[4]: Entering directory '/cygdrive/c/Users/jenkins/workspace/PullRequest-Sanity-JDK9-win_x86-64_cmprssptrs-OpenJ9'
make[4]: Leaving directory '/cygdrive/c/Users/jenkins/workspace/PullRequest-Sanity-JDK9-win_x86-64_cmprssptrs-OpenJ9'

ERROR: Build failed for target 'all' in configuration 'windows-x86_64-normal-server-release' (exit code 2) 

=== Output from failing command(s) repeated here ===
* For target buildtools_corba_tools_classes__the.BUILD_TOOLS_CORBA_batch:
An exception has occurred in the compiler (9.0.4-internal). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.lang.NoClassDefFoundError: com/sun/tools/javac/processing/JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors.iterator(JavacProcessingEnvironment.java:814)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.atLeastOneProcessor(JavacProcessingEnvironment.java:615)
    at com.sun.tools.javac.main.JavaCompiler.initProcessAnnotations(JavaCompiler.java:1124)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:908)
    at com.sun.tools.javac.main.Main.compile(Main.java:302)
    at com.sun.tools.sjavac.comp.SjavacImpl.compile(SjavacImpl.java:117)
    at com.sun.tools.sjavac.comp.PooledSjavac.lambda$compile$0(PooledSjavac.java:63)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   ... (rest of output omitted)
* For target buildtools_idlj_classes__the.BUILD_IDLJ_batch:
An exception has occurred in the compiler (9.0.4-internal). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.lang.NoClassDefFoundError: com/sun/tools/javac/processing/JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors.iterator(JavacProcessingEnvironment.java:814)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.atLeastOneProcessor(JavacProcessingEnvironment.java:615)
    at com.sun.tools.javac.main.JavaCompiler.initProcessAnnotations(JavaCompiler.java:1124)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:908)
    at com.sun.tools.javac.main.Main.compile(Main.java:302)
    at com.sun.tools.sjavac.comp.SjavacImpl.compile(SjavacImpl.java:117)
    at com.sun.tools.sjavac.comp.PooledSjavac.lambda$compile$0(PooledSjavac.java:63)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   ... (rest of output omitted)
* For target buildtools_jdk_tools_classes__the.BUILD_TOOLS_JDK_batch:
An exception has occurred in the compiler (9.0.4-internal). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.lang.NoClassDefFoundError: com/sun/tools/javac/processing/JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors.iterator(JavacProcessingEnvironment.java:814)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.atLeastOneProcessor(JavacProcessingEnvironment.java:615)
    at com.sun.tools.javac.main.JavaCompiler.initProcessAnnotations(JavaCompiler.java:1124)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:908)
    at com.sun.tools.javac.main.Main.compile(Main.java:302)
    at com.sun.tools.sjavac.comp.SjavacImpl.compile(SjavacImpl.java:117)
    at com.sun.tools.sjavac.comp.PooledSjavac.lambda$compile$0(PooledSjavac.java:63)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   ... (rest of output omitted)
* For target buildtools_override_modules_jdk.rmic__the.BUILD_INTERIM_RMIC_batch:
An exception has occurred in the compiler (9.0.4-internal). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.lang.NoClassDefFoundError: com/sun/tools/javac/processing/JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors.iterator(JavacProcessingEnvironment.java:814)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.atLeastOneProcessor(JavacProcessingEnvironment.java:615)
    at com.sun.tools.javac.main.JavaCompiler.initProcessAnnotations(JavaCompiler.java:1124)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:908)
    at com.sun.tools.javac.main.Main.compile(Main.java:302)
    at com.sun.tools.sjavac.comp.SjavacImpl.compile(SjavacImpl.java:117)
    at com.sun.tools.sjavac.comp.PooledSjavac.lambda$compile$0(PooledSjavac.java:63)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   ... (rest of output omitted)
* For target support_jrtfs_classes__the.BUILD_JRTFS_batch:
An exception has occurred in the compiler (9.0.4-internal). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.lang.NoClassDefFoundError: com/sun/tools/javac/processing/JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors.iterator(JavacProcessingEnvironment.java:814)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.atLeastOneProcessor(JavacProcessingEnvironment.java:615)
    at com.sun.tools.javac.main.JavaCompiler.initProcessAnnotations(JavaCompiler.java:1124)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:908)
    at com.sun.tools.javac.main.Main.compile(Main.java:302)
    at com.sun.tools.sjavac.comp.SjavacImpl.compile(SjavacImpl.java:117)
    at com.sun.tools.sjavac.comp.PooledSjavac.lambda$compile$0(PooledSjavac.java:63)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   ... (rest of output omitted)

* All command lines available in /cygdrive/c/Users/jenkins/workspace/PullRequest-Sanity-JDK9-win_x86-64_cmprssptrs-OpenJ9/build/windows-x86_64-normal-server-release/make-support/failure-logs.
=== End of repeated output ===

As @vsebe suggested, the issue with the PR JDK9 builds is the filepath is too long. Compiling by hand in the PullRequest folder produces the error, renaming the directory to something shorter (PullRequest-adam) resolves the compile error. Not sure if its the entire path that is too long or just the folder PullRequest-Sanity-JDK9-win_x86-64_cmprssptrs-OpenJ9. We've been trying to look through the mercurial repos to see if we can find the change in 8 or 10 that resolves the issue.

I suggest OpenJ9 turn off all JDK9 builds #2020

Since #2020, I will leave win 9 sanity/ext fully disabled (compile still enabled) and consider this closed.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

M-Davies picture M-Davies  路  76Comments

pshipton picture pshipton  路  59Comments

M-Davies picture M-Davies  路  52Comments

jsimomaa picture jsimomaa  路  109Comments

AlenBadel picture AlenBadel  路  106Comments