Openj9: xLinux builds of openjdk8-openj9 are failing due to openj9 build failure

[ben-walsh]

See https://github.com/ibmruntimes/openj9-openjdk-jdk8/issues/44.
Specifically https://github.com/ibmruntimes/openj9-openjdk-jdk8/issues/44#issuecomment-365925203.

NB : Builds have been failing like this since 6th February.

[keithc-ca]

This is a long-standing issue with JIT when compiled with a pedantic compiler. It normally doesn't use the 'warnings-are-errors' option.

[sxa]

The build machine that is being used has gcc5 on it as well as 4.8. It would appear that there is an issue in the build process whereby parts of the build process will pick up the CC/CXX environment variables to override the defaults but some parts won't. This sounds like something that should be fixed as the Tools summary section of the log suggests it's using the gcc-4.8 when the variable is set, but later on in the OMR configure step it's still picking up the default gcc

[pshipton]

Based on the last comment, I'm going to re-tag this as a build bug.

OpenJ9 supports gcc 4.8 atm. 4.8.5 is the version used for ci testing and in the OpenJ9 nightly builds. Other 4.x versions may work but since they are not regularly tested, problems could be introduced.

[sxa]

Reference to the CI: https://ci.adoptopenjdk.net/job/openjdk10_build_x86-64_linux/

[andrew-m-leonard]

@pshipton Peter this will block releasing JDK8, since we cannot build it at AdoptOpenJDK... Adopt are trying to pass the necessary autoconf "configure" env options through to openj9/omr, but they seem to be being ignored... can we look at that issue to workaround the lack of 5 support?
We probably ought to tag this as JDK8 as well?

[pshipton]

It doesn't seem the problem is specific to jdk8 even though it affects jdk8. We can tag it as high priority.

[DanHeidinga]

@sxa555 Can I get access to the Adopt machine to do some manual testing? Is there a procedure at Adopt I can follow to get a temporary login / password?

[smlambert]

@DanHeidinga - you raise an issue at the openjdk-infrastructure repo to request temporary access (example: https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/203)

[DanHeidinga]

Thanks @smlambert! I've requested access in https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/231

[DanHeidinga]

Here's a Java 8 build failure with this issue: https://ci.adoptopenjdk.net/job/openjdk8_openj9_build_x86-64_linux/83/console

[DanHeidinga]

Current theory - we're "losing" the correct compiler setting when we hit this makefile https://github.com/eclipse/openj9/blob/26f8c7edb1fe37253d001243c0ce14473334d07f/runtime/compiler/trj9/Makefile

[JamesKingdon]

That seems likely true, but https://github.com/ibmruntimes/openj9-openjdk-jdk8/issues/44 showed the problem in jswalk.c which I think is compiled from vm/codert_vm/ suggesting there may be more than one source of the problem.

[DanHeidinga]

Have access to a ubuntu16 machine with both gcc 5 and 4.8.

Adopt runs with:

 export CC=gcc-4.8
 export CC_PATH=gcc-4.8
 export CXX=g++-4.8

Setting the same and attempting the build to reproduce the issue locally.

[ymanton]

That seems likely true, but ibmruntimes/openj9-openjdk-jdk8#44 showed the problem in jswalk.c which I think is compiled from vm/codert_vm/ suggesting there may be more than one source of the problem.

IIRC the files in codert_vm/ are compiled as part of the JIT, despite their location, so I think it's just a single problem.

Anyhow, the JIT makefiles don't respect CC or CXX, instead they look at the (AFAIK) non-standard CC_PATH and CXX_PATH. Adopt seems to set CC_PATH so I would simply try setting CXX_PATH as well and see if that gets you to a complete build, but it seems to me the JIT makefiles should just look at CC and CXX rather than the _PATH variations.

[DanHeidinga]

I've set CXX_PATH & CX_PATH to g++-4.8, in addition to the exports used in https://github.com/eclipse/openj9/issues/1180#issuecomment-367803319.

No luck - build still fails in the same way.

[DanHeidinga]

Hacked all the CXX_PATH / CC_PATH setting lines to force them to g++-4.8 / gcc-4.8. Still failing

[DanHeidinga]

export CC=gcc-4.8
export CC_PATH=gcc-4.8
export CXX=g++-4.8
export CXX_PATH=gcc-4.8

and patched the uma spec file as:

diff --git a/buildspecs/linux_x86-64_cmprssptrs.spec b/buildspecs/linux_x86-64_cmprssptrs.spec
index 9ffddd14..63d513c4 100644
--- a/buildspecs/linux_x86-64_cmprssptrs.spec
+++ b/buildspecs/linux_x86-64_cmprssptrs.spec
@@ -91,9 +91,9 @@
                <property name="svn_stream" value=""/>
                <property name="uma_make_cmd_ar" value="ar"/>
                <property name="uma_make_cmd_as" value="as"/>
-               <property name="uma_make_cmd_cc" value="cc"/>
-               <property name="uma_make_cmd_cpp" value="cpp -E -P"/>
-               <property name="uma_make_cmd_cxx" value="c++"/>
+               <property name="uma_make_cmd_cc" value="gcc-4.8"/>
+               <property name="uma_make_cmd_cpp" value="g++-4.8 -E -P"/>
+               <property name="uma_make_cmd_cxx" value="g++-4.8)"/>
                <property name="uma_make_cmd_cxx_dll_ld" value="$(CC)"/>
                <property name="uma_make_cmd_cxx_exe_ld" value="$(CC)"/>
                <property name="uma_make_cmd_dll_ld" value="$(CC)"/>

but that leads to another problem:

Creating file oti/jilvalues.m4
/bin/sh: 1: Syntax error: ")" unexpected
/bin/sh: 1: Syntax error: ")" unexpected
../makelib/targets.mk:305: recipe for target 'ObjectHash.o' failed
gmake[3]: *** [ObjectHash.o] Error 2
gmake[3]: *** Waiting for unfinished jobs....
../makelib/targets.mk:305: recipe for target 'VMThreadFlags.o' failed

[keithc-ca]

You probably need to omit the ')' in

     <property name="uma_make_cmd_cxx" value="g++-4.8)"/>

[DanHeidinga]

Thanks @keithc-ca! I've been looking for where I messed the formatting up. A second set of eyes is big help.

[DanHeidinga]

That works:

/tmp/danh/openj9-openjdk-jdk8$ ./build/linux-x86_64-normal-server-release/images/j2sdk-image/bin/java -version
openjdk version "1.8.0_162-internal"
OpenJDK Runtime Environment (build 1.8.0_162-internal-j9build_2018_02_25_05_28-b00)
Eclipse OpenJ9 VM (build master-ebc19708, JRE 1.8.0 Linux amd64-64 Compressed References 20180226_000000 (JIT enabled, AOT enabled)
OpenJ9   - ebc19708
OMR      - f698070a
JCL      - e2cb8c715c based on jdk8u162-b12)
j9build@ub16hcxrt1:/tmp/danh/openj9-openjdk-jdk8$ 

[ymanton]

Seems @JamesKingdon was right, there are multiple problems. When I point /usr/bin/{gcc,g++} to /bin/false or similar and set CC and CXX to point to a working GCC 4.8 multiple things fail because they're blindly accessing cc or gcc or g++. The JIT fails as expected, the VM fails as above, and OMR fails (partly because OpenJ9 seems to hardcode setting OMR_CROSS_CONFIGURE=yes which affects CC and CXX are treated).

I actually have a need to swap compilers for other reasons and being able to override quickly without having to change things globally so I can take a look at this @DanHeidinga unless you have it sorted.

[DanHeidinga]

@ymanton Using the changes in https://github.com/eclipse/openj9/issues/1180#issuecomment-368284923 I was able to successfully build after removing the extra ) but I don't think that's the right patch yet. I tried to change it to this:

diff --git a/runtime/makelib/mkconstants.mk.ftl b/runtime/makelib/mkconstants.mk.ftl
index 79017ad3..1ad5aa22 100644
--- a/runtime/makelib/mkconstants.mk.ftl
+++ b/runtime/makelib/mkconstants.mk.ftl
@@ -34,10 +34,10 @@ endif

 # Define all the tool used for compilation and linking.
 <#if uma.spec.tools.interp_gcc.needed>INTERP_GCC=${uma.spec.tools.interp_gcc.name}<#else>#INTERP_GCC not used</#if>
-<#if uma.spec.tools.as.needed>AS=${uma.spec.tools.as.name}<#else># AS not used</#if>
-<#if uma.spec.tools.cc.needed>CC=${uma.spec.tools.cc.name}<#else># CC not used</#if>
-<#if uma.spec.tools.cpp.needed>CPP=${uma.spec.tools.cpp.name}<#else># CPP not used</#if>
-<#if uma.spec.tools.cxx.needed>CXX=${uma.spec.tools.cxx.name}<#else># CXX not used</#if>
+<#if uma.spec.tools.as.needed>AS?=${uma.spec.tools.as.name}<#else># AS not used</#if>
+<#if uma.spec.tools.cc.needed>CC?=${uma.spec.tools.cc.name}<#else># CC not used</#if>
+<#if uma.spec.tools.cpp.needed>CPP?=${uma.spec.tools.cpp.name}<#else># CPP not used</#if>
+<#if uma.spec.tools.cxx.needed>CXX?=${uma.spec.tools.cxx.name}<#else># CXX not used</#if>
 <#if uma.spec.tools.rc.needed>RC=${uma.spec.tools.rc.name}<#else># RC not used</#if>
 <#if uma.spec.tools.link.needed>LINK=${uma.spec.tools.link.name}<#else># LINK not used</#if>
 <#if uma.spec.tools.mt.needed>MT=${uma.spec.tools.mt.name}<#else># MT not used</#if>

in an attempt to conditionally set CPP etc only if they aren't passed in.

This seems to have broken in another way (java compiles fail which shouldn't be possible....)

[ymanton]

Possibly because CC, CPP, and friends have a default value, so setting them via ?= might never work. You would end up with the environment value or the default value supplied by make but never the value you're trying to set it to.

[DanHeidinga]

I was also explicitly setting CC, CXX, etc.

[DanHeidinga]

export CC=gcc-4.8
export CC_PATH=gcc-4.8
export CXX=g++-4.8
export CXX_PATH=gcc-4.8
export CPP=gcc-4.8 -E -P

• the patch in https://github.com/eclipse/openj9/issues/1180#issuecomment-368592980 allows the build to complete but fails to load the JIT:

$ ./build/linux-x86_64-normal-server-release/images/j2sdk-image/bin/java -version
JVMJ9VM011W Unable to load j9jit29: /tmp/danh/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so: undefined symbol: _ZTVN10__cxxabiv121__vmi_class_type_infoE
openjdk version "1.8.0_162-internal"
OpenJDK Runtime Environment (build 1.8.0_162-internal-j9build_2018_02_25_05_28-b00)
Eclipse OpenJ9 VM (build master-ebc19708, JRE 1.8.0 Linux amd64-64 Compressed References 20180226_000000 (JIT abled, AOT abled)
OpenJ9   - ebc19708
OMR      - f698070a
JCL      - e2cb8c715c based on jdk8u162-b12)

$ c++filt _ZTVN10__cxxabiv121__vmi_class_type_infoE
vtable for __cxxabiv1::__vmi_class_type_info

[DanHeidinga]

Exported these as well - jit still fails to load.

export AR=gcc-ar-4.8 
export RANLIB=gcc-ranlib-4.8

Hand hacking the OMR configure.mk.in file

CC := gcc-4.8
CXX := g++-4.8
CCLINK := @CCLINK@
CCLINKSHARED := @CCLINKSHARED@
CCLINKEXE := @CCLINKEXE@
CXXLINK := @CXXLINK@
CXXLINKSHARED := @CXXLINKSHARED@
CXXLINKEXE := @CXXLINKEXE@

AS := @AS@
AR := gcc-ar-4.8

Still having issues in the JIT.

[DanHeidinga]

Discussed on Slack, plan is for Adopt to create a machine with gcc 4.8 as the default gcc and adapt the hotspot builds to set CC/CXX until we can fix our build scripts.

See conversation starting at: https://openj9.slack.com/archives/C8312LCV9/p1519749353000502

Dan Heidinga [11:35 AM]
While we work on fixing the openj9 makefiles to respect the `CC` & `CXX` settings, can the machines default to gcc 4.8 and the Hotspot builds set CC/CXX as required?

gdams [11:36 AM]
yes that's fine
I will setup another machine and use symlinks to handle this
but the ENV vars would be ideal in the long term as it will allow us to quickly switch between gcc4.8 and other versions

[DanHeidinga]

@gdams FYI - we've merged change to the OpenJ9 build scripts to respect setting CC & CXX.

[sxa]

I'll hold off on fully testing this until we've shipped the current release as I'm not going to risk changing anything even temporarily at this stage in the cycle.

[sxa]

@DanHeidinga Just to be clear that fix has only gone into master and not the 0.8.0 branch right?

[DanHeidinga]

Yes, this only applies to the master branch. Good catch!