Openj9: Support for compiling on gcc 7.3

[pshipton]

OpenJDK has modified Linux compilation for Java 11+ to use gcc 7.3 and OpenJ9 should support the same.

https://bugs.openjdk.java.net/browse/JDK-8200375

[fjeremic]

I think https://github.com/eclipse/openj9/pull/1682 is a prerequisite to this. We need to fix the JIT to be able to compile with the latest GCC versions.

[mstoodle]

Just to mark it here explicitly, #1682 fixes have been merged at this point (actually 12 days ago :) )

[pshipton]

I've heard that OpenJ9 compiles with gcc 7.3 with warnings as errors disabled.

[babsingh]

@fjeremic i tried building a Java 11 SDK using OpenJ9 with gcc-7 and g++-7. i got the following compile errors related to JIT source code:

In file included from jswalk.c:30:0:
../compiler/trj9/runtime/MethodMetaData.h:384:18: error: inline function 'getByteCodeInfo' declared but never defined [-Werror]
 JITINLINE void * getByteCodeInfo(void *inlinedCallSite);
                  ^~~~~~~~~~~~~~~
../compiler/trj9/runtime/MethodMetaData.h:369:18: error: inline function 'getByteCodeInfoFromStackMap' declared but never defined [-Werror]
 JITINLINE void * getByteCodeInfoFromStackMap(J9TR_MethodMetaData * metaData, void * stackMap);
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~
../compiler/trj9/runtime/MethodMetaData.h:350:17: error: inline function 'getFirstStackMap' declared but never defined [-Werror]
 JITINLINE U_8 * getFirstStackMap(J9TR_StackAtlas * stackAtlas);
                 ^~~~~~~~~~~~~~~~
...

is there a workaround? --std=gnu89 or --std=c11?

[DanHeidinga]

@mpirvu Have you been able to compile with gcc 7? I recall mention of it from the fix devirtualization PR.

[mpirvu]

I've compiled the JIT dll for Java8 with gcc 7.3 on Ubuntu and could run successfully with it.
I did not try to compile the entire VM.

[JamesKingdon]

I built the complete VM with gcc7.2.1 for arm. I hit the same 'declared but never defined' errors as Babneet until I used UMA_SUPPRESS_WARNINGS_AS_ERRORS=1 on the make command line. The resulting build seems to run ok.

[DanHeidinga]

UMA_SUPPRESS_WARNINGS_AS_ERRORS=1

Longer (medium?) term that'll need to be addressed. If anyone has suggestions on right approach, would love to hear them.

[mpirvu]

I don't see those functions being used outside MethodMetaData.c. I am thinking that it should be possible to simply remove their declaration from MethodMetaData.h

[babsingh]

I build the complete Java 11 OpenJ9 SDK with gcc-7.3 and g++7.3 on Linux x86. I encountered the following issues:

1. ../compiler/trj9/runtime/MethodMetaData.h:384:18: error: inline function '${FUNCTION_NAME}' declared but never defined [-Werror]. gcc 7.3 defaults to GNU11 but JVM uses GNU89 for inline semantics. Setting CFLAGS to either -std=gnu89 or -fgnu89-inline resolves this issue. [Reference] Porting to GCC 5 - https://gcc.gnu.org/gcc-5/porting_to.html.

2. -Werror=unused-const-variable=: This error is resolved by removing the unused constants. Unused constants were found in the following files:
   a. jniinv.c
   b. linearswalk.c
   c. jvmtiExtensionMechanism.c

3. -Werror=literal-suffix: gcc-7.3 expects a space between literal and string macro. This can be resolved by adding a space between a literal and string macro. This issue was identified in the following files:
   a. shrinit.h
   b. CompiledMethodTest.cpp

4. -Werror=format-overflow=: gcc-7.3 identifies buffer overflow. This error was seen for few testcases within CacheDirPerm.cpp. This error was resolved by increasing buffer size.

The top six commits in the following branch address the above gcc-7.3 issues: https://github.com/babsingh/openj9/commits/jdk11_gcc73. With these changes, Java 11 OpenJ9 SDK is built successfully without any errors using gcc-7.3.

[pshipton]

You could open a PR to fix issues 2, 3, 4. Not sure about the solution to issue 1, the JIT team should weigh in.

[pshipton]

You can try Marius's suggestion https://github.com/eclipse/openj9/issues/1684#issuecomment-388065325

[babsingh]

You can try Marius's suggestion #1684 (comment)

there may be other places in OpenJ9 where GNU89 inline semantics may be used. so, we should update makefiles to include CFLAGS=-fgnu89-inline in order to resolve this issue.

[babsingh]

You could open a PR to fix issues 2, 3, 4.

OpenJ9 PR - https://github.com/eclipse/openj9/pull/1890

[babsingh]

Update: OpenJDK11 with OpenJ9 should now compile with gcc7.3 on xLinux (related pull requests - https://github.com/eclipse/openj9/pull/1921, https://github.com/eclipse/openj9/pull/1890). On pLinux, OpenJDK11 with OpenJ9 fails compilation in native OpenJDK code. OpenJDK11 code still doesn't have code changes to support gcc7.3 on pLinux. There is no zLinux machine available to test OpenJDK11 with OpenJ9 compilation using gcc7.3. Based upon lack of pLinux support, I am guessing zLinux support for gcc7.3 should be missing in OpenJDK11.

For https://github.com/eclipse/openj9/pull/1921 changes, @keithc-ca put a pull request to replace gcc with $(CC) since multiple gcc versions may be available on a machine, and the default gcc compiler may not be used for compilation. @keithc-ca please provide a link to the pull request which addresses the above issue.

[keithc-ca]

The PR for Java 8 to use $(CC) instead of gcc is https://github.com/ibmruntimes/openj9-openjdk-jdk8/pull/88, but it had to be reverted because it broke AIX builds using xlc. I have been working towards an OpenJ9 PR that will allow ibmruntimes/openj9-openjdk-jdk8#88 (and similar changes for other extension repos) to be restored.

[irinarada]

@keithc-ca - any updates on your fix?

[pshipton]

@irinarada I think this is the wrong issue to be asking, I think you meant to ask in #1919. Anyway, there are still some problems to resolve before #1919 is fixed.

[pshipton]

@babsingh what testing did you run on the xLinux JVM compiled with gcc 7.3?

[babsingh]

just java -version. would you like me to run the sanity test suite?

[pshipton]

would you like me to run the sanity test suite?

please do, and if they pass and its easy might as well run the extended tests as well.

[babsingh]

i tried compiling/running sanity test suite with JDK11, which is built using GCC-7.3. because of https://github.com/eclipse/openj9/issues/992, openj9/tests can't be compiled with JDK11. JEP 320 removes the Java EE and CORBA modules in JDK11. there is no quick fix for this issue. we will have to wait for the test team to resolve this issue. for future testing, i have a uploaded JDK11 built using GCC-7.3 on Box: https://ibm.box.com/s/3o2ztz67fdbhll0ey3kqdyv13py7qjg3.

fyi - @pshipton @llxia

[llxia]

Ran sanity test locally with above JDK11. For now, I excluded following projects as they need to be updated (see #2071 for detail)

Java8andUp
Java9andUp
Jsr335
VM_Test
jvmtitests
cmdLineTests/jvmtitests
SharedCPEntryInvokerTests

Test Result:

FAILED test targets:
    cmdLineTester_SCCommandLineOptionTests_1

TOTAL: 77   EXECUTED: 49   PASSED: 48   FAILED: 1   SKIPPED: 28

Output:

Testing: nameOption2
Test start time: 2018/06/07 19:47:00 Coordinated Universal Time
Running command: sh nameOption2.sh /java/bin/../bin
Time spent starting: 5 milliseconds
Time spent executing: 962 milliseconds
Test result: FAILED
 [OUT] Hello, world!
 [OUT] TEST PASSED
 [OUT] nameOption2: TEST FAILED
 [OUT] nameOption2: Expected to find the cache name
 [OUT] TEST PASSED
 [ERR] JVMSHRC806I Compressed references persistent shared cache "root" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
>> Success condition was found: [Output match: TEST PASSED]
>> Failure condition was found: [Output match: TEST FAILED]
>> Failure condition was not found: [Output match: Error:]
>> Failure condition was not found: [Output match: Unhandled Exception]
>> Failure condition was not found: [Output match: Exception:]
...
Testing: nameOption4
Test start time: 2018/06/07 19:47:02 Coordinated Universal Time
Running command: sh nameOption4.sh /java/bin/../bin
Time spent starting: 3 milliseconds
Time spent executing: 953 milliseconds
Test result: FAILED
 [OUT] Hello, world!
 [OUT] TEST PASSED
 [OUT] nameOption4: TEST FAILED
 [OUT] nameOption4: Expected to find the cache name
 [OUT] TEST PASSED
 [ERR] JVMSHRC806I Compressed references persistent shared cache "_root_root" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
>> Success condition was found: [Output match: TEST PASSED]
>> Failure condition was found: [Output match: TEST FAILED]
>> Failure condition was not found: [Output match: Error:]
>> Failure condition was not found: [Output match: Unhandled Exception]
>> Failure condition was not found: [Output match: Exception:]

root@1b1649416ec0:/test/TestConfig# java -version
openjdk version "11-internal" 2018-09-25
OpenJDK Runtime Environment (build 11-internal+0-adhoc..openj9-openjdk)
Eclipse OpenJ9 VM (build master-c85be19, JRE 11 Linux amd64-64-Bit Compressed References 20180605_000000 (JIT enabled, AOT enabled)
OpenJ9   - c85be19
OMR      - 7b106b8
JCL      - c67ebd4 based on jdk-11+15)

[llxia]

I ran the test in docker and it does not have LOGNAME. If I set it to the current login username root, cmdLineTester_SCCommandLineOptionTests_1 passes.

[llxia]

Ran extended.functional

FAILED test targets:
    testSCCMLTests2_1

TOTAL: 154   EXECUTED: 51   PASSED: 50   FAILED: 1   SKIPPED: 103

testSCCMLTests2_1 failure is related to missing VM_TEST.jar

In summary, excluding the following projects, the rest sanity and extended tests are ok.

Java8andUp
Java9andUp
Jsr335
VM_Test
jvmtitests
cmdLineTests/jvmtitests
SharedCPEntryInvokerTests

[babsingh]

Update: OpenJ9 JDK11 builds successfully on xLinux and pLinux with gcc-7.3.

• OpenJ9 JDK11 xLinux (build with gcc-7.3)
• OpenJ9 JDK11 pLinux (build with gcc-7.3)

OpenJ9 JDK11 build fails on zLinux/gcc-7.3 with compile failures in OpenJDK code. But, OpenJ9 compile is successful.

• make_jdk11_linux_s390x.log

[pshipton]

@babsingh can you please re-try zlinux with the latest jdk11 extensions and see where it stands now.

[babsingh]

@pshipton do you also want me to save the build log and the SDK if it passes successfully?

[pshipton]

If its working we can change OpenJ9 Java 11 to compile with gcc 7. If its not working, I think we'll have to see about fixing the problems in the openjdk11 extensions. I suppose the build log might help identify the first problem, but I don't think we need the entire log.

Not sure anybody needs the sdk right now.

[babsingh]

@pshipton JDK11 zLinux didn't compile with gcc-7.3.0. I saw two errors:
1) error: module not found: jdk.hotspot.agent
2) error: array subscript is below array bounds

/root/openj9-openjdk-jdk11/src/java.base/share/native/libfdlibm/k_rem_pio2.c: In function ‘__j__kernel_rem_pio2’:
/root/openj9-openjdk-jdk11/src/java.base/share/native/libfdlibm/k_rem_pio2.c:255:27: error: array subscript is below array bounds [-Werror=array-bounds]
                 for(k=1;iq[jk-k]==0;k++);   /* k = no. of terms needed */
                         ~~^~~~~~

Compilation stopped due to the second error. The second error is related to OpenJDK native code.

OpenJDK Extensions repo used: https://github.com/ibmruntimes/openj9-openjdk-jdk11.

Build log: jdk11_zlinux_build.log

[irinarada]

@babsingh any ETA for the fix for the second error above?

[babsingh]

@irinarada this is an issue with the latest OpenJDK11 extensions repo (https://github.com/ibmruntimes/openj9-openjdk-jdk11). The JDK/JCL team will have to fix this issue.

OpenJ9 (VM/GC/JIT/OMR) should compile fine with gcc-7.3.0 on zLinux; I have properly compiled OpenJ9 JDK11 using gcc-7.3.0 on zLinux with an older extensions repo.

[sxa]

I'm getting an issue with OMR compilation using GCC7.3 unless --disable-warnings-as-errors is specified:

HeapRegionManager.cpp: In member function 'void MM_HeapRegionManager::insertHeapRegion(MM_EnvironmentBase*, MM_HeapRegionDescriptor*)':
HeapRegionManager.cpp:118:132: error: self-comparison always evaluates to true [-Werror=tautological-compare]
    J9_LINEAR_LINKED_LIST_ADD(_nextRegion, _previousRegion, _auxRegionDescriptorList, (MM_HeapRegionDescriptor*)heapRegion);
                                                                                                         ~~~~~~~~~~~~~~~~~~~        ^                            
HeapRegionManager.cpp:118:292: error: self-comparison always evaluates to true [-Werror=tautological-compare]
    J9_LINEAR_LINKED_LIST_ADD(_nextRegion, _previousRegion, _auxRegionDescriptorList, (MM_HeapRegionDescriptor*)heapRegion);

[babsingh]

@sxa555 on which platform, did you see this error? was this error seen while compiling OpenJ9 with OMR?

there are Dockerfiles [xLinux, pLinux and zLinux] to compile OpenJ9 JDK11 with gcc-7.3.0. Dockerfiles: https://github.com/eclipse/openj9/tree/master/buildenv/docker/jdk11. do you see the same error with these Dockerfiles?

[sxa]

The warnings appears on (at least) zlinux and xlinux with both JDK8 and JDK11 source with GCC 7.3.0 - it's a new check that the compiler warns on that was introduced in either gcc 5 or 6.

[babsingh]

HeapRegionManager.cpp:118:132: error: self-comparison always evaluates to true [-Werror=tautological-compare]

After some investigation, I found that the above error only occurs with ccache enabled. The error is resolved by upgrading ccache to version 3.4.2 or by disabling ccache.

[pshipton]

gcc 7.3 is supported now. #2718 enables it for xLinux, zLinux in pull request and nightly builds.
pLinux will come later once #2788 is resolved, I believe this issue is resolved.