Temurin-build: Latest jtreg builds are not good

Created on 7 Mar 2018 · 18Comments · Source: adoptium/temurin-build

jtreg version details:
jtreg, version 4.2 dev b12
Built with 1.8.0_121 on 02/23/2018 08:12 PM.
JTHarness: version 5.0
JCov 3.0-2
TestNG (testng.jar): version unknown
TestNG (jcommander.jar): version 1.48.0
AsmTools: version 7.0

On running any regression test cases, it fails with similar reason.

Steps to reproduce:
Download or clone test base from https://github.com/AdoptOpenJDK/openjdk-jdk8u
Download latest JDK(Any version, have corresponding test base)
Download last successful jtreg from https://ci.adoptopenjdk.net/view/Dependencies/job/jtreg/, the zip file jtreg-4.2-b12.tar.gz not jtreg-4.2.0-tip.tar.gz
Execute any jtreg test case:
Eg: ./jtreg-adopt/bin/jtreg -verbose:all -jdk:./jdk8u172-b03-adopt/ -dir:./adopt-jdk8u/jdk/test java/math/BigDecimal/AddTests.java

Exception/Error from Log:

-----testresult-----

end=Wed Mar 07 13:17:55 MSK 2018
execStatus=Error. Unexpected error caught from test java/math/BigDecimal/AddTests.java: java.lang.NoSuchMethodError: com.sun.javatest.regtest.agent.JDK_Version.compareTo(Lcom/sun/javatest/regtest/agent/JDK_Version;)I
sections=script_messages Details

section:script_messages

----------messages:(0/0)----------

section:Details

----------messages:(0/0)----------
----------Stack trace:(7/555)----------
java.lang.NoSuchMethodError: com.sun.javatest.regtest.agent.JDK_Version.compareTo(Lcom/sun/javatest/regtest/agent/JDK_Version;)I
at com.sun.javatest.regtest.config.JDK.getSystemModules(JDK.java:405)
at com.sun.javatest.regtest.exec.RegressionScript.run(RegressionScript.java:108)
at com.sun.javatest.Script.run(Script.java:241)
at com.sun.javatest.DefaultTestRunner.runTest(DefaultTestRunner.java:174)
at com.sun.javatest.DefaultTestRunner.access$100(DefaultTestRunner.java:43)
at com.sun.javatest.DefaultTestRunner$1.run(DefaultTestRunner.java:66)
result: Not run. Test running...

test result: Error. Unexpected error caught from test java/math/BigDecimal/AddTests.java: java.lang.NoSuchMethodError: com.sun.javatest.regtest.agent.JDK_Version.compareTo(Lcom/sun/javatest/regtest/agent/JDK_Version;)I

Reported to OpenJDK / JBS bug important

Source

akolarkunnu

All 18 comments

Run with latest jtreg https://ci.adoptopenjdk.net/view/Dependencies/job/jtreg/.
jtreg-4.2-b12.tar.gz and jtreg-4.2.0-tip.tar.gz have same behavior.
Test passed. However at the end of test we got:
java.lang.IllegalStateException
at com.sun.javatest.TestResult.setTestDescription(TestResult.java:1919)
at com.sun.javatest.TestResultTable.getCachedResult(TestResultTable.java:2193)
at com.sun.javatest.TRT_TreeNode.processFile(TRT_TreeNode.java:1115)
at com.sun.javatest.TRT_TreeNode.scanIfNeeded(TRT_TreeNode.java:651)
at com.sun.javatest.TRT_TreeNode.getChildCount(TRT_TreeNode.java:125)
at com.sun.javatest.TRT_Iterator$PseudoFrame.nextIndex(TRT_Iterator.java:989)
at com.sun.javatest.TRT_Iterator.findNext(TRT_Iterator.java:616)
at com.sun.javatest.TRT_Iterator.nextElement(TRT_Iterator.java:186)
at com.sun.javatest.TRT_Iterator.next(TRT_Iterator.java:208)
at com.sun.javatest.report.ReportSettings.setupSortedResults(ReportSettings.java:483)
at com.sun.javatest.report.Report.writeReports(Report.java:111)
at com.sun.javatest.report.Report.writeReport(Report.java:90)
at com.sun.javatest.regtest.report.RegressionReporter.report(RegressionReporter.java:99)
at com.sun.javatest.regtest.tool.Tool.batchHarness(Tool.java:1835)
at com.sun.javatest.regtest.tool.Tool.run(Tool.java:1216)
at com.sun.javatest.regtest.tool.Tool.run(Tool.java:962)
at com.sun.javatest.regtest.tool.Tool.main(Tool.java:134)
at com.sun.javatest.regtest.Main.main(Main.java:58)

sophia-guo on 9 Apr 2018

This one might need to get reported to the jtreg-dev list on openjdk - @sophia-guo are you a member?

karianna on 10 Apr 2018

No, I'm not

sophia-guo on 10 Apr 2018

@sophia-guo So is this any test that fails with that message?

karianna on 10 Apr 2018

Wait, I need to double check if I messed up with different version. I ran it today for other issues it seems ok. I will double check and update it.

sophia-guo on 10 Apr 2018

You may have had a previous bad binary with the same version number (bug in the build system) if you grab the latest successful build, hopefully that'll work..

karianna on 11 Apr 2018

I'm confused now.

I have run test with latest jtreg-4.2-b12.tar.gz and jtreg-4.2.0-tip.tar.gz against openjdk9 tests and openjdk10 tests. All is good.

However running test with latest jtreg-4.2-b12.tar.gz, jtreg-4.2.0-tip.tar.gz and earlier good treg-4.2.0-tip.tar.gz against openjdk8 test will get the exception same as mentioned in earlier comment:

TEST RESULT: Passed. Execution successful

Test results: passed: 1
Error: Unexpected exception occurred! java.lang.IllegalStateException
java.lang.IllegalStateException
at com.sun.javatest.TestResult.setTestDescription(TestResult.java:1919)
at com.sun.javatest.TestResultTable.getCachedResult(TestResultTable.java:2193)
at com.sun.javatest.TRT_TreeNode.processFile(TRT_TreeNode.java:1115)
at com.sun.javatest.TRT_TreeNode.scanIfNeeded(TRT_TreeNode.java:651)
at com.sun.javatest.TRT_TreeNode.getChildCount(TRT_TreeNode.java:125)
at com.sun.javatest.TRT_Iterator$PseudoFrame.nextIndex(TRT_Iterator.java:989)
at com.sun.javatest.TRT_Iterator.findNext(TRT_Iterator.java:616)
at com.sun.javatest.TRT_Iterator.nextElement(TRT_Iterator.java:186)
at com.sun.javatest.TRT_Iterator.next(TRT_Iterator.java:208)
at com.sun.javatest.report.ReportSettings.setupSortedResults(ReportSettings.java:483)
at com.sun.javatest.report.Report.writeReports(Report.java:111)
at com.sun.javatest.report.Report.writeReport(Report.java:90)
at com.sun.javatest.regtest.report.RegressionReporter.report(RegressionReporter.java:102)
at com.sun.javatest.regtest.tool.Tool.batchHarness(Tool.java:1868)
at com.sun.javatest.regtest.tool.Tool.run(Tool.java:1242)
at com.sun.javatest.regtest.tool.Tool.run(Tool.java:967)
at com.sun.javatest.regtest.tool.Tool.main(Tool.java:139)
at com.sun.javatest.regtest.Main.main(Main.java:58)

sophia-guo on 12 Apr 2018

Looks like it is failing to write out the test results, and maybe its choking on a particular piece of test output that happens only for openjdk8 (there are tests that are excluded for openj9, as they are hotspot specific, maybe one of those tests fails and produces output the Report.writeReports() method can't parse/write.

I only see part of your output there, but can you reproduce running only a single test class (that passed just prior)?

smlambert on 12 Apr 2018

👍1

@sophia-guo @smlambert Could this be environment specific, is this reproducible with other test suites from other versions of OpenJDK? If not, it should be reported to the jtreg mailing list (mailing list email address: [email protected])

neomatrix369 on 13 Apr 2018

👍1

I'm using same test java/math/BigDecimal/AddTests.java against openjdk8/9/10. Only against openjdk8 have this issue. I also tried other simple test, same behavior.

Command without specify report and work dir:

sophiaguos-mbp:adoptjdktest sophiag$ /Users/sophiag/adoptjdktest/jtregnew/jtreg/bin/jtreg -verbose:all -jdk:/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03/ /Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal/AddTests.java

TEST: java/math/BigDecimal/AddTests.java
TEST JDK: /Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03

ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME: 0.001 seconds
messages:
command: build AddTests
reason: Named class compiled on demand
elapsed time (seconds): 0.001
...

...
1E+2147483647 + 2E-2147483647 = 1.0E+2147483647
1E+2147483647 + -2E-2147483647 = 9.9E+2147483646
STDERR:
STATUS:Passed.
rerun:
cd /Users/sophiag/adoptjdktest/JTwork/scratch && \
HOME=/Users/sophiag \
LANG=en_CA.UTF-8 \
PATH=/bin:/usr/bin \
CLASSPATH=/Users/sophiag/adoptjdktest/JTwork/classes/java/math/BigDecimal:/Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal:/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03/lib/tools.jar:/Users/sophiag/adoptjdktest/jtregnew/jtreg/lib/javatest.jar:/Users/sophiag/adoptjdktest/jtregnew/jtreg/lib/jtreg.jar \
/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03/bin/java \
-Dtest.src=/Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal \
-Dtest.src.path=/Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal \
-Dtest.classes=/Users/sophiag/adoptjdktest/JTwork/classes/java/math/BigDecimal \
-Dtest.class.path=/Users/sophiag/adoptjdktest/JTwork/classes/java/math/BigDecimal \
-Dtest.vm.opts= \
-Dtest.tool.vm.opts= \
-Dtest.compiler.opts= \
-Dtest.java.opts= \
-Dtest.jdk=/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03 \
-Dcompile.jdk=/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03 \
-Dtest.timeout.factor=1.0 \
com.sun.javatest.regtest.agent.MainWrapper /Users/sophiag/adoptjdktest/JTwork/java/math/BigDecimal/AddTests.d/main.0.jta

TEST RESULT: Passed. Execution successful

Against openjdk9:
sophiaguos-mbp:adoptjdktest sophiag$ /Users/sophiag/adoptjdktest/jtregnew/jtreg/bin/jtreg -verbose:all -jdk:/Users/sophiag/adoptjdktest/openjdk9binary/j2sdk-image/ /Users/sophiag/adoptjdktest/openjdk9/openjdk-jdk9/jdk/test/java/math/BigDecimal/AddTests.java

TEST: java/math/BigDecimal/AddTests.java
TEST JDK: /Users/sophiag/adoptjdktest/openjdk9binary/j2sdk-image

ACTION: build -- Passed. All files up to date
REASON: Named class compiled on demand
TIME: 0.002 seconds
messages:
command: build AddTests
reason: Named class compiled on demand
elapsed time (seconds): 0.002

ACTION: main -- Passed. Execution successful
REASON: Assumed action based on file name: run main AddTests
TIME: 0.2 seconds
...
STDOUT:
1E+2147483647 + 2E-2147483647 = 1.0E+2147483647
1E+2147483647 + -2E-2147483647 = 9.9E+2147483646
STDERR:
STATUS:Passed.
rerun:
cd /Users/sophiag/adoptjdktest/JTwork/scratch && \
HOME=/Users/sophiag \
LANG=en_CA.UTF-8 \
PATH=/bin:/usr/bin \
CLASSPATH=/Users/sophiag/adoptjdktest/JTwork/classes/java/math/BigDecimal:/Users/sophiag/adoptjdktest/openjdk9/openjdk-jdk9/jdk/test/java/math/BigDecimal:/Users/sophiag/adoptjdktest/jtregnew/jtreg/lib/javatest.jar:/Users/sophiag/adoptjdktest/jtregnew/jtreg/lib/jtreg.jar \
/Users/sophiag/adoptjdktest/openjdk9binary/j2sdk-image/bin/java \
-Dtest.src=/Users/sophiag/adoptjdktest/openjdk9/openjdk-jdk9/jdk/test/java/math/BigDecimal \
-Dtest.src.path=/Users/sophiag/adoptjdktest/openjdk9/openjdk-jdk9/jdk/test/java/math/BigDecimal \
-Dtest.classes=/Users/sophiag/adoptjdktest/JTwork/classes/java/math/BigDecimal \
-Dtest.class.path=/Users/sophiag/adoptjdktest/JTwork/classes/java/math/BigDecimal \
-Dtest.vm.opts= \
-Dtest.tool.vm.opts= \
-Dtest.compiler.opts= \
-Dtest.java.opts= \
-Dtest.jdk=/Users/sophiag/adoptjdktest/openjdk9binary/j2sdk-image \
-Dcompile.jdk=/Users/sophiag/adoptjdktest/openjdk9binary/j2sdk-image \
-Dtest.timeout.factor=1.0 \
com.sun.javatest.regtest.agent.MainWrapper /Users/sophiag/adoptjdktest/JTwork/java/math/BigDecimal/AddTests.d/main.0.jta

TEST RESULT: Passed. Execution successful

Test results: passed: 1
Report written to /Users/sophiag/adoptjdktest/JTreport/html/report.html
Results written to /Users/sophiag/adoptjdktest/JTwork

sophia-guo on 13 Apr 2018

If I specify the work and report directory it would be fine:

sophiaguos-mbp:adoptjdktest sophiag$ /Users/sophiag/adoptjdktest/jtregnew/jtregb12/jtreg/bin/jtreg -verbose:all -w /Users/sophiag/adoptjdktest/work -r /Users/sophiag/adoptjdktest/report -jdk:/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03/ /Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal/AddTests.java
Directory "/Users/sophiag/adoptjdktest/work" not found: creating

Directory "/Users/sophiag/adoptjdktest/report" not found: creating

TEST: java/math/BigDecimal/AddTests.java
TEST JDK: /Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03

ACTION: build -- Passed. Build successful
REASON: Named class compiled on demand
TIME: 0.595 seconds
messages:
command: build AddTests
reason: Named class compiled on demand
Test directory:
compile: AddTests
elapsed time (seconds): 0.595

ACTION: compile -- Passed. Compilation successful
REASON: .class file out of date or does not exist
TIME: 0.586 seconds
messages:
command: compile -XDignore.symbol.file=true /Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal/AddTests.java
reason: .class file out of date or does not exist
Mode: othervm
elapsed time (seconds): 0.586
configuration:
javac compilation environment
source path: /Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal
class path: /Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal
/Users/sophiag/adoptjdktest/work/classes/java/math/BigDecimal
/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03/lib/tools.jar

rerun:
cd /Users/sophiag/adoptjdktest/work/scratch && \
HOME=/Users/sophiag \
LANG=en_CA.UTF-8 \
PATH=/bin:/usr/bin \
/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03/bin/javac \
-J-Dtest.src=/Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal \
-J-Dtest.src.path=/Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal \
-J-Dtest.classes=/Users/sophiag/adoptjdktest/work/classes/java/math/BigDecimal \
-J-Dtest.class.path=/Users/sophiag/adoptjdktest/work/classes/java/math/BigDecimal \
-J-Dtest.vm.opts= \
-J-Dtest.tool.vm.opts= \
-J-Dtest.compiler.opts= \
-J-Dtest.java.opts= \
-J-Dtest.jdk=/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03 \
-J-Dcompile.jdk=/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03 \
-J-Dtest.timeout.factor=1.0 \
-d /Users/sophiag/adoptjdktest/work/classes/java/math/BigDecimal \
-sourcepath /Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal \
-classpath /Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal:/Users/sophiag/adoptjdktest/work/classes/java/math/BigDecimal:/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03/lib/tools.jar \
-XDignore.symbol.file=true /Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal/AddTests.java
STDOUT:
STDERR:

ACTION: main -- Passed. Execution successful
REASON: Assumed action based on file name: run main AddTests
TIME: 0.163 seconds
messages:
command: main AddTests
reason: Assumed action based on file name: run main AddTests
Mode: othervm
elapsed time (seconds): 0.163
configuration:
STDOUT:
1E+2147483647 + 2E-2147483647 = 1.0E+2147483647
1E+2147483647 + -2E-2147483647 = 9.9E+2147483646
STDERR:
STATUS:Passed.
rerun:
cd /Users/sophiag/adoptjdktest/work/scratch && \
HOME=/Users/sophiag \
LANG=en_CA.UTF-8 \
PATH=/bin:/usr/bin \
CLASSPATH=/Users/sophiag/adoptjdktest/work/classes/java/math/BigDecimal:/Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal:/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03/lib/tools.jar:/Users/sophiag/adoptjdktest/jtregnew/jtregb12/jtreg/lib/javatest.jar:/Users/sophiag/adoptjdktest/jtregnew/jtregb12/jtreg/lib/jtreg.jar \
/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03/bin/java \
-Dtest.src=/Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal \
-Dtest.src.path=/Users/sophiag/adoptjdktest/openjdk8/openjdk-jdk8u/jdk/test/java/math/BigDecimal \
-Dtest.classes=/Users/sophiag/adoptjdktest/work/classes/java/math/BigDecimal \
-Dtest.class.path=/Users/sophiag/adoptjdktest/work/classes/java/math/BigDecimal \
-Dtest.vm.opts= \
-Dtest.tool.vm.opts= \
-Dtest.compiler.opts= \
-Dtest.java.opts= \
-Dtest.jdk=/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03 \
-Dcompile.jdk=/Users/sophiag/adoptjdktest/openjdk8binary/jdk8u172-b03 \
-Dtest.timeout.factor=1.0 \
com.sun.javatest.regtest.agent.MainWrapper /Users/sophiag/adoptjdktest/work/java/math/BigDecimal/AddTests.d/main.0.jta

TEST RESULT: Passed. Execution successful

Test results: passed: 1
Report written to /Users/sophiag/adoptjdktest/report/html/report.html
Results written to /Users/sophiag/adoptjdktest/work
sophiaguos-mbp:adoptjdktest sophiag$

sophia-guo on 13 Apr 2018

That's also the reason why we didn't get the issue before. As most of time we have specified the work and report directory instead of using the default. @neomatrix369 I'm not a member of jtreg-dev, do I have the permission to report it?

sophia-guo on 13 Apr 2018

@karianna this issue should be good to close. The issue I mentioned should be reported to jtreg-dev as you and mani mentioned.

sophia-guo on 13 Apr 2018

@sophia-guo you don't need permissions, signing up is very easy, just go to http://mail.openjdk.java.net/mailman/listinfo/jtreg-dev and enter your email.

I highly recommend everyone using jtreg to signup to the mailing list, queries/issues posted there are answered in a timely manner.

Pls also consider signing up on _jtreg-use_ ==> http://mail.openjdk.java.net/mailman/listinfo/jtreg-use

/cc @smlambert

neomatrix369 on 14 Apr 2018

I'll set this to reported upstream until we get a fix from them

karianna on 15 Apr 2018

Have reported to the jtreg mailing list.

sophia-guo on 17 Apr 2018

Hi, we want to run our java9 and java10 based on jtreg latest version. Could anyone help to provide the right link or method to get the right version for openjdk9 and openjdk10?

Thanks,
Michelle