Openj9: Test-sanity.system-JDK8-osx_x86-64_cmprssptrs MauveSingleInvocationLoadTest_0 MauveMultiThreadLoadTest_0

Created on 18 Dec 2018 · 20Comments · Source: eclipse/openj9

https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK8-osx_x86-64_cmprssptrs/5
Similar failure for both tests _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.

22:38:30.331 - Completed 0.0%. Number of tests started=326 (+0)
22:38:30.869 - **POSSIBLE HANG DETECTED**

infra test excluded test failure

Source

pshipton

Most helpful comment

osxvm1:~ j9admin$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

Locale has been changed to US.

cwillhelm on 16 Jan 2019

👍3

All 20 comments

@Mesbah-Alam seems more like a test problem than OpenJ9 problem.

pshipton on 18 Dec 2018

Running Grinder on internal osx machine yields no crash, but test failure such as :

```LT FAIL: gnu.testlet.java.text.MessageFormat.format14: time (number 3)
LT got 2:38:01 EST PM but expected 2:38:01 PM EST
LT FAIL: gnu.testlet.java.text.MessageFormat.format14: time (number 4)
LT got 2:38:01 o'clock PM EST but expected 2:38:01 PM EST

LT FAIL: gnu.testlet.java.text.MessageFormat.format14: date (number 0)
LT got 18-Dec-2018 but expected Dec 18, 2018
LT FAIL: gnu.testlet.java.text.MessageFormat.format14: date (number 1)
LT got 18/12/18 but expected 12/18/18
LT FAIL: gnu.testlet.java.text.MessageFormat.format14: date (number 2)

Will run grinder on openj9 to see if the same issue is reproduced.

Mesbah-Alam on 18 Dec 2018

https://github.com/eclipse/openj9/issues/4091#issuecomment-448371280 appears a machine issue due to incorrect locale settings, or a failure in the test to set the expected locale before querying the result. i.e. the machine is configured one way but the tests expect something else.

pshipton on 18 Dec 2018

Same locale issue as https://github.com/eclipse/openj9/issues/4091#issuecomment-448371280 reproduced using openj9 grinder too: https://ci.eclipse.org/openj9/view/Test/job/Test-Grinder/129/tapResults/

Mesbah-Alam on 18 Dec 2018

@Mesbah-Alam Have you worked with @AdamBrousseau / @jdekonin to have the locale corrected?

DanHeidinga on 15 Jan 2019

Based on https://github.com/eclipse/openj9/issues/4091#issuecomment-454521908, and a fair number of other openj9 issues, I wonder if we need a comp:infra tag to help identify and report openj9 machine config issues.

smlambert on 15 Jan 2019

@AdamBrousseau , @jdekonin - could you please look into the locale settings in the osx machines at Openj9? Are they using different locales than the other platforms?

Mesbah-Alam on 15 Jan 2019

External OSX 10.11 machine

osx1011vm1:~ jenkins$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

Internal OSX 10.11 machine

osxvm1:~ jenkins$ locale
LANG="en_CA.UTF-8"
LC_COLLATE="en_CA.UTF-8"
LC_CTYPE="en_CA.UTF-8"
LC_MESSAGES="en_CA.UTF-8"
LC_MONETARY="en_CA.UTF-8"
LC_NUMERIC="en_CA.UTF-8"
LC_TIME="en_CA.UTF-8"
LC_ALL=

Seems one is US and one is CA. Is this what you were expecting? Might need to ask @cwillhelm if you want to change it.

AdamBrousseau on 15 Jan 2019

👍1

FYI @smlambert I've created and added the comp:infra tag

DanHeidinga on 16 Jan 2019

👍1

osxvm1:~ j9admin$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

Locale has been changed to US.

cwillhelm on 16 Jan 2019

👍3

Thanks Dan, I think that new label will help the releng/infra team better categorize and track what needs attending to, as I know from experience if I've just been mentioned in an issue, its sometimes challenging to find it again.

smlambert on 16 Jan 2019

On a Mac, the locale above is impacted by changing the region within the Language & Region system preferences. The current setup is Canada, because that is where they are located; UTF-8 is also a default for Canada or US.

Looking at the a ub16-390-1 machine, it looks somewhat different.

$ locale
LANG=C
LANGUAGE=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

ub16-ppcle-1

$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

aix71-p8-1

LANG=en_US
LC_COLLATE="en_US"
LC_CTYPE="en_US"
LC_MONETARY="en_US"
LC_NUMERIC="en_US"
LC_TIME="en_US"
LC_MESSAGES="en_US"
LC_ALL=

ub16-x86-1

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

How are the tests expecting the machines locale to be setup? Is the test something expecting US? Which values are looked up/used?

jdekonin on 16 Jan 2019

Why the WindowServer error for OSX. This is detailed in the description of this issue.

jdekonin on 16 Jan 2019

@smlambert @llxia The following sanity.system tests are consistently failing in osx jdk8 testing. Can we please exclude them until they can be fixed. (I would create a PR but not sure where to do this).

https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK8-osx_x86-64_cmprssptrs/73
MauveSingleThreadLoadTest_0
MauveSingleInvocationLoadTest_0
MauveMultiThreadLoadTest_0

pshipton on 21 Feb 2019

👍1

The problem has been seen on Linux as well:

FAIL: gnu.testlet.java.text.MessageFormat.format14: time (number 3)
got 5:02:52 AM ADT but expected 8:02:52 AM GMT

https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK8-linux_x86-64/269/tapResults/

Could we have similar machine issue on the Linux machines in OpenJ9 too?

Excluding the test from JDK8 on Linux platform.

Mesbah-Alam on 23 Mar 2019

Re https://github.com/eclipse/openj9/issues/4091#issuecomment-475878099, these are not functional tests and we don't need these specific sub-tests, they are just part of the system test load. We want testing to be robust and not depend on the exact setup of the machines. I'm happy to exclude and leave these sub-tests like gnu.testlet.java.text.MessageFormat.format14 excluded forever. And also to exclude them from all versions and platforms.

pshipton on 23 Mar 2019

@pshipton - I'll add individual sub-test level excludes for each failing sub-test (which will remain excluded permanently, across all platforms and sdk versions), and include back all mauve load test suites on all platforms on both Hotspot and OpenJ9.

Mesbah-Alam on 25 Mar 2019

👍1

I ran a few grinders to reproduce the "POSSIBLE HANG DETECTED" issue using the 3 variations of the mauve load tests:

MauveSingleThreadLoadTest :
OpenJ9 Java 8 : https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/56/ <--fails
OpenJ9 Java 11: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/60 <--fails
HotSpot Java 8 : https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/57/ <- passes
Hotspot Java 11: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/61 <- passes

MauveMultiThreadLoadTest
HotSpot Java 8 : https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/59 <- passes
Hotspot Java 11: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/62 <- passes
OpenJ9 Java 8: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/58 <-- fails
OpenJ9 Java 11: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/63 <- passes
OpenJ9 Java 12: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/66 <- passes

MauveSingleInvocationLoadTest
OpenJ9 Java 8 : https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder-systemtest/65 <- fails

In short, the issue still exists on OpenJ9 (while it doesn't appear on HotSpot) - so, all three mauve load test playlists will remain excluded from running on OSX, for OpenJ9 SDK, while we investigate further the reason for the hang (Note: the failure using the multithreaded version of the mauve load test only failed on JDK8, but passed on JDK11 and JDK12 -- I need to run some grinder on the JDK11 and JDK12 only to find out if it consistently passes on these JDKs while consistently reproducing the Hang issue on JDK8).

Mesbah-Alam on 25 Mar 2019

Another occurrence.
https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_special.system_x86-64_mac_Nightly/413

06:17:13  LT  06:17:12.165 - Completed 0.2%. Number of tests started=8 (+0)
06:17:34  LT  06:17:32.203 - Completed 0.2%. Number of tests started=8 (+0)
06:17:50  STF 06:17:48.418 - Heartbeat: Process LT  is still running
06:17:52  LT  06:17:52.140 - Completed 0.2%. Number of tests started=8 (+0)
06:17:52  LT  06:17:52.361 - **POSSIBLE HANG DETECTED**
06:17:52  STF 06:17:52.488 - **FAILED** Process LT  has hung
06:17:52  STF 06:17:52.488 - Collecting dumps for: LT 
06:17:52  STF 06:17:52.488 - Sending SIG 3 to the java process to generate a javacore
06:17:52  STF 06:17:52.489 - Running command: kill -3 72913
06:17:52  STF 06:17:52.489 - Redirecting stderr to /Users/jenkins/workspace/Test_openjdk8_j9_special.system_x86-64_mac_Nightly/openjdk-tests/TestConfig/test_output_15732756559540/MauveSingleInvocationLoadTest_special_22/20191109-060247-MauveSingleInvocationLoadTest/results/1.LT.kill_3.stderr
06:17:52  STF 06:17:52.489 - Redirecting stdout to /Users/jenkins/workspace/Test_openjdk8_j9_special.system_x86-64_mac_Nightly/openjdk-tests/TestConfig/test_output_15732756559540/MauveSingleInvocationLoadTest_special_22/20191109-060247-MauveSingleInvocationLoadTest/results/1.LT.kill_3.stdout