Openj9: JTReg jdk11-m2 Failure: java/util/Locale/LocaleCategory.java

[M-Davies]

Failure link

• test category, sanity.openjdk
• OS/architecture, aarch_64

17:48:19  openjdk version "11.0.7" 2020-04-14
17:48:19  OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+9)
17:48:19  Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.20.0-m2, JRE 11 Linux aarch64-64-Bit Compressed References 20200330_255 (JIT enabled, AOT enabled)
17:48:19  OpenJ9   - c93e4dabc
17:48:19  OMR      - 1b6abd044
17:48:19  JCL      - 644e9abfc0 based on jdk-11.0.7+9)

Optional info

• intermittent failure (yes):
• Part of the jdk11-m2 triage https://github.com/AdoptOpenJDK/openjdk-tests/issues/1705

Failure output (captured from console output)

java.util.IllformedLocaleException: Ill-formed language: c.u [at index 0]
    at java.base/java.util.Locale$Builder.setLanguage(Locale.java:2585)
    at LocaleCategory.main(LocaleCategory.java:56)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
    at java.base/java.lang.Thread.run(Thread.java:834)

[pshipton]

@knn-k

[knn-k]

Never seen this before.

[knn-k]

I cannot recreate the failure running the test 100+ times on cent7-aarch64-1.

[M-Davies]

I also can't within a grinder tailored to the failing test's conditions
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2708/

[knn-k]

Failed in https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_sanity.openjdk_aarch64_linux/111/

[knn-k]

I ran this test on test-packet-ubuntu1604-armv8-1 400 times, but I couldn't reproduce the failure at all.
I used the binary from https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_sanity.openjdk_aarch64_linux/111/:

OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+9-202004062324)
Eclipse OpenJ9 VM AdoptOpenJDK (build master-956678929, JRE 11 Linux aarch64-64-Bit Compressed References 20200406_263 (JIT enabled, AOT enabled)
OpenJ9   - 956678929
OMR      - 9d422b0b0
JCL      - 634f577760 based on jdk-11.0.7+9)

[M-Davies]

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2771/ openjdk_test_output (2).tar.gz
Ran this test on test-aws-ubuntu1804-armv8-1 and saw 100/100 failures. Could this perhaps be missing a dependency on the machine? @sxa ?

Binary is the same

09:14:13  openjdk version "11.0.7" 2020-04-14
09:14:13  OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+9-202004062324)
09:14:13  Eclipse OpenJ9 VM AdoptOpenJDK (build master-956678929, JRE 11 Linux aarch64-64-Bit Compressed References 20200406_263 (JIT enabled, AOT enabled)
09:14:13  OpenJ9   - 956678929
09:14:13  OMR      - 9d422b0b0
09:14:13  JCL      - 634f577760 based on jdk-11.0.7+9)

[knn-k]

Hmm.
The JIT compiler does not change its code generation depending on the CPU architecture. It uses the ARMv8-A instructions on all the platforms, and doesn't know ARMv8.1-A instructions or newer.

[knn-k]

Is it possible to request access to test-aws-ubuntu1804-armv8-1?

[M-Davies]

@knn-k Yes. Open up an issue at https://github.com/AdoptOpenJDK/openjdk-infrastructure 😄

[knn-k]

I opened https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1267 .

[knn-k]

I cannot reproduce the failure on test-aws-ubuntu1804-armv8-1 at all by manual runs, using the files from "openjdk_test_output (2).tar.gz" attached above.
I have no idea what makes the difference.

OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+9-202004062324)
Eclipse OpenJ9 VM AdoptOpenJDK (build master-956678929, JRE 11 Linux aarch64-64-Bit Compressed References 20200406_263 (JIT enabled, AOT enabled)
OpenJ9   - 956678929
OMR      - 9d422b0b0
JCL      - 634f577760 based on jdk-11.0.7+9)

[M-Davies]

@knn-k Can you output this line
https://github.com/ibmruntimes/openj9-openjdk-jdk11/blob/ea4a5a16094680be0e5a1fb46343e6c55382dcda/test/jdk/java/util/Locale/LocaleCategory.java#L56
to the console? See what language it's trying to pick up?

[knn-k]

This test passes -Duser.language=en as one of the command-line options. So System.getProperty("user.language", "") should return en.
On the other hand, the LANG environment variable on this machine is set to C.UTF-8 by default, which makes System.getProperty("user.language", "") return c.u. When I run the test manually without -Duser.language=en, the test fails with the following exception.

java.util.IllformedLocaleException: Ill-formed language: c.u [at index 0]
        at java.base/java.util.Locale$Builder.setLanguage(Locale.java:2585)
        at LocaleCategory.main(LocaleCategory.java:56)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

So the next question would be: Why is the language information lost when the test fails?

[M-Davies]

https://github.com/ibmruntimes/openj9-openjdk-jdk11/blob/ea4a5a16094680be0e5a1fb46343e6c55382dcda/test/jdk/java/util/Locale/LocaleCategory.java#L80 appears to set the test language back to the default after failing. Could that be it?

[knn-k]

It is the LANG environment variable that matters, now I believe.

The variable is set to en_US.UTF-8 by default on test-packet-armv8-ubuntu-16-04. The LocaleCategory.jtr file on the machine contains -Duser.language=en, and the test passes.

On the other hand, the variable is set to C.UTF-8 by default on test-aws-ubuntu1804-armv8-1. The LocaleCategory.jtr file does not contain -Duser.language=en, and the test fails.
When I set LANG=en_US.UTF-8 before starting ./maketest.sh $TEST_DIR _jdk_util_j9_0, the test passes.

When I ran this test manually, my command line contained -Duser.language=en, which makes the test pass. I took the command line options from the .jtr file from another machine.

[M-Davies]

Thanks @knn-k . If you believe that is the root cause, an issue should be opened at https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues to request that LANG is permantly changed to en_US.UTF-8 on that machine. Further discussion can continue there :-)

[M-Davies]

If you're happy with that?

[knn-k]

It seems the LANG environment variable is not the only factor.
It is set to C.UTF-8 on cent7-aarch64-1. The jtr file contains -Duser.language=en, and the test passes on the machine.

I don't think it is a problem of the OpenJ9 runtime side, anyway.

[M-Davies]

@knn-k FYI https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1268

[M-Davies]

Inf issue is closed and test is passing now. Thanks @Haroon-Khel