Openjdk-tests: system tests does not take jvm options

[llxia]

The current set up in systemtest playlist.xml will only run no option regardless what <variation> is added. The tests need to be updated to take JVM_OPTIONS.

Also, there is a test called LangLoDaaLoadTest_daaAlladTest. Is it intended or is it typo?

[Mesbah-Alam]

LangLoDaaLoadTest_daaAlladTest - is definitely a typo. Will fix. Thanks.

Will investigate why STF is not accepting JVM_OPTIONS. @llxia - is there a link that you can share where you observed STF not accepting JVM_OPTIONS?

[llxia]

JVM_OPTIONS did not get passed into stf.pl at all.

[Mesbah-Alam]

The reason why STF command line is not receiving any options passed in Variation is due to the fact that, we moved away from STF's make layer and started to use STF command line explicitly in playlist.xml.

There exists some code in STF's make layer that figures out the value for -java-args STF option (which is what we use to pass in parameters to the execute stage of STF):

```# Provide a means for users to pass -java-args to the STF tests

Java arguments may contain imbedded quotes. These need to be escaped on the STF command line.

An IBM (test / debugging) arguments example:

java -Xjit:counts="- - - - - - 1 1 1 1000 250 250 - - - 10000 100000 10000" -Xgcpolicy:optthruput

requires on Windows

stf.pl -java-args="-Xjit:counts="""- - - - - - 1 1 1 1000 250 250 - - - 10000 100000 10000""" -Xgcpolicy:optthruput"

and on unix

stf.pl -java-args="-Xjit:counts=\"- - - - - - 1 1 1 1000 250 250 - - - 10000 100000 10000\" -Xgcpolicy:optthruput"

ifneq (,$(JAVA_ARGS))
JAVA_ARGS_ARG:=-java-args="$(subst ",$(ESC_DQ),$(JAVA_ARGS))"
endif

ifeq ($(RM_PASS),1)
RM_PASS_ARG:=-rm-pass
endif

STF_COMMAND:=perl $(TARGET_ROOT)$(D)stf.core$(D)scripts$(D)stf.pl $(JAVA_ARGS_ARG) -systemtest-prereqs=$(PREREQS_ROOT) $(RM_PASS_ARG)
```
We need to do something equivalent somewhere in the new world.

@ShelleyLambert @renfeiw - Should we encapsulate this somewhere in TestConfig?

[smlambert]

In my opinion, STF should have a parseArgs() method, rather than it living in the make layer, and its make layer eventually simplified.

We should not have to adopt their make layer for this (as its proven to be difficult to follow, slow to debug, etc).

[llxia]

Will we run JCK tests with different variations? If so, JCK has the same issue - JVM_OPTIONS is not passed in. fyi @TianyuZuo

I agree with Shelley, testKitGen should not need to have knowledge about stf variable.
Alternatively, systemtest can have its own makefile for this. If both system and JCK need it, then maybe a common makefile for system and JCK.

[smlambert]

We should not rule out the requirement to run JCKs with JVM_OPTIONS.

[Mesbah-Alam]

@llxia - STF already has a make layer : https://github.com/AdoptOpenJDK/openjdk-systemtest/blob/master/openjdk.build/makefile.

We can choose one from the two following alternatives :-
1) We can change the test command lines used in the playlist.xml for systemtest and JCK to use the make targets we'd define in the above make layer (e.g. test.SampleEchoFile).
Pros: We can take advantage of existing code that formulates the stf command line with jvm options for us (it also has support for platform dependent details, e.g. embedded quotes etc (see code snippet in my previous comment)). Cons: We will not have explicit stf.pl command line in playlist.xml.

2) Make STF have a parseArgs() method.

[terryzuoty]

To reply @llxia question,

JCK has the ability to pass in user defined EXTRA_OPTIONS, e.g.

<command>export JAVA_HOME=$(JDK_HOME)$(D)  && \
    perl $(TEST_RESROOT)$(D)..$(D)..$(D)systemtest$(D)stf$(D)stf.core$(D)scripts$(D)stf.pl  \
    -java-args-setup=$(Q)$(EXTRA_OPTIONS)$(Q) \
    -test-root=$(Q)$(TEST_RESROOT)$(D)..$(D)..$(D)systemtest$(D)stf;$(TEST_RESROOT)$(D)..$(D)..$(D)systemtest$(D)openjdk-systemtest$(Q) \
    -systemtest-prereqs=$(Q)$(JCK_ROOT)$(Q)  \
    -results-root=$(REPORTDIR)  \
    -test=Jck -test-args=$(Q)tests=$(JCK_TEST_TARGET),jckversion=$(JCK_VERSION),testsuite=RUNTIME$(Q); \
    $(TEST_STATUS)</command>

The JCK is using -java-args-setup=xxx, and I think other svt tests could use -java-args=xxx

[llxia]

• EXTRA_OPTIONS is not JVM_OPTIONS
• java-args-setup is only added in jck-runtime-custom test

[Mesbah-Alam]

@llxia, to clarify:
Is $(JVM_OPTIONS) == The command line specified in the <variation> tag in playlist.xml? If yes, is it an empty string when variation is NoOptions?

If so, then a simple workaround would be what Terry had indicated above, we simply have to add -java-args=$(Q)$(JVM_OPTIONS)$(Q) in all the STF command lines in playlist.xml.

[llxia]

yes, JVM_OPTIONS value is generated based on <variation> value. If JVM_OPTIONS is not passed into test cmd, adding <variation> has no effect.

[terryzuoty]

Currently, only jck-runtime-custom needs this option java-args-setup since user may want to add extra options for their testing purpose. Other jck tests are created as requested since they do not need other variants.

To reply @Mesbah-Alam $(EXTRA_OPTIONS) != The command line specified in tag. Since it only exists in custom test target, user need to define the EXTRA_OPTIONS themselves.

[llxia]

We have issue to run tests with different <variation> in playlist.xml. It has nothing to do with user defined options.

[Mesbah-Alam]

@llxia - your last comment leads me to think about the following:
What if we have multiple tags inside the test?

<variations>
    <variation>-Xjit:count=0</variation>
    <variation>-Xjit:count=100</variation>
</variations>

In the above case, will the infrastructure invoke the <command> twice, and have JVM_OPTIONS=-Xjit:count=0 in the first time, and JVM_OPTIONS=-Xjit:count=100 the second time?

[llxia]

In this case, we will have two test targets with different suffix. Please refer OpenJ9 test readme for detail.

[Mesbah-Alam]

FYI..
Just to try out the -java-args route: I added a sample <variation> in the systemtest playlist.xml (https://github.com/AdoptOpenJDK/openjdk-tests/compare/master...Mesbah-Alam:addModeToSysTest?expand=1)

It does ensure that the provided Java option gets passed onto the actual test command line: https://ci.adoptopenjdk.net/view/work%20in%20progress/job/test_personal/375/console

STF 21:23:14.445 - Running command: /home/jenkins/workspace/test_personal/openjdkbinary/j2sdk-image/jre/bin/../../bin/java -Xnocompressedrefs -Xjit -classpath /home/

However, it also passes an extra option -Xnocompressedrefs , which eventually makes the job fail:

DBLT Selected VM [default] by option -Xnocompressedrefs does not exist.
        STF 21:23:14.476 - **FAILED** Process DBLT ended with exit code (255) and not the expected exit code/s (0)

@llxia - Do you know why -Xnocompressedrefs is being passed in when it's not part of the options being set in <variation>?

[smlambert]

This issue talks about -Xcompressedrefs option gets appended, assume its similar for "no" option, looks at JAVA_IMPL.

[llxia]

@Mesbah-Alam , looking at your console output, you used JAVA_IMPL=openj9 and SPEC=linux_x86-64. That is probably why you have -Xnocompressedrefs appended.

[Mesbah-Alam]

The SDK the job is using is:

22:23:14 STF 21:23:14.279 - Running: /home/jenkins/workspace/test_personal/openjdkbinary/j2sdk-image/jre/bin/../../bin/java -version
22:23:14 openjdk version "1.8.0_162"
22:23:14 OpenJDK Runtime Environment (build 1.8.0_162-b12)
22:23:14 Eclipse OpenJ9 VM (build openj9-0.8.0-rc2, JRE 1.8.0 Linux amd64-64 Compressed References 20180308_114 (JIT enabled, AOT enabled)
22:23:14 OpenJ9   - e24e8aa9
22:23:14 OMR      - 3e8296b4
22:23:14 JCL      - ed7328e5 based on jdk8u162-b12)

What is not clear to me is why it complains about -Xnocompressedrefs:
DBLT Selected VM [default] by option -Xnocompressedrefs does not exist.

I logged into the machine and was able to reproduce the same problem:

jenkins@test-packet-ubuntu1604-x64-2:~$ /home/jenkins/workspace/test_personal/openjdkbinary/j2sdk-image/jre/bin/../../bin/java -Xnocompressedrefs -version
Selected VM [default] by option -Xnocompressedrefs does not exist.

The question is : does not the Linux x64 openjdk8-openj9 SDK support -Xnocompressedrefs?

[Mesbah-Alam]

Got the answer from Pete. It is yet not supported : https://github.com/eclipse/openj9/issues/479.

So, we should bump into the issue I faced above on all platforms where JVM_OPTIONS contains -Xnocompressedrefs by default, so long as the above issue is resolved.

[sophia-guo]

@Mesbah-Alam you should not have this issue. The issue has been fixed https://github.com/AdoptOpenJDK/openjdk-tests/pull/294

The problem you are facing is in your personal build. For personal build please setting SPEC as cmpressedref one.
/home/jenkins/workspace/test_personal/openjdkbinary/j2sdk-image/jre/bin/../../bin/java

[smlambert]

And if we want to be thorough, we can update the README for JVM_OPTIONS/EXTRA_OPTIONS that -Xnocompressrefs/-Xcompressedrefs values may be overridden depending on what SPEC you run with.

I am not certain how many of the set of pre-existing variations (a.k.a. modes) contained them, but any variations that contained "-Xnocompressedrefs" are invalid for testing the openj9 builds at this time.

[Mesbah-Alam]

@sophia-guo - thanks for pointing out how I should use the cr spec. I used the cr spec and the personal build works fine: https://ci.adoptopenjdk.net/view/work%20in%20progress/job/test_personal/383/console

[Mesbah-Alam]

So, to summarize, the change I need to make in all systemtest, jck and any other test playlists which run on STF, is to add -java-args=$(Q)$(JVM_OPTIONS)$(Q) \ in the command line. Note, this will only work when we have explicit command lines in the <variation> tag. Using modeXXX as value in <variation> will still result in modes being ignored.

@ShelleyLambert - Do we need STF to support modeXXX type values? If not, I will not worry about that at this point (it needs work on STF side for it to accept modeXXX style mode values. STF does offer an option called -mode via which you could pass in modeXX, however, currently, STF will fail to decode modeXXX values, as it depends on modes.xml and variations.xml for mode definition-- files which were not made available in Adopt for obvious reasons)).

[smlambert]

No, STF does not need to support modexxx type values.

In fact, we would like to shift all test playlists (including FV ones) away from using that approach, which adds an unnecessary level of indirection and which has no little meaning to open developers that do not know the background of it (and then would have to look at the modes.xml file to understand what each one mapped to).

[llxia]

Using modeXXX as value in will still result in modes being ignored.

To correct, using ModeXXX as value in will work as TKG will interpret ModeXXX into corresponding JVM options.

[Mesbah-Alam]

Pull request for this Issue: https://github.com/AdoptOpenJDK/openjdk-tests/pull/317

[smlambert]

I believe this issue is now resolved (with the exception of the extra enhancement for processing options with complex values, which is raised in a separate issue #321), so closing.