Openjdk-tests: Time-based system tests

[Mesbah-Alam]

We want a mechanism to allow open system tests to run a workload for 'n' number of minutes.

As step 1 - We will create a system test playlist target that exposes a parameter 'n' to specify the time duration the test workload should run for. Underneath: a mechanism will be implemented in STF where a mixed load will be run with number of iterations increased sufficiently enough to cause the test to keeps running - which will then will be capped after 'n' minutes of execution.

As for which load should be run - we will start off with mini-mix load or something equivalent to it. We may add parameters to allow users to specify test iterations or load mixture later, as the mechanism evolves.

Developers can run this target in grinder. At one point, we could incorporate timing-based system tests in the special.system builds also.

Key points:

• Tests should find bugs that are reproducible.
• Need to revisit the modes being used which find defects internally.

Related: https://github.com/AdoptOpenJDK/openjdk-systemtest/issues/294

@pshipton @lumpfish @ShelleyLambert @llxia @fjeremic

[pshipton]

We can run the new playlist the nightly builds as machine resources allow, assuming the tooling allows this, or once the tooling can be modified to support it. I think it's unlikely any developer will run the test, at least not on a regular basis. My understanding is the new playlist is for experimentation so we can get the timing based testing working, with the end goal modifying the special.system tests to use it.

[lumpfish]

I believe stf.load already has the functionality required via its timelimit option, which tells it to stop submitting new tests after that time has elapsed. I assume the number of tests needs to be set high enough so that they don't all finish before that limit - some testing can be done to confirm that.

To expose the timelimit to a playlist, the argument needs to be able to be passed from stf.pl to the LoadTest command line in the generated perl script which runs the test.

This can be achieved by using the STF -test-args command line option:

Each STF test has a pluginInit method

    public void pluginInit(StfCoreExtension stf) throws StfException {
    }

Most tests do nothing (like the example above) but you can change it to get test arguments from the stf command line via stf.pl -test-args="a=1,b=2"

Something like:

    public void pluginInit(StfCoreExtension test) throws Exception {
                // [1] and [2] are the default values if the args are not set via -test-args
        StfTestArguments testArgs = test.env().getTestProperties("a=[1]","b=[2]");
        String a = testArgs.get("a");   
        String b = testArgs.get("b");

So the load test(s) could be amended to accept, say, -test-args="timelimit=nnn" which would then be used in .setTimeLimit(timeout) when the LoadProcessDefinition is created.

[Mesbah-Alam]

@lumpfish : I've tried your suggestions locally. They are working fine:

...
...
LT  12:41:15.658 - Completed 100.0%. Number of tests started=10611 (+0)
LT  17:41:25.206 - Thread completed. Suite=0 thread=9
LT  12:41:25.288 - Test stopped due to reaching runtime limit
LT  12:41:25.288 - Load test completed
LT  12:41:25.289 -   Ran     : 10611
LT  12:41:25.289 -   Passed  : 10611
LT  12:41:25.289 -   Failed  : 0
LT  12:41:25.289 -   Result  : PASSED
STF 12:41:25.608 - Monitoring Report Summary:

I will have the PRs submitted for review shortly.

[pshipton]

Actually, not just special.system should be changed to be timing based, but more importantly sanity and extended (if applicable) as well.

Looking at a sanity.system run, I'm seeing some very short run times for tests that need to run longer. The following is just a subset.

2 - - DaaLoadTest_daa1_0 | 1 min 4 sec | OK | No | No
3 - - DaaLoadTest_daa2_0 | 57 sec | OK | No | No
4 - - DaaLoadTest_daa3_ConcurrentScavenge_0 | 15 sec | OK | No | No
11 - - LambdaLoadTest_ConcurrentScavenge_0 | 33 sec | OK | No | No
12 - - MathLoadTest_autosimd_0 | 8.9 sec | OK | No | No
13 - - MathLoadTest_bigdecimal_0 | 46 sec | OK | No | No
14 - - MathLoadTest_bigdecimal_ConcurrentScavenge_0 | 50 sec | OK | No | No
15 - - MauveSingleInvocationLoadTest_OpenJ9_0 | 11 sec | OK | No | No
16 - - MauveMultiThreadLoadTest_0 | 1 min 43 sec | OK | No | No

[pshipton]

I can see how this needs to be carefully rolled out, as changing how the tests are run could expose a number of failures, resulting in non-green runs.

[smlambert]

Based on our discussion today, 1st step was to add an additional target (for the sake of argument, called something like MiniMix_5min). It can be added to extended level (and therefore would be run nightly).

Further changes definitely need to be considered more fully, whether we leave existing targets as is, and add extra time-based targets, or iteratively change-over targets changes the comparative-ness of those targets (and disconnects historical results in TRSS... though that is a minor point).

[smlambert]

Both PRs are merged, believe this is now addressed. Thanks @Mesbah-Alam and @lumpfish !

[pshipton]

Is there another issue open to cover fixing the other system tests which don't actually run long enough to find any problems, or should I be opening one?

[smlambert]

I think we should open an "assessment" issue where we visit each LoadTest (the more exhaustive list to what you've put in https://github.com/AdoptOpenJDK/openjdk-tests/issues/2104#issuecomment-742689685).

[smlambert]

Scratch that, I will reopen this one and we can list the entire set here.

[Mesbah-Alam]

fixing the other system tests which don't actually run long enough to find any problems

Here's what I understand that needs to happen:

1. Make every STF based load test capable of running a time based load by updating them in a way similar to how we updated the MixedLoadTest.
2. Update system test playlists for the load tests and specify a timeLimit (e.g. 5 minutes).

At the end of rolling this out - all STF based system tests will become time based, as opposed to load based. (Note: the tests would still be perfectly capable of running in their original load-based fashion manually / locally).

Does this sound reasonable?

@lumpfish @ShelleyLambert @pshipton

[pshipton]

Works for me. @fjeremic @gita-omr @0xdaryl for your opinions as well.

[fjeremic]

Sounds reasonable to me as well. Left an implementation comment as well:
https://github.com/AdoptOpenJDK/openjdk-systemtest/pull/384/files#r555330996

[gita-omr]

I think it's a good idea but I think we need to be ready for the failures to become less deterministic. Will old style tests still be running?

[lumpfish]

What @gita-omr says above. If only time based tests are run then some machines / platforms / modes will run more iterations of the tests so if a test fails in one environment but not another it may well not be due to the difference in machine / platform / mode.

[pshipton]

I think we should retire the old style (iteration based) testing. They have problems, either running for too short a time, or taking too much time. Although the failures will become less deterministic, we're likely to catch more problems by running the short running tests for longer. We may catch fewer problems by shortening some test runs, but I expect most problems will be caught by picking a shorter appropriate period of time, i.e. 5 - 30 min, rather than an hour+. Because traditionally we've used 5 min, I suggest we start with that, but if machine resources are available and it helps find additional problem we can make it longer. We can also look into having some test runs outside of the nightly builds which use a longer period of time.

Note legacy testing (i.e. vmfarm) doesn't use iteration based testing, but (5min) time based testing.

[pshipton]

Also note that just because a test runs for X iterations rather than for some fixed period of time, it doesn't make the test results deterministic. All JIT developers should understand that.

[gita-omr]

We definitely understand (and encounter every day) the non-deterministic nature of JIT related failures. I am not saying that time-based system is not good but it definitely adds another parameter to consider when we are trying to make some failure more predictable while debugging it. For example, we cannot step through time-based test case using debugger.

[pshipton]

Is it worth keeping iteration based testing enabled with a small number of iterations, that can be used under the debugger? It might be tricky to find the equivalent iteration based test. Perhaps a simple documented way to increase the time used for the test, or setting it to infinity when the debugger is used. I definitely don't want to keep the iteration based testing we have now that runs for 1+ hours.

[gita-omr]

I think we just need an option for the same test case to switch to some number of iterations. Basically, the way I see it: time based testing will increase coverage and efficiency of testing. But when some test case fails, we would need to see how many iterations happened before it failed, and then have an option to experiment with the number of iterations in order to debug it.

[Mesbah-Alam]

I think we just need an option for the same test case to switch to some number of iterations. Basically, the way I see it: time based testing will increase coverage and efficiency of testing. But when some test case fails, we would need to see how many iterations happened before it failed, and then have an option to experiment with the number of iterations in order to debug it.

I guess this can be done using a custom target which allows users to run any test in its original form (not time based).

To be precise, the original form of iteration based stf system tests can not really be run for a given 'n' number of iterations. They can be run using what's called 'load configurations' defined in xml files in which test names are listed that are to be repeatedly executed in random or in a specific order. However we should be able to play around with this mechanism and have something similar to what's being asked here.

This can happen as a second phase. The first phase may consist of all the changes that need to be made in order to get the time-based tests running in the builds.

[Mesbah-Alam]

All pieces of work related to the time-based system test have been delivered. Close this issue at this point. We will open new issues for any future work related to this.