Openj9: Discussion: How to organize tests to support Java 8, 9, and java.next?

[DanHeidinga]

The current approach to test code management leads to duplication between tests targeting Java 8, tests targeting Java 9, and eventually those targeting Java.next (currently 18.3).

As Java changes to release more frequently with releases occurring every 6 months, we need to take a hard look at the current code duplication in the test projects or we'll have a significant problem when there have been a couple of these 6 month releases.

Some initial ideas / options to start the discussion:

• Branch the tests code for each Java release (awkward in current repo)
• Separate the test code into a separate repo (awkward for coordinated changes between JVM & tests)
• An inclusion / exclusion mechanism based on the Java level (awkward for methods that are only available in release of Java)

Hopefully this kicks a broader discussion on how to manage multiple test levels.

[pdbain-ibm]

We also need a common utility project for generic utility classes. An example of this is the attach API utility code duplicated in the Jav8AndUp and Java9AndUp projects. The rule here would be that it would be legal (i.e. no feature dependencies) on a base Java level such as Java 8.

[terryzuoty]

I think the the baseline of how test codes are organized is depending on how JCL codes are organized.

For example, if we have a repo and all its contents are designed for Java9, then anything designed only to Java8 and below should be removed from that repo.

One question is that how are we going to organize all the JCL code. If we need to add new APIs for java.next for some classes which already existed in Java8 and Java9, are we going to put all the JCL code together, or split them into different branches, repos etc? Test codes (as least FV test and unit test) need to follow the same organization.

[DanHeidinga]

The way JCL (java class library) code is organized depends on which repo it's part of.

Class library code that is tightly tied to OpenJ9 and lives in the openj9 repo will have a single version of the code that is preprocessed to generate the 8, 9, and java.next source.

Class library code that lives in the upstream OpenJDK extension project (runtimes/openj9-openjdk-jdk9) will likely have one repo per Java release. At least for 8 and 9. I'm not 100% clear on how the repos will evolve for the new OpenJDK release model of a new drop every 6 months.

[DanHeidinga]

One other factor to consider here is which releases are Long Term Support (LTS) releases. Currently, Java 8 is LTS while both Java 9 and 18.3 will not be. The next LTS release will be 18.9.

Does this impact our test organization? We need to keep LTS test levels around for longer than non-LTS levels. While there may be s short period of overlap while i.e.: both 9 & 18.3 are active, the usual case will be 3 active streams - Java 8 LTS, the current dev stream, and the 18.9 LTS.

This may alleviate some of the pressure to support multiple test versions / levels.

[smlambert]

Ideally, the general rule of thumb should be whatever tests can live in Java8andUp should live there, then Java9andUp contains tests for just those methods features that are introduced in Java9. Common code that is needed by both can live in a/the Utils dir (though it will be good if we consider what options we have for testing functionality (versus system level testing), redesign and err towards tests that require less supporting code, such as separate servers, etc, as these are typically 'flakier' during execution).

Currently, I am aware of 2 groups of tests that have a large portion of duplicate code, the attach api tests and the jsr 292 tests. Are there other tests that fall into the category of carrying a lot of duplication? Should we consider rewriting or restructuring these tests differently to not need such duplication?

I agree that each of the options presented is fraught with its own awkwardness. Duplication (same repo) versus potential version skew (different repo) versus opaqueness causing confusion (preprocessor mechanism). I do not think there will be any choice that will be wholly satisfying. In the immediate term, I think we can address some of the duplication we have with a restructuring effort.

[DanHeidinga]

I've been thinking about this over the last few days in relation to union file systems and multi-release jars. (Aside: union file systems are one of the tools that enables Docker images use to enable the layers created in a docker file to add new items, shadow items, or remove items from lower levels.)

If we viewed each supported java version as its own layer, with later versions layered over top of earlier ones, we would have a picture like this for Java 18.3:

----------------------------------------
|     |   B    |      |      |  Java 18.3 (Replaces B)
----------------------------------------
| A  |   B    |   C  |      |  Java 9 (Replaces A, adds B & C)
----------------------------------------
| A  |        |       | D   | Java 8 (the base layer)
----------------------------------------

(Sorry for the ascii art)

So Java 18.3 would run the following tests { Java 9:A, Java 18.3:B, Java 9:C, Java 8:D } which sounds both obvious and simple. The complexity comes with making this work in the world of files, java compilation, and including / excluding of tests.

An easy way to map this is one test per file but that doesn't match the reality of the existing tests. There are many files that have multiple tests in them for perfectly valid reasons - ease of test development, maintenance, only having to look in one place, sharing setup, and on and on.

When one of those tests no longer applies to newer versions, a decision has to be made on how to support the test suite (file?). As Shelley mentioned, the easiest path forward (and the one we've taken for 292 and attach tests) has been to copy the tests and maintain two copies of the source but this is unsustainable.

Looking at this problem through the union fs goggles, the answer appears to be to move the tests that don't apply to the new N release into their own file and only run that file for N-1 releases and delete (exclude?) that file from N releases.

What would this look like in a repo?
Maybe a /tests/common/ directory where tests applicable to all supported java levels are added either as files or in subdirectories.

When a test is no longer applicable to all levels, an empty file is placed in the level that doesn't support the test. i.e.:

/tests/common/A.java  // tests for all levels
/tests/java18.3/A.java   // empty file to shadow the common version.

If only a set of subtests aren't applicable, then they can be moved to a new file:

/tests/common/A.java  // tests for all levels
/tests/java8/A_subset.java  // subset of tests from A that are not applicable to other versions
/tests/java18.3/A_subset.java   // empty file to shadow the Java 8 version.

The empty file overrides are similar to the way the OpenJDK extensions for OpenJ9 project removes certain OJDK files so it follows an existing and understood pattern.

It's a lot of work to get to a model like this given where the projects at now, but I thought it was worth throwing the idea out there.

[mstoodle]

It's funny. This model is reminiscent to how the OMR and OpenJ9 JIT technology defines its classes in layers.

There are classes in OMR (common and per-platform), classes in J9 (common and per-platform). In principle, one can add additional layers on top of/below J9 too (depending on how you draw your class hierarchies :) ).

A particular class (say Instruction) is composed in two ways: the IPATH describes the order of the layers, and then the classes themselves define what concrete classes they extend. If we're building a JIT for the z platform, for example, the IPATH has the conceptual form of "J9 z platform":"J9 common":"OMR z platform":"OMR common"). These layers are top-level directories within the compiler component. The first layer on the IPATH that provides a header file being #include'd (say Instruction.hpp) will then directly #include and explicitly reference the concrete classes that will compose the full Instruction class. So you build an Instruction class by J9::Z::Instruction extending J9::Instruction extending OMR::Z::Instruction extending OMR::Instruction.

I've glossed over quite a few details and the actual sequence of extensions isn't quite as described, but hopefully you can see the basic idea and I didn't make it too confusing. The model Dan describes doesn't need all the complexity of the compiler's extension mechanisms, but it has the same flavour (to me, at least :) ).

In this case, there is presumably a per-JCL level set of test lists sitting next to each of the rows in Dan's diagram. Those lists define the tests (columns) that "matter" to a particular JCL level. Then it's the "highest" entry in each of those columns looking down from the current "row". If you have such test lists, then you probably don't have to introduce "empty" files: just remove them from that row and higher lists. But if you want test self-discovery, then I suppose you might need to add the empty files.

In pre-Java 9 parlance, the class path could specify the order of "row" directories to search. Below each directory, you can layout the tests in a consistent set of directories and Java files. So you run a test that attempts to load jsr292.TestInvokeExact1 and it finds that based on the classpath which could be something like tests/java9;tests/java8;tests/common (if you're running tests for Java 9).

To load jsr292.TestInvokeExact1, the class loader would look for tests/java9/jsr292/TestInvokeExact1.java then tests/java8/TestInvokeExact1.java then tests/common/TestInvokeExact1.java

you could also introduce the possibility of platform specific stuff via either the class path entries (which is how the JIT decided to do it) or by making directories akin to jsr292 . I doubt that's as important an issue for Java testing as it is for the JIT.

[DanHeidinga]

There are no new ideas - just the same ideas applied differently =)

[smlambert]

The ascii art is useful, no need for apologies! :)

What I like about the direction of this conversation is that I believe that we can try the approach (of a common dir, which I was considering to be Java8andUp, but to incorporate utility classes can be its own 'layer' called common) without having to convert all the tests at once...

The gotchas will be in the details (but the main 'gotcha' was the compilation story, which we have recently improved, enhancing the test framework to allow us to compile selected dirs if we want, rather than compiling everything in one step/target).

[mstoodle]

Yes @DanHeidinga the problems are similar, so the same ideas can be brought to bear on it. It's always satisfying, somehow, when there are recurrent themes and therefore reusable mental models :) .

[smlambert]

related #592

[smlambert]

Closing this discussion now, as I believe that we have a satisfactory/working approach to this (summarized above), with the addition of TestUtilities (Dan's /test/common idea and in fact is likely a better name) which can be used for both reusable test code and in the event that the Base test class can not be housed in Java8AndUp.