Openjdk-tests: Create a Disabled tests pipeline

[smlambert]

This test pipeline could check state of the issue that the test was disabled under and then run all disabled test targets whose issues are resolved to check if they really are in a state to reinclude/reenable.

Since there are various granularities on how we can and do disable tests, this work would look at several parts:

• [x] disabled test targets in playlists
• [ ] disabled openjdk tests in openjdk subdirectory (in problemlist*.txt files)
• [ ] disabled functional tests in openj9 repo (several types of exclusions, testNG, cmdlinetester, others?)

Likely do it as a pipeline that launched Grinders (similar to Grinder of Grinders, see https://github.com/AdoptOpenJDK/openjdk-tests/issues/1816).

One job was already added, but is a WIP, see https://ci.adoptopenjdk.net/view/Test_grinder/job/Test_jdk8_j9_Disabled_Weekly/

[d3r3kk]

At Microsoft we have a plan to do something along this line as well, and we would love to collaborate here.

Currently our idea is to:

For each platform/OS:

• Obtain the latest known good build (latest nightly)
• Obtain the ProblemList.txt files (there will certainly be more than 1).
• Parse the ProblemList.txt files to enumerate the list of tests.
• Run each test 1 at a time against the last known good build and record the status.
• Generate a report that shows the status for each test run
• Collate week-over-week runs of these tests to show trends toward success (candidates for re-introduction).

...something like that.

[d3r3kk]

cc: @junyzheng

[smlambert]

Great! since AQA invokes multiple underlying test frameworks there are several ways tests can be excluded.

You've identified how one could approach the openjdk test group. It will be good to break down the components of the solution into runnable units so we can use some of the same pieces for the different types of excludes.

Generically, we

• create a list of tests and the issues they are excluded under and platform
• check the state of the issue (if github, API is easy to call, if JBS, not entirely straightforward... does not allow for anonymous queries, believe it may need to scrape the webpage unless we can have a userid with access, or a different site that mirrors JBS that has API)
• if issue is still open, no action
• if issue is closed, initiate a rerun of the test on the platform(s)/versions(s) with latest nightly
• if rerun fails, log it but do not reinclude
• if rerun passes, can we automatically check if it was an intermittent issue? (likely not, unless there was a tag on the issue that could be queried). if intermittent issue, rerun XX times to verify
• for reincludes, generate a PR for reinclusion of test

Related to another issue, I was going to create a simple script called rerun.sh that could either be passed a rerun link or be rerun with particular env vars that we use (BUILD_LIST, TARGET, JDK_IMPL, JDK_VERSION, etc) to run a particular test case

This will give us flexibility to then run in any CI system (Jenkins/AzDo/laptop of devs).

[junyuan-z]

here is what I am thinking based on @smlambert 's comment.

1. find all the problemlist.txt files
2. iterate the problemlist.txt file line by line, send request to github, or openjdk bug tracker, or azure devops to check bug status.
   do nothing if it is still open, run test again if it is closed.
3. run test based on the intermittent tag, run N times if intermittent tag exist, otherwise only run once.
4. report the result and create PR by removing passed entry from the problemlist.txt file.

please let me know what you think.

I am planning to write a Python script for this and it will call grinder, etc.

[llxia]

Thanks @junyzheng . This is a great start. I got inspired by it and have a couple of thoughts:

• I would like to collaborate with you to enhance this to support playlist.xml. In this way, we are covering all tests
• I would like to create auto PR for exclude tests as well. Something like, keywords in git issues (i.e., exclude test testAAA) triggers Jenkins job which git clone repo and edit playlist.xml/problemlist.txt to exclude the test, and create auto PR using git API
• If an issue is closed, it does not mean it is fixed. (i.e., it can be closed as dup). Even if the issue is fixed, we may need to make sure the fixes are in the SDK we are using. There may be a delay in the nightly build. Maybe we can check SHA; only query issues that are closed 3 days ago or get other info to verify this.
• We may need to create/reopen the issue if the problem still exists

[llxia]

PR: https://github.com/AdoptOpenJDK/openjdk-tests/pull/1978
PR: https://github.com/AdoptOpenJDK/TKG/pull/105