Openjdk-tests: Tests leave processes on ppcle machines

[adamfarley]

A ps-aux command on test-osuosl-ppc64le-ubuntu-16-04-1 shows dozens of old test processes, some months old. Snippet below. They seem to come from our tests, so it's worth fixing that.

The vast majority were running classes with "Targ" in the name.

jenkins 30007 0.0 0.1 2562496 14272 ? SLl Mar17 0:26 /home/jenkins/workspace/openjdk8_j9_openjdktest_extended_ppc64le_linux/openjdkbinary/j2sdk-image/jre/bin/../../bin/java -ea -esa -Xmx512m -Xcompressedrefs -cp /home/jenkins/workspace/openjdk8_j9_openjdktest_extended_ppc64le_linux/jvmtest/openjdk_regression/work/classes/2/java/lang/management/MemoryMXBean:/home/jenkins/workspace/openjdk8_j9_openjdktest_extended_ppc64le_linux/jvmtest/openjdk_regression/work/classes/2/lib/testlibrary -XX:+PrintGCDetails CollectionUsageThreshold$TestMain
jenkins 30239 0.0 0.2 2374592 18048 ? SLl Mar03 0:11 /home/jenkins/workspace/openjdk8_j9_openjdktest_extended_ppc64le_linux/openjdkbinary/j2sdk-image/jre/bin/java -classpath /home/jenkins/workspace/openjdk8_j9_openjdktest_extended_ppc64le_linux/jvmtest/openjdk_regression/work/classes/2/com/sun/jdi -Xdebug -Xrunjdwp:transport=dt_socket,address=test-osuosl-ubuntu1604-ppc64le-1.adoptopenjdk.net:45820,suspend=y StackOverflowCaughtTarg
jenkins 30271 0.0 0.3 2562496 25984 ? SLl Mar24 0:19 /home/jenkins/workspace/openjdk8_j9_openjdktest_extended_ppc64le_linux/openjdkbinary/j2sdk-image/jre/bin/../../bin/java -ea -esa -Xmx512m -Xcompressedrefs -cp /home/jenkins/workspace/openjdk8_j9_openjdktest_extended_ppc64le_linux/jvmtest/openjdk_regression/work/classes/0/java/lang/management/MemoryMXBean:/home/jenkins/workspace/openjdk8_j9_openjdktest_extended_ppc64le_linux/jvmtest/openjdk_regression/work/classes/0/lib/testlibrary -XX:+PrintGCDetails CollectionUsageThreshold$TestMain

[smlambert]

When tests time out and do not complete properly, the underlying framework does not appear to kill the processes it starts (in this case underlying framework is jtreg). We should look at and understand what clean-up is done by jtreg when running openjdk tests (perhaps there is a command-line option that needs to be used, or a JBS bug that should be raised).

However, to protect ourselves against this type of problem, the easiest way to address this as a user of jtreg and other underlying test frameworks that may not have been able to clean up after themselves, is to check for running processes during the Jenkins test pipeline setup stage, and kill any strays before the start of the test job. This will then be a solution that should not only address processes left by jtreg, but additionally those left by other underlying frameworks that are used by the various tests we run.

[smlambert]

Someone should review the long list of issues that are found in the JBS DB, when searching jdi hang: https://bugs.openjdk.java.net/browse/JDK-4821208?jql=text%20~%20%22com%2Fsun%2Fjdi%20hang%22

To see if some issues reveal what we are hitting. I also note that some jdi tests appear to rely on HotSpotDiagnosticMXBean, which may explain why these are problematic against OpenJ9 builds (looking at the processes that are hung, they appear only to be the openj9 jobs, not hotspot).

[sxa]

For reference https://ci.adoptopenjdk.net/job/SXA-processCheck/ can be used to see the process listing in a "quiet" state on many of the machines. If a machine is red that usually means there are rogue processes left on it, potentially interrupting the successful execution of test runs.

[smlambert]

This will be addressed under https://github.com/AdoptOpenJDK/TKG/issues/45, so closing this issue in favour of the TKG one.

[sxa]

FYI The process check job mentioned above has been set to kill most rogue jobs for a while now so this has already been mitigated for most cases