Openjdk-tests: Archive test results from docker image from external tests

[Mesbah-Alam]

Archive the test results from docker images from external test builds in our Jenkinsfiles so that user can download them from Jenkins and perform debugging investigation on test diagnostics stored inside the docker image.

[smlambert]

Actually changing this slightly, rather than archive the entire Docker image, to extracting the test results from it, and archiving only them. Improve ability for easier triage, and viewing of test results).

[smlambert]

FYI @mamatha-jv - as discussed last week, if you update the example in thirdparty_containers to extract the results from the docker image, I can update the CI scripts to archive and display the results for easy viewing and download.

[Mesbah-Alam]

Looking at https://ci.adoptopenjdk.net/view/External%20tests/job/openjdk9_hs_externaltest_x86-64_linux/21/console:
While the job is running, if I log into the jenkins slave, I see that a text file called TestTargetResult is created under the workspace folder (/workspace/openjdk9_hs_externaltest_x86-64_linux/openjdk-tests/TestConfig/test_output_15289992106364). This file already contains the text output of the whole job run so far (as seen on the "console output" view from web UI).

TestTargetResult file (as defined in openj9/test/TestConfig/settings.mk) is later on passed into resultsSum.pl to be processed for tap output, and subsequently deleted.

@ShelleyLambert , Are we looking to archive a file containing the text output of the docker based external tests? If yes, then TestTargetResult contains that output, and we could be archiving that!

---

Just an FYI..

I tried out the other option of mounting a folder from the host machine onto the docker image and getting the test to output the result in that folder too. Here's what I tried:

• Updated openjdk-tests/thirdparty_containers/example-test/playlist.xml and added a second mount point in the docker run command as : -v $(REPORTDIR)/:/report. ($REPORTDIR points to the job output directory on disc of the hostmachine).
• Updated test executable to print a report text file in the '/report' folder (e.g. java -cp ./src net.adoptopenjdk.example.test.ExampleTest >/report/output.txt 2>&1).

This drops output.txt onto $(REPORTDIR) on the host machine. We could take this route and archive such reports generated by tests; but in that case, we need to update all the external tests to generate output in the mapped directory.

[smlambert]

Couple of notes:

• goal is to provide a solution that helps expedite triaging of failures
• like our other types of tests, we should not archive test results if they pass
• in case of unstable/failed tests, then it would be good to make available both the
  
  
  
  • TestTargetResult contents (if that is the one the contains .tap files)
  
  
  • actual application test output (in many cases app tests use Junit, and it would be good to get those output files)
  
  
• it is less interesting to archive the results (in a zip/tar.gz form that someone has to download, then unzip to view) then to view results in Jenkins GUI (via the TAP and Junit plugins that let us see summarized views if we provide .tap or .xml/.html files - see https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/buildenv/jenkins/JenkinsfileBase#L193 and L194 (those lines tell the plugins to display the contents)

Archiving can also be done, but is less important. People do not like the multi-step process (download/unzip) to be able to see results, only benefit would be to attach output to defects, but only valuable in cases of intermittent failures which we see very few of in these tests.

[Mesbah-Alam]

Thanks for clarifying.

it is less interesting to archive the results (in a zip/tar.gz form that someone has to download, then unzip to view) then to view results in Jenkins GUI

That will be ideal for debugging purposes.

I was looking to know why we do not have the tap output for failed tests (like we have for other (e.g. systemtests)) in case of external tests. The output details for external tests are actually turned off, here:
https://github.com/AdoptOpenJDK/openjdk-tests/blob/6c01a59152f7671094f51fecd6ab074a81b7461a/buildenv/jenkins/JenkinsfileBase#L73.

Sophia mentioned that this was since some external tests were producing corrupt output which was causing tap output generator to malfunction. Since it was done in March, we no longer have the builds to check which test exactly was causing the issue. I will be investigating this in a personal build.

[smlambert]

As per our discussion, I agree, for each of the 3rd party app tests, we should check if there is a way to direct the test output to be junit style, and be put in a common location.

Example:
scala tests: "-u - causes test results to be written to junit-style xml files in the named directory" from http://www.scalatest.org/user_guide/using_scalatest_with_sbt

[Mesbah-Alam]

@smlambert , agreed.

Here's the link to the Lucene-solr tests where the junit report is being displayed as we are mapping the output folder containing the junit xml reports onto the host machine : https://ci.adoptopenjdk.net/view/work%20in%20progress/job/test_personal/614/testReport/

[smlambert]

wonderful! that's both a great improvement and great progress! 👍

[smlambert]

Wildfly test results info (taken from Wildfly tests user guide):

Creating test reports
Test reports are created in the form known from EAP 5. To create them, simply run the testsuite, which will create Surefire XML files.

Creation of the reports is bound to the site Maven phase, so it must be run separatedly afterwards. Use one of these:

./integration-tests.sh site

cd testsuite; mvn site

mvn -pl testsuite site

Note that it will take all test results under testsuite/integration/ - the pattern is */TestCase.xml, without need to specify -DallTests.

[smlambert]

Jenkins test results info (taken from Jenkins developer doc):

Running Tests
From the Command Line
mvn test will run all test cases, report progress and results on the command line, and write those results to JUnit XML files following the pattern target/surefire-reports/TEST-<class name>.xml

[Mesbah-Alam]

There is an issue in mapping output reports directory of the docker image onto the host machine, in that, the files that get copied over down to the host machines are owned by root, whereas the user that tries to wipe clean Jenkins workspace directories in the host machines uses jenkins user. This causes workspaces to not get cleaned up properly.

We need to run chown inside the docker container to change the owner from root to jenkins (for which we need to have a user called jenkins with the same UUID inside the container).

This issue is being tracked by https://github.com/AdoptOpenJDK/openjdk-tests/issues/475.

[smlambert]

Closing this in favour of #976