Openjdk-tests: Parent Jenkins test pipeline result should match status of worst child test pipeline

[AdamBrousseau]

Describe the bug
When a Jenkins test jobs is launched with IS_PARALLEL, child jobs are launched in parallel. When these children return the parent job's result will be set to FAILURE if any of the children are not PASSED. In my opinion, the parent status should be equal to the worst child's status. For example, if all but 1 children are PASSED (green), and a single child is UNSTABLE (yellow), the parent should also be yellow, whereas it currently would be FAILED (red).

To Reproduce
Launch a target in parallel and have one of the children fail (red) or abort (grey).

Expected behavior
Parent status should match worst child status.

Screenshots
If applicable, add screenshots to help explain your problem.

You can see in the screenshot above, 1 child was UNSTABLE but the parent was marked as FAILURE.

Additional context

[AdamBrousseau]

Not sure what the priority of this would be or how it fits with #1563

[llxia]

This is default Jenkins pipeline behavior. I think Pipeline-Build-Test-All has the same behavior as the parent Jenkins test pipeline.
https://ci.eclipse.org/openj9/job/Pipeline-Build-Test-All/814/

ERROR: Pipeline_Build_Test_JDK8_ppc64_aix #1365 - Nightly completed with status UNSTABLE (propagate: false to ignore)
Finished: FAILURE

That being said, I would like to understand how parent status matches the worst child status could help us. In the worst case, if the child builds have all different states (i.e., UNSTABLE, FAILED, SUCCESS, ABORT), what the parent state should be? I think users still have to navigate into the child build to understand what went wrong.

[AdamBrousseau]

Agree it is default Jenkins behaviour. Agree PBT-All behaves the same way. Where we do it "right" is the second level down in Pipeline-build-test-<version>-<platform>. If any compile/test job is not green, we match the status at this level. Code is a bit messy because there's lots more going on but here it is. The implementation for Test pipelines or PBT-All would be much more simple than this.

[AdamBrousseau]

Some info here about RESULT
https://javadoc.jenkins-ci.org/hudson/model/Result.html

[smlambert]

I do not feel comfortable diverging from the default behaviour of Jenkins. We have other consumers of AQA and it would be very unexpected for most users for us to change the default behaviour. It also feels a bit like taking an open-source tool and making it proprietary. For these reasons, I would vote to close this issue as "won't address".

I will leave the issue open for a bit longer to hear more input and/or any other reasons we would diverge from default Jenkins behaviour, in case we are missing something compelling.

[karianna]

Pinging @johnoliver in case he has some thoughts here - I know he was looking at unwinding some of our groovy code since Jenkins fixed some of the pipeline return code logic.

[smlambert]

My preference is to not add this type of custom code to every pipeline on the planet, but if considered valuable, create a PR in Jenkins repo.

@johnoliver has put a comment in the related item https://github.com/AdoptOpenJDK/openjdk-build/issues/1564 saying that they now have that code, and blue ocean view is using it.

If I am not mistaken, this request is based on the desire for some chiclets/badges to appear yellow instead of red, etc. So, the right approach would be to raise an issue at the Jenkins repo to request that the badges also convert to use the newly available api (as blue ocean now does), or better yet, create a PR for them for that functionality.

[johnoliver]

On that issue we would be waiting on https://issues.jenkins-ci.org/browse/JENKINS-58783 although that claims that the PR that has already been merged partially resolves this, so its possible that it could be done now.

[AdamBrousseau]

I don't think it's that issue. I think it's this
https://issues.jenkins-ci.org/browse/JENKINS-49073?focusedCommentId=380598&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-380598
Which was released in Pipeline: Build Step on March 17.
I haven't tested yet but it we are scheduling an upgrade of all our plugins this week. I will update with my findings when available.

[karianna]

We need to do an upgrade of Jenkins, I'll try to schedule one in.

[smlambert]

Thanks @AdamBrousseau and @karianna !

[smlambert]

The Jenkins update has fixed this issue, confirmed that parents now display correct status based on worst of children status.

[smlambert]

Leave this open until we also upgrade Adopt Jenkins server (which is why I assume @karianna has assigned it to himself).

This was confirmed on both internal server and openj9 server (https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_x86-64_linux_Nightly/371/)

[smlambert]

this has now been addressed by updating Jenkins version that supports it.