Temurin-build: Executors on ppcle64 build boxes can cause deadlock

[sxa]

The https://ci.adoptopenjdk.net/job/openjdk8_openj9_nightly_pipeline/ job was running on the executor on https://ci.adoptopenjdk.net/computer/build-osuosl-ubuntu1604-ppc64le-1/ - As part of that pipeline it tries to run the https://ci.adoptopenjdk.net/job/openjdk8_openj9_build_ppc64le_linux/ job which can currently only run on that box. This meant that the executor was never available to run the bulid. I have temporarily increased the executors to 2 to allow it to run and dropped it back but we need to do something to stop the pipeline job executing on machines of which there are no alternative systems.

[gdams]

This shouldn't be so much of an issue once we can share ppcle machines between hotspot and openj9 builds but I agree that we shouldn't be blocking up executor time on valuable ppcle machines when we could be running jobs on x64 linux

[smlambert]

Thanks for this @sxa555 !

[AdamBrousseau]

It sounds like the pipeline code needs to be fixed so that it is not occupying an executor when it is trying to launch downstream builds.

[smlambert]

Example of one of the build pasted below. These pipelines are not in a repo currently, though they should likely be put in one. They can/should be updated to not grab and run on a node that one will launch the subsequent actual job on...

---

stage 'build OpenJDK'
def Platforms = [:]
Platforms["Linux"] = {
node {
build job: 'openjdk9_openj9_build_x86-64_linux', propagate: false
}
}
Platforms["Windows"] = {
node {
build job: 'openjdk9_openj9_build_x86-64_windows'
}
}
Platforms["s390x"] = {
node {
build job: 'openjdk9_openj9_build_s390x_linux'
}
}
Platforms["ppc64le"] = {
node {
build job: 'openjdk9_openj9_build_ppc64le_linux'
}
}
Platforms["aix"] = {
node {
build job: 'openjdk9_openj9_build_ppc64_aix'
}
}
parallel Platforms
stage 'checksums'
node {
def job = build job: 'openjdk9_openj9_build_checksum'
}
stage 'publish nightly'
node {
def job = build job: 'openjdk_release_tool', parameters: [string(name: 'REPO', value: 'nightly'), string(name: 'TAG', value: 'jdk-9+181'), string(name: 'VERSION', value: 'jdk9-openj9')]
}

[karianna]

Hmm, I wonder where pipeline code should live. I forget, we made the pipelines only visible to a subset of logged in users? Or to all users? Is there a security concern around us posting the pipeline code in a GitHub repo?

[smlambert]

They could live in the openjdk-build repo, but I do not know the history of these pipelines if there was a reason to hide visibility. I assume they are hidden to stop anyone from launching/running them, rather than from looking at what they do, as they simply run publicly visible build jobs, then call the release tool to upload the build to the website. Even if they are in a repo, we can still keep the Jenkins jobs touchable by admins only.

[karianna]

I'm adding the groovy scripts now #261

[AdamBrousseau]

Likely resolved by the new build scripts?

[karianna]

Yes, we haven't seen this occur in the new world