Openj9: SDK archiving is very slow on zLinux machines

[AdamBrousseau]

Currently in all Pull Request builds, once the SDK is compiled, we "archive" the sdk so it can be downloaded by anyone from the Jenkins site. This was requested via #91. The problem is that the zLinux machines we are using, appear to have a very small pipe to the outside world and it can take 10 minutes to push a ~200M sdk back to the Jenkins master host machine. The pLinux machines generally complete the archiving in under 2 minutes.

The current implementation of the pull request build archive step is serially done between the compile and test steps.

The other, more severe use case, is when a compile and test happen on different machines. The SDK has to be pushed up to the master and then pulled down to the node(s) for testing. Currently the PR builds compile/test on 1 machine, but this may not always be the case, especially if/when we have cross compiled platforms.

Is it useful to keep the sdk for a PR build and how often are they being downloaded by users? If the answer is no and almost never, then we should just stop doing it to save the time. If this is not the case then I think we should investigate archiving it in parallel with the testing (after the compile completes). I think this should be an easy fix. Having the archive run at the same time as testing I imagine wouldn't be a problem. The other thing we could think about implementing is an optional archive. We could do this by adding an extra "comment flag" in the PR Trigger Comment to indicate you would like the sdk to be archived. By default the flag would be off.

[fjeremic]

Is it useful to keep the sdk for a PR build and how often are they being downloaded by users?

Personally I didn't even know this was an option.

If this is not the case then I think we should investigate archiving it in parallel with the testing (after the compile completes).

This sounds like a great improvement since the tests usually take more than 10 minutes anyway.

We could do this by adding an extra "comment flag" in the PR Trigger Comment to indicate you would like the sdk to be archived.

This is also a good solution.

[joransiu]

The problem is that the zLinux machines we are using, appear to have a very small pipe to the outside world and it can take 10 minutes to push a ~200M sdk back to the Jenkins master host machine.

Has anyone tried contacting the community system admins to see if this can be improved?

[AdamBrousseau]

Has anyone tried contacting the community system admins to see if this can be improved?

We have not. I didn't necessarily want to "complain" about something that is free. I do have a contact name I can reach out to but I wouldn't mind hearing from @mstoodle first since he was the one who got the machines.

[mstoodle]

We don't have to "complain" :) . We can just reach out to ask if that performance is unusual or "by design. Then we can decide if we want to change how we handle PR build archiving.

Sound reasonable?

[AdamBrousseau]

Email sent to Cindy asking about the network perf

[AdamBrousseau]

Paraphrased summary response from Cindy back in November was

• Not a known issue (nobody else has complained)
• They did have a CC (community cloud) issue a few months prior that was fixed in some network configs
• Machines are geographically located in a college in New York
• Not much they can do about network perf if the problem is a small pipe

While zLInux continues to be the slowest platform (by far) to archive the SDKs, we no longer keep them in the PR builds because we simply do not have the storage space at Eclipse (#1017). Therefore this is only a problem in the OMR acceptance and the nightly. Since zLinux isn't our slowest platform overall, this is likely not a pressing issue. I will close this but we can revisit later if we feel the need.