Temurin-build: Create a debug symbols-only package

[milderhc]

Option to create debug symbols in a different archive than within the JDK one.

We can either create the JDK archive with or without the debug symbols but there isn't an option to create them separately.

[smlambert]

@milderhc - it looks like you have made great progress on this (with #2043 merged), thanks!

Are the next steps to:

• update the build pipeline scripts, so that these artifacts are produced and pushed to the releases repo
• served up via the API
• visible on the website for download

Will you be taking that on? Do you need any support to move this forward?

[milderhc]

@smlambert Sure. I can take those.

Could you tell me where the pipeline scripts are? And also, the idea in the API would be a new method that serves the debug symbols exclusively, as well as in the website?

I'm not familiar with any of those repos, it would take a while.

[smlambert]

Thanks @milderhc! Pipeline scripts are in this repo. Though, as I look at one of the PRs that added support for debugimages for OpenJ9, I think a bunch of the work may already be in place... but defer to build team to give guidance.

@andrew-m-leonard @sxa - Would one of you be able to assist @milderhc ?
Is all that is needed to actually update the pipeline config to build the debug images?

API work: (work is done) https://github.com/AdoptOpenJDK/openjdk-api-v3/pull/130
Website work: (still outstanding?) https://github.com/AdoptOpenJDK/openjdk-website/issues/696 (this is less critical than being able to serve up debuginfo through the API)

[andrew-m-leonard]

Sure I can help. @smlambert @milderhc I wanted to check if the following bug I helped integrate into OpenJDK for jdk-16 is also relevant here?
https://bugs.openjdk.java.net/browse/JDK-8252233 Put debug symbols in symbols-image

[smlambert]

I would guess it is relevant @andrew-m-leonard - can you outline what, if any further changes are needed to produce these artifacts at the project (for hotspot builds)?

[andrew-m-leonard]

So the change https://bugs.openjdk.java.net/browse/JDK-8252233 enabled the production of a pure debug symbols image via the target:

make symbols-image

Which will result in a "build/*/images/symbols" image
The default "make product-images" which the Adopt builds use includes "symbols-image", so you should find we already have a symbols image folder being produced, we just need to zip it up... and archive.

[smlambert]

@andrew-m-leonard - is that make target only for jdk16+ or for all versions (is it possible for all versions, or what is the debuginfo artifact that should be produced for pre-jdk16)? perhaps you can point to the place in the code where one would make the change to enable this given https://github.com/AdoptOpenJDK/openjdk-build/issues/2042#issuecomment-742898474

[andrew-m-leonard]

@milderhc hi Milder, i've been having a look at what upstream openjdk supports and currently they vary by version:
jdk16+ : With support in https://bugs.openjdk.java.net/browse/JDK-8252233 you can now simply issue "make symbols-image". The "symbols-image" is part of the "product-images" make target, and for jdk8+ Adopt already builds that target: https://github.com/AdoptOpenJDK/openjdk-build/blob/87f799024a58d392255eead1cd83d93515410b2e/configureBuild.sh#L206 . This means for jdk16+ we already build the "build/*/symbols/.." image, we just need to archive it at the end of the build, so this line needs to be moving the "symbols" folder: https://github.com/AdoptOpenJDK/openjdk-build/blob/88c0d77255264cc92423d534de9c3563d4fb0903/sbin/build.sh#L632 ie."${BUILD_CONFIG[DEBUG_IMAGE_PATH]}" for jdk16+ needs to be "symbols".

jdk11u & jdk8u: So both of these support make "symbols-image", however they do not have JDK-8252233, and so it does not produce an image bundle, but just places the symbols in the jdk image "lib" folder. This means we have two options here:
1) Get JDK-8252233 backported and use the same method as jdk16+. This is probably ideal, but will take time...
2) Add logic to sbin/build.sh to move the symbols from lib to a new location, it looks like you have already done this? https://github.com/AdoptOpenJDK/openjdk-build/blob/88c0d77255264cc92423d534de9c3563d4fb0903/sbin/build.sh#L651 Does this work for jdk16+ ? if so then maybe ignore JDK-8252233 for the moment.

I see now you added a new build arg: --create-debug-symbols-package, so I think the question to ask is when do we want this set to true? always? if so it's just a matter of setting it in: https://github.com/AdoptOpenJDK/openjdk-build/blob/master/build-farm/make-adopt-build-farm.sh
If it only wants setting on specific platforms or versions then in here: https://github.com/AdoptOpenJDK/openjdk-build/tree/master/build-farm/platform-specific-configurations

[andrew-m-leonard]

@milderhc another important fact that needs evaluating before we enable --create-debug-symbols-package, is how much more space is the symbols archive going to add to our builds, for capacity planning? As this is going to add xMB per osbuild x variant x version... ?
@sxa fyi

[milderhc]

Hi @andrew-m-leonard, thanks for looking into this and for the detailed explanation.

The --create-debug-symbols-package option works in jdk8u, jdk11u, and jdk16+. We can include it directly in make-adopt-build-farm.sh which I believe is called by the build pipeline?

The space in MB for jdk11u in Linux is around 230MB and around 40MB for macOS and Windows. In jdk8u is around 100MB for LInux and 5MB for Windows. Sizes for jdk16 are pretty similar to jdk11u.

[andrew-m-leonard]

@milderhc hi, one quick question, what is the difference between the current openj9 "-debug-image" and the new "-debug-symbols" ?

[andrew-m-leonard]

isn't openj9 "debug-image" the same thing? https://github.com/ibmruntimes/openj9-openjdk-jdk11/blob/cf49ddabb9e46e90af63e85f03e3e1e71338f6a2/closed/DebugImage.gmk

https://github.com/AdoptOpenJDK/openjdk-build/blob/88c0d77255264cc92423d534de9c3563d4fb0903/sbin/build.sh#L410

[andrew-m-leonard]

it would make sense to call Hotspot one "debug-image" as well?

[andrew-m-leonard]

So some rough calculations on size:
Platforms that cache builds for each version:

• arm : 5x230MB = +1.15Gb
• pLinux : 5x230MB = +1.15Gb

- mac : 5x10MB = +50MB

Platforms that do not cache currently:
AIX : ws cleared
zLinux : ws cleared
Win : only caches last build in tmp ws
xLinux : Docker
aarch64 : Docker

"master" node archive storage for all versions x platforms:

jdk8:
- 11 Linux = 1100MB
- 5 mac & win = 25MB
jdk11:
- 9 Linux = 2160MB
- 4 mac & win = 160MB
jdk15:
- 7 Linux = 1680MB
- 3 mac & win = 120MB
jdk16:
- 7 Linux =1680MB
- 4 mac & win = 160MB
jdk17:
- 7 Linux = 1680MB
- 4 mac & win = 160MB

So adding roughly an extra 1.1-2.3Gb increase in size of each new archived pipeline build on Jenkins "master"

@sxa fyi

[milderhc]

I was asking the same myself. I think they are the same. In fact, if we call the script for openj9 along with --with-debug-symbols-package we might get two packages. It makes sense to rename debug-symbols to debug-image as well.

[andrew-m-leonard]

I was asking the same myself. I think they are the same. In fact, if we call the script for openj9 along with --with-debug-symbols-package we might get two packages. It makes sense to rename debug-symbols to debug-image as well.

yes, agree

[andrew-m-leonard]

I was asking the same myself. I think they are the same. In fact, if we call the script for openj9 along with --with-debug-symbols-package we might get two packages. It makes sense to rename debug-symbols to debug-image as well.

So I think we need to update the logic so for openj9 it uses the current "debug-image" make target as it currently does, and for the other variants uses the new logic you have added to create their "debug-image".

[milderhc]

Ok, I will update that first.

[andrew-m-leonard]

@milderhc hi Milder, how is it going?

[milderhc]

@andrew-m-leonard hi, sorry for the late answer. I created the pull request: https://github.com/AdoptOpenJDK/openjdk-build/pull/2393

[andrew-m-leonard]

@milderhc thanks, that PR looks good. Once that is merged it's just a matter of:

1. Perform Jenkins infrastructure capacity evaluation/impact: https://github.com/AdoptOpenJDK/openjdk-build/issues/2042#issuecomment-756665091
2. Having completed (1), adding --create-debug-image if variant != "openj9" somewhere around here: https://github.com/AdoptOpenJDK/openjdk-build/blob/102237341c7f0737f0dd4dc57fcc7e9e3ffe3bd5/build-farm/make-adopt-build-farm.sh#L164

[andrew-m-leonard]

The build output cleaning PR is now merged, and needs to propagate across the build nodes to take affect. Currently the build nodes with the most marginal space are the 2 arm nodes, with spare space of 3Gb, 4Gb respectively as of now. Introducing at this at the moment would marginalize this 2 machines by about another 1.2Gb.
I will examine each to check for any dedundent usage and monitor nodes once the clean build output has propagated across them.

Having just examined build-scaleway-ubuntu1604-armv7-1, the build output cleaning should release 8Gb from it.

@sxa fyi

[andrew-m-leonard]

Submitted build pipelines for arm on jdk8/11/16/17
New arm node space:

• build-scaleway-ubuntu1604-armv7-1 : 18Gb
• build-scaleway-ubuntu1604-armv7-2 : 19Gb
  So we've freed up at least 6Gb
  This easily gives us enough headroom for debug-images.

[andrew-m-leonard]

@milderhc Would you like to propose a PR to enable --create-debug-image ?
https://github.com/AdoptOpenJDK/openjdk-build/blob/ecb4ce1182b079fb88cc77ce871e3ad1216e4612/build-farm/make-adopt-build-farm.sh#L163

[milderhc]

Thanks, @andrew-m-leonard, would that be enough for the debug-images to be downloadable?

[milderhc]

Here it is: https://github.com/AdoptOpenJDK/openjdk-build/pull/2443

[andrew-m-leonard]

Nightly builds for hotspot with debug-image's are now available from the website: https://adoptopenjdk.net/nightly.html?variant=openjdk8&jvmVariant=hotspot

[andrew-m-leonard]

Also via the API eg.: https://api.adoptopenjdk.net/v3/assets/feature_releases/8/ea?architecture=x64&heap_size=normal&image_type=debugimage&jvm_impl=hotspot&os=linux&page=0&page_size=10&project=jdk&sort_method=DEFAULT&sort_order=DESC&vendor=adoptopenjdk

[andrew-m-leonard]

Things remaining:

• Upstream mac debug jdk8 patch
• Upstream win debug jdk8 fix
• Upstream AIX debug fix for all versions