Openj9: OpenJ9 JLink produces a bigger JRE than Hotspot

[adamfarley]

Problem: When jlinking, I noticed the resulting directories have distinctly different sizes.

This is a footprint bug, as one of jlinking's primary advantages is disk size, and this negates part of that advantage.

Details: Dinakar discovered that the main difference (on Linux at least) is in the properties and dat files.

Example:

jlink --add-modules java.base --output C:\output_folder

Hotspot 11 result on Windows: 38.0mb
OpenJ9 11 result on Windows: 50.6mb

Hotspot 11 result on Linux: 46.8mb
OpenJ9 11 result on Linux: 52.6mb

[pshipton]

We could consider adding jlink options to further strip the OpenJ9 artifacts, like java.properties files.

[Ealrann]

Hello,
I'm switching to OpenJ9 too, and encounter the same issue. Not a big deal, but well, if something can be done about that, I'll take it!

If you need, here two build of my app, one with hotspot, and one with openJ9:
https://www.dropbox.com/s/rg3unwvj228j8aj/VSand_v1.2.0_windows.zip?dl=0

[DanHeidinga]

@Ealrann Can you share your jlink commandline as well?

[Ealrann]

@Ealrann Can you share your jlink commandline as well?

In fact, I use a gradle plugin to do it: badass-jlink-plugin
Here the options I use:

options = ['--strip-debug', '--compress', '2', '--no-header-files', '--no-man-pages']

And then, I simply use gradle:

git clone --recursive [email protected]:Ealrann/VSand.git
cd VSand
gradle build -xTest
gradle jlink

The build is in this location: org.sheepy.vsand/build/image/

[ashu-mehra]

@ShishirH can you please take a look at this.

[ShishirH]

Yeah, the difference is the presence of the properties files and dat files.
While the dat files can be safely removed, we do need the properties files for any messages being printed, otherwise we only get the tags. The properties files themselves are around 150KB, but due to there being so many files for different languages and regions, they end up being around 4.2MB in total.

One way of solving that in my opinion, would be to just generate the properties files based on the locale, and not everything else. That could be the default behaviour, with another option added to jlink if you want the dat files and the rest of the properties files.

Any thoughts @DanHeidinga @ashu-mehra ?

[DanHeidinga]

@ShishirH which dat files are you referring to? I see the following ones in a local build:

../build/macosx-x86_64-normal-server-release/images/jdk//lib/compressedrefs/j9ddr.dat
../build/macosx-x86_64-normal-server-release/images/jdk//lib/security/public_suffix_list.dat
../build/macosx-x86_64-normal-server-release/images/jdk//lib/tzdb.dat
../build/macosx-x86_64-normal-server-release/images/jdk//lib/J9TraceFormat.dat
../build/macosx-x86_64-normal-server-release/images/jdk//lib/OMRTraceFormat.dat

I'm iffy on removing the trace / ddr related files as it will be difficult to debug issues with jlinked builds.

Deleting the non-english properties files in the default build seems reasonable, especially with an option to add them back.

[Ealrann]

In my case, on Windows x64, I also have few more .dll with openj9 :

    24  KB | bin/jncrypto.dll
    12  KB | bin/jsig.dll
    3.2 MB | bin/libcrypto-1_1-x64.dll
    12  KB | bin/server/jsig.dll

[DanHeidinga]

@keithc-ca Were you working on something to limit the number of jsig.dll's we shipped?

The jncrypto and libcrypto are related to the openssl-optimized JCE provider openj9 uses.

[keithc-ca]

I see three copies of jsig.dll in a build from adopt; even if we were to remove two of them, it won't help much because they're small (only 12kB each).

[ShishirH]

@adamfarley Is the difference in the sizes in Windows too because of the properties and dat files?

[DanHeidinga]

We can look at shrinking the ddr.dat file but either

• using gzip (or similar) to compress it and then teaching the tools that read it to handle compressed blobs, or
• changing the format to be more similar to the dwarf format, and having each byte have 7 bits of data and 1 bit to indicate if there is more data.

This would only address ~670k though.

[adamfarley]

@ShishirH - Half of it, yes. The difference on windows is just over 10mb, and the properties and dat files on J9 add up to 5.9mb, while on hs they add up to 339kb.

Another 3.196mb can be found in a file called libcrypto-1_1-x64.dll, which doesn't seem to exist in the hs jlinked jre (as Ealrann noticed). More space is taken up by j9 files in /bin/compressedrefs.

Also, the windows size in the latest 11 release is up to 54mb, while hs remains steady at 39mb and change.

[keithc-ca]

I spent some time today looking at this with the latest release builds available from https://adoptopenjdk.net/.

$ ./hotspot/bin/java --version
openjdk 11.0.3 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.3+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.3+7, mixed mode)

$ ./openj9/bin/java --version
openjdk 11.0.3 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.3+7)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.14.0, JRE 11 Linux amd64-64-Bit Compressed References 20190417_202 (JIT enabled, AOT enabled)
OpenJ9   - bad1d4d06
OMR      - 4a4278e6
JCL      - 5cc996a803 based on jdk-11.0.3+7)

Using jlink options: --verbose --add-modules java.base --strip-debug --compress 2 --no-header-files --no-man-pages

$ du -ks hotspot openj9
32916   hotspot
38492   openj9

The difference is 5576kB which is less than the combined size (5788kB) of java*properties and *.dat files.

$ du -ck openj9/lib/java*properties
164 openj9/lib/java_ca.properties
200 openj9/lib/java_cs.properties
164 openj9/lib/java_de.properties
164 openj9/lib/java_es.properties
172 openj9/lib/java_fr.properties
208 openj9/lib/java_hu.properties
152 openj9/lib/java_it.properties
320 openj9/lib/java_ja.properties
256 openj9/lib/java_ko.properties
192 openj9/lib/java_pl.properties
124 openj9/lib/java.properties
160 openj9/lib/java_pt_BR.properties
532 openj9/lib/java_ru.properties
180 openj9/lib/java_sk.properties
152 openj9/lib/java_sl.properties
200 openj9/lib/java_tr.properties
196 openj9/lib/java_zh_CN.properties
196 openj9/lib/java_zh.properties
192 openj9/lib/java_zh_TW.properties
3924    total

$ du -ck openj9/lib/*.dat openj9/lib/compressedrefs/j9ddr.dat 
968 openj9/lib/J9TraceFormat.dat
100 openj9/lib/OMRTraceFormat.dat
104 openj9/lib/tzdb.dat
692 openj9/lib/compressedrefs/j9ddr.dat
1864    total

Adding a jlink option to remove unneeded java*.properties files seems a reasonable idea.

[ShishirH]

@keithc-ca Should all the properties files be generated by default with an option to generate only the EN ones or should only the EN properties file be generated by default, with an option to generate all the other ones too?

[keithc-ca]

I'm not familiar with adding a plugin to jlink (I think that's how we would approach this), but I think the option should take a value (or list of values) that identify the subset of java.properties files that should be included in the output. Then the user can choose the languages they want supported.

[pshipton]

jlink already has a plugin include-locales. One option is to only include en by default, and "teach" the include-locales plugin to understand the java.properties files and only include other languages when requested.

[pshipton]

Similarly there is an existing strip-debug plugin which could be extended to remove the TraceFormat.dat files. We should consider just moving these files to a different module than java.base, like openj9.traceformat.

I don't think that j9ddr.dat should be removed by strip-debug as removing this file impacts the ability to diagnose OpenJ9 crashes. I suggest a different plugin be created to remove this file, with a warning that removing it voids creation of usable system dumps, using jdmpview, Eclipse MAT, etc, and getting any related OpenJ9 support.

[keithc-ca]

An attempt at using --include-locales en offers a hint:

Error: jdk.localedata module was not specified with --add-modules option

Perhaps openj9 should package those property files in that module rather than java.base.

[pshipton]

Since we have a tentative plan to move some things around, I've added this to the next milestone. Which isn't a commitment to deliver the changes in this milestone.

[keithc-ca]

*TraceFormat.dat should be associated with the module openj9.traceformat: those files are only used by that module and modules that require it.

[pshipton]

*TraceFormat.dat should be associated with the module openj9.traceformat: those files are only used by that module and modules that require it.

Not quite true (i.e. -Xtrace:iprint), but agree we should move them (https://github.com/eclipse/openj9/issues/4488#issuecomment-495753537).

[keithc-ca]

It's true that removing *TraceFormat.dat means that -Xtrace:print stops working, but you can still use -Xtrace:output={file} which can read by an image that includes openj9.traceformat.

[pshipton]

I was able to remove all the java_<language>.properties files by using the jlink command --exclude-files=**java_*.properties. This doesn't affect the default English properties file java.properties.

[DanHeidinga]

https://github.com/ibmruntimes/openj9-openjdk-jdk11/pull/179

What does this do to something like healthcenter? Is it still able to process the tracepoints required for HC to operate? @keithc-ca

[pshipton]

To get healthcenter to work from a jlinked image, I had to include the following modules:

java.logging
java.management
java.management.rmi
java.naming
java.rmi
java.security.sasl
jdk.naming.rmi

Also, changed the com.ibm.java.diagnostics.healthcenter.agent.transport in the lib/healthcenter.properties file to jrmp from iiop.

I'm not sure what I'm looking for, but everything I tried seemed to be working, with the TraceFormat.dat files included.

[pshipton]

Tried with the TraceFormat.dat files removed, and it still seems to work.

[DanHeidinga]

I think the only thing left here is to document how to remove the properties files: https://github.com/eclipse/openj9/issues/4488#issuecomment-496279476

@SueChaplain Where would be the best place to put this kind of doc?

[pshipton]

Create a doc issue for https://github.com/eclipse/openj9/issues/4488#issuecomment-503257764.

Using the latest xlinux nighty jdk11 build, I get the following sizes using du -h

50M - this is with the TraceFormat.dat files removed, they are about 1M
49M - after excluding the java_*.properties files, although the sizes add up to 3.7M

Breakdown of the biggest elements:
48M lib
23M lib/compressedrefs
25M lib/modules

Not sure there is more we can easily do, but I'll keep this open.

[SueChaplain]

Although this is showing as closed in the 0.16 milestone, the docs were updated for 0.15.0. Issue still applied to that release, nothing more to do here for docs.

Edit (Peter Shipton): this issue is not closed in the 0.16 milestone, it is currently in the 0.16 milestone in order to do more investigation, which may result in further actions.