Openj9: AdoptOpenJDK11 with EclipseJ9 3x slower than HotSpot on AES & RSA encoding & decoding

[avermeer]

Hello,

I have identified that EclipseJ9 is x3 shower than HotSpot (using AdoptOpenJDK11 on Linux) for AES and RSA encoding and decoding.

Here's a sample code to reproduce the issue with 200000 iterations doing the following sequence:

1. AES 128 decoding
2. RSA 512 decoding
3. RSA 512 encoding
4. AES 128 encoding

Results on a physical machine Linux CentOS 7.4.1708 with 8 cores - Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz (fam: 06, model: 55, stepping: 04), 32GB RAM :

HOTSPOT
$ time ./java -showversion /u/users/ple/TestCrypto.java
openjdk version "11" 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)
server: numberOfRequests 200000 average duration 87 microseconds

real 0m18.48s
user 0m20.99s
sys 0m0.44s

OPENJ9
$ time ./java -showversion /u/users/ple/TestCrypto.java
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.1+13)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.11.0, JRE 11 Linux amd64-64-Bit Compressed References 20181020_70 (JIT enabled, AOT enabled)
OpenJ9 - 090ff9dc
OMR - ea548a66
JCL - f62696f378 based on jdk-11.0.1+13)
server: numberOfRequests 200000 average duration 256 microseconds

real 0m52.43s
user 0m59.70s
sys 0m1.29s

See attached file for the code code of the sample.

TestCrypto.zip

Not sure, but got the feeling that AdoptOpenJDK11 with EclipseJ9 (latest version downloaded from AdoptOpenJDK site) isn't using opensll to accelerate encryption.

Any idea if that could be fixed in an update of ElipseJ9, at least in the GitHut some I could rebuild AdoptOpenJDK11 with this fix to check if we have other performance loss with J9 vs HotSpot with our app?

Kind regards,
Alexandre

[pshipton]

@ashbm5

[pshipton]

@vijaysun-omr

[pshipton]

@avermeer you are correct, jdk11 is not using openssl acceleration yet, only jdk8. jdk11 support for openssl is expected in a future version https://github.com/ibmruntimes/openj9-openjdk-jdk11/issues/31

[DanHeidinga]

@pshipton is ibmruntimes/openj9-openjdk-jdk11#31 expected to land in the 0.12.0 release? If that's the case, we should open a tracking issue here and assign to the 0.12.0 milestone

[pshipton]

is ibmruntimes/openj9-openjdk-jdk11#31 expected to land in the 0.12.0 release?

@ashbm5 @enasser ?

[avermeer]

Hello,

First of all, thank you very much to all of you who answered on this issue.
However, I must confess that it isn't clear whether or not there is any hope to see openssl-based acceleration for AdoptOpenJDK11 version based on EclipseJ9.
What's confusing is that in the following "what's new?" page of OpenJ9 0.11.0, the openssl-based acceleration of crytographics operations was announced: https://www.eclipse.org/openj9/oj9_whatsnew.html
Also I read in current answers provided to this incident that some fixes could be made in 0.12.0 release.
But isn't 0.12.0 targeted for JDK12 and not for JDK11?

Kind regards,
Alexandre

[vijaysun-omr]

Hi Alexandre,

v0.11.0 is a release of OpenJ9 that is used in both JDK11 and JDK8.
The "what's new" page does mention the fact that OpenSSL support was only available on JDK8 but maybe you were considering that page to be about what is in JDK11 (perhaps thinking that v0.11.0 is somehow associated only with JDK11 ?).
I think @enasser will likely be the one to confirm if the JDK11 support will land in v0.12.0 of OpenJ9 but I think it is possible that it will.
If v0.12.0 of OpenJ9 releases before JDK12 comes out then I expect it will be used in both JDK11 and JDK8.

My apologies if I misunderstood your last comment in some way.

[pshipton]

If v0.12.0 of OpenJ9 releases before JDK12 comes out then I expect it will be used in both JDK11 and JDK8.

Correct, 0.12.0 [1] is scheduled for January and will support jdk8 and jdk11.
0.13.0 is scheduled for March to introduce Java 12 support.
List of scheduled releases https://www.eclipse.org/openj9/docs/openj9_support/

[1] https://projects.eclipse.org/projects/technology.openj9/releases/0.12.0/plan

[avermeer]

Hello,

I'd love to see an AdoptOpenJDK11 release based on 0.12.0 that fixes this OpenSSL acceleration issue, but the plan https://www.eclipse.org/openj9/docs/openj9_support/ explicitly mentions that 0.12.0 is not targeted for JDK11(LTS).
Is https://www.eclipse.org/openj9/docs/openj9_support/ table outdated, or is OpenSSL acceleration "dead" for AdoptOpenJDK11 ?

Kind regards,
Alexandre

[DanHeidinga]

That page is incorrect. I think I messed the table up when updating it for our 6 release a year policy. The project intends to ship quarterly releases that line up with the OpenJDK updates and to have two additional releases (March & Sept) that target only the new JDK levels.

I'll get that page updated to indicate that 0.12.0 will include updates for JDK8 & JDK11

[DanHeidinga]

Pull request opened to update the table - https://github.com/eclipse/openj9-docs/pull/139

[avermeer]

Hello,

Is there any hope that this performance issue on Java crypto APIs will be fixed in OpenJ9 upcoming 0.12.0 release? it's quite an hard blocker for our adoption of J9 vs HotSpot...

Kind regards,
Alexandre

[fjeremic]

Is there any hope that this performance issue on Java crypto APIs will be fixed in OpenJ9 upcoming 0.12.0 release? it's quite an hard blocker for our adoption of J9 vs HotSpot...

The fix has landed in ibmruntimes/openj9-openjdk-jdk11#73. I think it should be available now. @pshipton can you confirm?

[enasser]

Yes @fjeremic Openssl support feature is merged. There were few build configuration issues while merging which is mostly fixed and the builds should be available soon. @pshipton please confirm.

[pshipton]

@avermeer According to @sxa555 in Slack https://openj9.slack.com/messages/CBCEMDZE2, it seems the latest set of jdk11 nightly builds at AdoptOpenJDK.net should have openssl enabled and resolve this issue.

[pshipton]

Sorry, the dates of the AdoptOpenJDK builds are misleading. The latest build shows Dec 13 but is a build from the 10th according to the dates of the files. Please check the next nightly builds.

[pshipton]

The builds for Dec 14 have openssl enabled, except on macOS.

[avermeer]

Hello,

I have bad news : we ran tests with the latest AdoptOpenJDK11 with OpenJ9 builds, and it shows this performance issue has not been fixed.
Let me give details below, please let me know if I was mistaken somewhere.

These tests are based on the same TestCrypto.java sample code which I shared while opening this ticket.

As a reminder, with October AdoptOpenJD11 with HotSpot we had:

Hotspot :

time ./java -showversion /u/users/ple/TestCrypto.java
openjdk version "11" 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)
server: numberOfRequests 200000 average duration 87 microseconds

real    0m20.01s
user    0m21.23s
sys     0m0.52s

With AdoptOpenJDK11 with OpenJ9 october release we had:

time ./java -showversion /u/users/ple/TestCrypto.java
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.1+13)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.11.0, JRE 11 Linux amd64-64-Bit Compressed References 20181020_70 (JIT enabled, AOT enabled)
OpenJ9   - 090ff9dc
OMR      - ea548a66
JCL      - f62696f378 based on jdk-11.0.1+13)
server: numberOfRequests 200000 average duration 260 microseconds

real    0m53.46s
user    0m59.91s
sys     0m1.42s

With latest AdoptOpenJDK11 with OpenJ9 nightly build, we have still have disapointing results versus HotSpot, whether or not native crypto is activated:

time ./java -showversion -Djdk.nativeCrypto=true /u/users/ple/TestCrypto.java
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.1+13-201812161931)
Eclipse OpenJ9 VM AdoptOpenJDK (build master-f9ab45bfe, JRE 11 Linux amd64-64-Bit Compressed References 20181216_61 (JIT enabled, AOT enabled)
OpenJ9   - f9ab45bfe
OMR      - 0d514697
JCL      - f6d23ec079 based on jdk-11.0.1+13)
server: numberOfRequests 200000 average duration 256 microseconds

real    0m52.73s
user    0m58.47s
sys     0m1.38s

time ./java -showversion -Djdk.nativeCrypto=false /u/users/ple/TestCrypto.java
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.1+13-201812161931)
Eclipse OpenJ9 VM AdoptOpenJDK (build master-f9ab45bfe, JRE 11 Linux amd64-64-Bit Compressed References 20181216_61 (JIT enabled, AOT enabled)
OpenJ9   - f9ab45bfe
OMR      - 0d514697
JCL      - f6d23ec079 based on jdk-11.0.1+13)
server: numberOfRequests 200000 average duration 263 microseconds

real    0m53.95s
user    1m0.90s
sys     0m1.55s

I wish you could tell me I'm wrong somewhere with these tests, but so far I still get HotSpot ~3x faster than OpenJ9.

Did I miss something?

Kind regards,
Alexandre

[pshipton]

@enasser can you please take a look

[pshipton]

I tried running TestCrypto with -Djdk.nativeCryptoTrace=true and got the following.

Warning: Native crypto library load failed. Using Java crypto implementation
Warning: Native crypto library load failed. Using Java crypto implementation

There doesn't seem to be any debug option to tell us why it failed, but I tried loading the crypto library and got Exception in thread "main" java.lang.UnsatisfiedLinkError: jncrypto (libssl.so.1.1: cannot open shared object file: No such file or directory)
Indeed libssl.so.1.1 is not included in the build.

ldd ./jdk-11.0.1+13/lib/libjncrypto.so
...
libssl.so.1.1 => not found
...

Checking the jdk8 libjncrypto.so, there is no reference to libssl.so.1.1. Seems something is wrong with the jdk11 jncrypto library build.

[enasser]

Thank you @pshipton . As you observed, we are still building dependecy with libssl but not bundling the libssl library.. I will create a PR to remove the dependency on libssl.

[pshipton]

The fix https://github.com/ibmruntimes/openj9-openjdk-jdk11/pull/83 is merged. Will check the next Adopt build to confirm.

[pshipton]

Built on xlinux myself to check. libssl.so is not longer a prereq of libjncrypto.so
Running TestCrypto with -Djdk.nativeCryptoTrace=true results in the following, now correct, behavior.

MessageDigest load - using Native crypto library.
CipherCore Load - using native crypto library.

Not sure time to run the test is improved. I got about the same poor results with and without -Djdk.nativeCrypto=false, but I'm using a cloud machine so results may vary.

[ashbm5]

The reason for the slowdown when compared to hotspot is that the OpenSSL functionality that was added to OpenJ9 doesn't support RSA yet. As announced, the following algorithms use OpenSSL: AES/GCM, AES/CBC and SHA* hashing routines.

You do notice a very small perf. gain when enabling OpenSSL support as compared to disabled OpenSSL due to the perf. increase of the AES/CBC algorithm in your benchmark. However, RSA/ECB is taking the vast majority of time in your benchmark and since it is not using OpenSSL it is significantly slower.

When benchmarking only RSA/ECB - I'm seeing 2-3x slower on OpenJ9, corresponding to your observations.

Given this, Having RSA also leverage OpenSSL looks like an important work item to work on next.

[enasser]

As @ashbm5 explained above the current openssl support did not help to improve the performance of the scenario in test case TestCrypto. Both Openj9 JDK 8 and JDK 11 giving similar results with OpenSSL support while testing with TestCrypto. Need to include the corresponding algorithm with OpenSSL support in order to fix this performance gap.

[avermeer]

Hello @enasser ,

Thank you (and the other involved developers) very much for your tenacious attempts to get this issue fixed.
And please apologies for this trivial question: would it be affordable to get this performance issue resolved in time for 0.12.0 release?

Kind regards,
Alexandre

[ashbm5]

@avermeer - we will try our best but no promises.

[ashbm5]

@avermeer is it essential for this RSA support to be rolled into 0.12 release or having the feature appear in one of the nightly builds a bit after 0.12 be satisfactory? is this for production use or prototyping/experimentation?

[pshipton]

It seems this is not going to be resolved in the 0.12 release, moving to the next one.

[avermeer]

Oh this is very sad that it's not going to be resolved in 0.12.0 release.
When would it resolved at best?

[ashbm5]

Probably in the next few weeks it will be in the nightly build. I have confirmed that accelerating RSA w/ OpenSSL resolves the performance issue mentioned above using the test code posted.

[avermeer]

@ashbm5, that would be outstanding, thanks!

[pshipton]

The 0.12 release is scheduled to GA on Jan 23, we should try to squeeze it in if possible.

[pshipton]

Set this back to the 0.12 milestone. Alon thinks jdk8 can have a PR opened on Monday, and jdk11 will follow, TBD how long it will take to port.

[vij-singh]

@ashbm5 Alon are there any updates?

[ashbm5]

@vij-singh we finished the RSA code for both jdk8 and jdk11. Unfortunately, there is another incoming PR https://github.com/ibmruntimes/openj9-openjdk-jdk8/pull/227 that we didn't account for - this is to support dynamic library loading for openssl. So we're working on implementing the dynamic library loading.

Also, with doing dynamic library loading - we need to support openssl 1.0.2, which is not done yet. Peter also pointed out some other changes that need to be made to the code (see PR thread)

[avermeer]

Hello @ashbm5 ,

You mentioned having finished the RSA code for jdk8 & jdk11.

Does it means that current issue is fixed, regardless of the dynamic library loading for openssl other topic which you mentioned above?

I mean, if I take the most recent AdoptOpenJDK11 with OpenJ9 build from https://adoptopenjdk.net/nightly.html?variant=openjdk11&jvmVariant=openj9, will this include the this of current performance issue?

Kind regards,
Alexandre

[pshipton]

No, the code is not yet delivered.

[Siarhei-Yarkavy]

I think it makes sense to wait with the release but deliver those important fixes.

[avermeer]

Hello @Siarhei-Yarkavy,

If you mean delay the 0.12.0 release until this issue is fixed, then I agree.
As soon as there is something which I can test on nightly builds, I will happily test & provide feedbacks.

Kind regards,
Alexandre

[SueChaplain]

I've added the doc:externals label as the addition of RSA support needs to be included in the user documentation. @vij-singh - please have someone open an issue at openj9-docs with the content we need: https://github.com/eclipse/openj9-docs/issues/new?template=new-documentation-change.md

I'll need to know whether it is enabled by default as per the other algorithms, and whether there will be a new -D system property as per the other algorithms.

[pshipton]

Thanks @SueChaplain. I've created https://github.com/eclipse/openj9-docs/issues/201

[pshipton]

Fixed by https://github.com/ibmruntimes/openj9-openjdk-jdk11/pull/100 and https://github.com/ibmruntimes/openj9-openjdk-jdk11/pull/103.

I won't close this until the jdk8 changes are delivered as well.

[pshipton]

For jdk8, fixed by ibmruntimes/openj9-openjdk-jdk8#250 and #251

[pshipton]

The RSA acceleration work is completed.

[avermeer]

Hello,

We have verified on our side, and the performance issue related to this ticket is indeed fixed, this is outstanding => thank you very much for having kept this for the 11.0.2 OpenJ9 release, just in time !!

Kind regards,
Alexandre

[DanHeidinga]

@avermeer I'm always curious to hear how OpenJ9 is being used. If you can share any details on how you're using OpenJ9, I'd love to hear about it! Feel free to respond here or reach out to me privately by email (in github profile), twitter (@DanHeidinga) etc.

Thanks!