Openj9: Elasticsearch "failed to load SSL configuration"

[mstoodle]

After resolving #7764 (via PR #9725), running elasticsearch encounters a new problem:

ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.http.ssl]]; nested: ElasticsearchException[failed to initialize a TrustManagerFactory]; nested: IllegalStateException[failed to find a X509ExtendedTrustManager];
Likely root cause: java.lang.IllegalStateException: failed to find a X509ExtendedTrustManager

Steps to reproduce (basically follow steps 1&2 from https://www.elastic.co/downloads/elasticsearch-no-jdk)

1. Download appropriate elasticsearch platform binary from "Downloads" section here: https://www.elastic.co/downloads/elasticsearch-no-jdk
2. Install/build OpenJDK with OpenJ9 (I used a recent JDK14)
3. Run bin/elasticsearch

[mstoodle]

Possibly influenced (but I don't think so) by an earlier error (warning?):
JVMJ9VM085W Malformed option: '-Xlog:gc*'

[mstoodle]

The fundamental problem is thrown from line 302 here:
https://github.com/elastic/elasticsearch/blob/7.7/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ssl/CertParsingUtils.java#L293-L302

[mstoodle]

When Hotspot runs, there is a trust manager found sun.security.ssl.X509TrustManagerImpl. When OpenJ9 runs, there is only a null trust manager. Requested algorithm is PKIX which is, I believe, the default.

[JasonFengJ9]

It seems work with my Mac OS X 10.15.3 JDK14 built with https://github.com/eclipse/openj9/pull/9725. The package elasticsearch-7.7.0-no-jdk-darwin-x86_64.tar.gz is downloaded from https://www.elastic.co/downloads/elasticsearch-no-jdk.

curl -X GET http://localhost:9200/ returns following:

{
  "name" : "jmp",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "YPmsX08QTBWMLQS0wPB6Hw",
  "version" : {
    "number" : "7.7.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
    "build_date" : "2020-05-12T02:01:37.602180Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I noticed the console output has [2020-05-28T18:05:15,297][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [jmp] Active license is now [BASIC]; Security is disabled (there was no additional setting), maybe that's the reason why it didn't throw ElasticsearchSecurityException.

[mstoodle]

Thanks for trying it @JasonFengJ9 . Given all the weird issues I was having yesterday (even hitting a bug in gmake 4.3), I'll try again fresh to see if maybe I was just unlucky yesterday (fingers crossed?).

[mstoodle]

Sadly, I am still seeing the exception. I do not see anything like the message "Security is disabled" that you see. I'll contact you directly to compare some notes from the output and will report findings back here.

[mstoodle]

Jason ended up sending me the JDK he built and, after fighting through the macOS security exceptions, I verified that it works on my system too. Much thanks for your help @JasonFengJ !

Examining his jdk and my jdk led to the source of the problem: I had been specifying JAVA_HOME based on build/macosx-x86_64-server-release/jdk and NOT the proper build/macosx-x86_64-server-release/images/jdk (note the key presence of images).

So, this issue can be explained by PEBKAC (ego suitably reined in).

On the plus side, that means elasticsearch is now able to start with OpenJ9.

[JasonFengJ9]

@mstoodle it is nice to hear that elasticsearch is now able to start with OpenJ9 out of the box after https://github.com/eclipse/openj9/pull/9567 & https://github.com/eclipse/openj9/pull/9725.