Elasticsearch: Notarize macOS artifacts for macOS Catalina readiness

[jasontedor]

macOS Catalina is the next major release of macOS, expected in September/October 2019. Catalina will require applications distributed outside of the Mac App Store to be signed and notarized. For Elasticsearch specifically, this requires that:

• the bundled JDK be signed/notarized
• we handle that JNA extracts a native library at runtime
• the ML binaries be signed/notarized

The bundled JDKs produced by Oracle are signed but not notarized, we are exploring our options here including using a bundled JDK from a different source. JNA is not signed and notarized, our best option here is probably that the bundled JDK have the unsigned library entitlement. For the ML artifacts, we are currently working on integrating signing and notarization into our release pipeline.

[elasticmachine]

Pinging @elastic/es-core-infra

[oeed]

Glad this is being worked on! Is there a rough ETA?

[jasontedor]

@oeed As a matter of policy, we don't give ETAs, even approximate, sorry.

I will say this though: we are aware that macOS Catalina was released earlier in the week, a lot of us are excited to try it out too and we want to meet our users there, so we understand the importance of wrapping up our work on this.

[oeed]

Quite understandable! Was rather silly of me to jump the gun and update.

[JRBeckstrom]

Any news on this? Wasn't aware that this was an issue for Elasticsearch and has upgraded to Catalina...

[jasontedor]

We are blocked by the fact that there is not an upstream distribution of the JDK that is correctly signed and notarized. We are working through these issues via other channels.

[eswenson1]

+1 While it seems completely reprehensible that a critical component like elasticsearch wouldn’t run on an OS that is been out a while now (readiness should have been achieved before release), a workaround is to run eleasticsearch from a Docker container. That is now what I’m forced to do since I develop on macOS and need elasticsearch to run locally.

[droberts195]

elasticsearch-6.8.5.tar.gz _is_ now properly notarized for macOS. (6.8.5 is the only such 6.x release at the moment.)

7.x is proving harder, as we redistribute a JDK and that needs notarizing as well as the machine learning native programs. In elasticsearch-7.4.2-darwin-x86_64.tar.gz the machine learning native programs are notarized, so if you use your own JDK with 7.4.2 on Catalina then at least there won't be further problems after Elasticsearch starts up.

We'll update this issue when there's a release where the JDK we bundle in version 7 is notarized too.

[eswenson1]

Thanks. This is perfect for us, as we're currently stuck on 6.x.

[droberts195]

Elasticsearch 7.5.0 is now GA and elasticsearch-7.5.0-darwin-x86_64.tar.gz is fully notarized, so will run without Gatekeeper warnings on macOS Catalina.

Therefore, it is now possible to download both 6.x and 7.x Elasticsearch releases that are notarized for macOS Catalina: the .tar.gz distributions of 6.8.5 and 7.5.0.

[hammerhg]

I am still having issues on MacOS Catalina with Elasticsearch 7.6.1 elasticsearch-7.6.1-darwin-x86_64.tar.gz

When running for the first time I still see an error message
“controller” can’t be opened because Apple cannot check it for malicious software.
and have to press Open Anyway in Settings -> Security and Privacy -> General tab during first run.

This issues is still valid for ML binaries

[oeed]

@hammerhg can't comment on that issue specifically, but I ended up instead hosting ElasticSearch in a Docker container which doesn't have the same issues.

[Conky5]

Hi @hammerhg and @oeed, Apple's stricter notarization requirements caused issues with the bundled jdk, instructions on working around this are in the Elasticsearch documentation https://www.elastic.co/guide/en/elasticsearch/reference/7.6/targz.html#targz-running in the macOS Gatekeeper warnings section. Alternatively you can download the artifacts outside of a browser, for example

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.1-darwin-x86_64.tar.gz

And avoid the quarantine extended attributes that trigger this behavior.

AdoptOpenJDK 14 has successfully implemented hardened runtime support https://github.com/AdoptOpenJDK/openjdk-build/issues/1130#issuecomment-600762039 and will be bundled with upcoming versions of Elasticsearch to solve this issue without any workarounds.

[botasky11]

opps I have the same problem

[MurshidMac]

Download the package with Curl
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch.7.x.x.tar.gz
& extract withtar -xvf elasticsearch-7.x.x.tar.gz. This would fix the error