Logstash: [META] Elasticsearch now requires Java 9 for compilation

[jakelandis]

As of https://github.com/elastic/elasticsearch/pull/28071 , Elasticsearch requires Java9 to compile Elasticsearch.

This impacts some of Logstash's test which pulls from Elasticsearch source and builds it for the tests. Since Logstash is not quite ready for Java9 all of our builds have expectations on Java8, and any tests that build Elasticsearch from source are currently broken.

This includes X-Pack and the Elasticsearch output plugin , maybe more...

We have a couple options for how to fix our builds

• keep building from source, just ensure Elasticsearch is built with Java 9. [1]
• switch out builds to use the nightly snapshots instead of building from source.

Thoughts ?

[1] It seems that there is some support for this approach that could be leveraged: https://github.com/elastic/elasticsearch/pull/28051

[original-brownbear]

I vote building against snapshots for now as suggested by @jsvd :)

[jsvd]

Here's my perspective on using snapshots:

Pros:

• Building elasticsearch/x-pack is no longer a concern
• CI runs will be much faster and require less dependencies
• 24h cycles of artifact stability means we won't see failures in the middle of the day after some merge on master

Cons:

• If an artifact is broken it will cause errors in builds for the remainder of the 24h cycle

[jakelandis]

I agree that using the pre-built snapshots is the best way forward. One issue however is that the download URL contains the build id, which is effectively a small random string, which causes issues programmatically discovering the latest SNAPSHOT for a given version.

Fortunately, our (Elastic) infra team has been working on a manifest service to be able to alias the latest build to discover the download URL. The service is in early days (private alpha would be the best categorization). I plan to adopt that service to implement snapshot downloads (instead of builds) for the Elasticsearch X-pack test dependency.

If that works well (which is so far so good), we should adopt it across the board for any snapshot Elasticsearch builds.

[mustafaakin]

Now it requires Java 10 😢