Logstash: [discuss] hybrid Ruby/Java plugin model and jar dependencies

[colinsurprenant]

A few plugins have been migrated to a hybrid Ruby+Java model where some classes are written in Java, some jars dependencies are required and gradle is used for the build. For example the snmp input, the dlq input and the date filter.

This hybrid model is not really documented at this point and practically speaking this is the model that will likely last for some time if someone wants to write Java code for their plugin. In that respect we should probably document it (and should be a separate issue).

One practical problem we face today with the hybrid model, as seen in the logstash-output-google_cloud_storage issue is the potential conflict with jar dependencies between a plugin and core.

I believe the easy answer is that all jars that are already loaded by core (we should make that list, but for example jackson) should be declared as compileOnly in the plugin gradle dependencies.

The harder part is dealing with compatibility across core versions: for example latest core depends on guava, but not prior releases. How can a plugin that also depends on guava deal with this situation where some versions of core might load it and other might not? Any ideas on how we can solve that?

[jakelandis]

the potential conflict with jar dependencies between a plugin and core

As of 6.2 plugins and core shouldn't compete with each for the classpath. With the introduction of the Java entry point core will use Java's primordial classloader for all it's declared dependencies and and plugins will use JRuby's class loader. Plugin's will still compete with each other, but not with core due to different class loaders.

For competing dependencies between plugins, we would need to implement a class loader per plugin and integrate it with JRuby. I looked into this a while a back, but didn't progress in hopes the pure Java plugins will address this concern.

I love guava, but have avoided it's use in many prior projects due to this exact issue. I don't have a good solution (short of custom class loaders) for when plugins compete with each other for dependencies.

EDIT: some additional class loader information via the investigation here: https://github.com/logstash-plugins/logstash-input-jdbc/issues/263

[danhermann]

This hybrid model is not really documented at this point and practically speaking this is the model that will likely last for some time if someone wants to write Java code for their plugin.

It would certainly be beneficial to document this model since it's being used in a number of places, but the Java API is targeted for 6.4 and https://github.com/elastic/logstash/pull/9342 will be merged soon, so hopefully there will soon be a pure Java option for those who wish to write plugins in Java. The question of jar dependencies among Java plugins remains, but @jakelandis outlined above one way in which we could address that.

[jakelandis]

@danhermann - will the Java API implementation targeted for 6.4 use separate class loaders for each plugin ?

[danhermann]

@jakelandis, the work for separate classloaders is described in https://github.com/elastic/logstash/issues/9521 which follows #9342. We'd like to get it into 6.4, too, but that's not as certain as the core Java API work.

[colinsurprenant]

So if I understand correctly starting at 6.2 with the separate classloaders between core and JRuby, a plugin such as the DLQ input which lists compileOnly dependencies such as com.fasterxml.jackson.core:jackson-core and log4j-core will not work because these jars included in core will not be visible in the plugin per the separate classloaders?

[jakelandis]

... will not work because these jars included in core will not be visible in the plugin per the separate classloaders?

No, the JRuby classloader is a child of the primordial class loader and implements a child first policy. (_as observed, I have not looked at the exact implementation_). Meaning that classes will come from the JRuby classloader first and if that fails it will search the primordial loader.

There are some exceptions (JNI, class loader path expectations, more ...) that may prevent falling back to the primordial to work...but for most cases if it is shipped in core, then plugins don't need to also ship it.

[jakelandis]

I also just realized I am using "primordial class loader" incorrectly .. What i really mean is the System/Application class loader , not primordial. There is a technical distinction ... but the general idea is the same (the one used by the core vs. JRuby)

[colinsurprenant]

@jakelandis Right, that makes sense. Thanks for clarifying. So that leaves the inter-plugins dependencies issues which will be solved with the separate class loaders which will land at some point.

In the mean time I guess a good recommendation would be to list all the dependencies included in all supported core versions, like jackson, so that these should be declared as compileOnly to avoid potential inter-plugins conflicts on these at least.

Also, I believe that the plugins who load jars through the jar-dependencies mechanism (require_jar(...)) should be correctly handled in the sense that jar-dependencies will warn on conflict and not load a conflicting jar.