Logstash: Published logstash-core not working for 5.1.x

[suyograo]

I just published a snapshot version of logstash-core and their friends: logstash-core-event-java, logstash-core-queue-jruby, logstash-core-plugin-api. There are 2 issues:

1. jars in lib are not included by the gemspec. This is fixed in https://github.com/elastic/logstash/pull/6373
2. Even with the fix from #6373, the gem is borked when it is loaded. See https://travis-ci.org/elastic/logstash/jobs/182136053#L937. Looks like the log4j* jars are not getting loaded.

[untergeek]

I had the log4j classpath error, too. What was weird was that sometimes I could click "Restart Build" and it would work.

[ph]

Taking a look

[ph]

The generated specification looks OK see https://gist.github.com/ph/aa0930b3dd7581c4748371ef131c41d1

[ph]

At install time we can see that jar dependencies picks up things and install them.

Installing logstash-core-event-java 5.0.2
  jar dependencies for logstash-core-event-java-5.0.2-java.gemspec . . .
      com.fasterxml.jackson.core:jackson-core:2.7.3:compile
      com.fasterxml.jackson.core:jackson-databind:2.7.3:compile
Installing sinatra 1.4.7
Installing rspec 3.5.0
Installing logstash-core 5.1.1.snapshot2
  jar dependencies for logstash-core-5.1.1.snapshot2-java.gemspec . . .
      org.apache.logging.log4j:log4j-1.2-api:2.6.2:compile
      org.apache.logging.log4j:log4j-api:2.6.2:compile
      org.apache.logging.log4j:log4j-core:2.6.2:compile
      com.fasterxml.jackson.core:jackson-core:2.7.4:compile
      com.fasterxml.jackson.core:jackson-databind:2.7.4:compile
Installing rspec-wait 0.0.9
Installing logstash-core-plugin-api 2.1.19
Installing logstash-codec-json 3.0.2
Installing logstash-codec-line 3.0.2
Installing logstash-codec-plain 3.0.2
Installing logstash-devutils 1.2.0
Installing logstash-codec-json_lines 3.0.2
Using logstash-input-stdin 3.2.0 from source at `.`
Bundle complete! 5 Gemfile dependencies, 54 gems now installed.
Bundled gems are installed into ./vendor.
Post-install message from jar-dependencies:

Note the version difference between the logstash-core-event and the logstash-core, we will need to pin the version, but its not the source of the described problem.

When you try to run the test in a plugin repository it will fail with the following error:

NameError: missing class name (`org.apache.logging.log4j.Level')
  org/jruby/javasupport/JavaUtilities.java:54:in `get_proxy_or_package_under_package'

My current theory is the require order changed or something is missing in the $LOAD_PATH and the logstash-core_jars is not correctly loaded. We can test it that by adding a require 'logstash-core_jars` at the top of the plugin class and make sure it's loaded before everything else in the spec file. By doing so it will fail somewhere else, so this seem to incline that something changed in the $LOAD_PATH

NameError: uninitialized constant LogStash::Environment
  org/jruby/RubyModule.java:2719:in `const_missing'
  /Users/ph/es/logstash-plugins/logstash-input-stdin/vendor/jruby/1.9/gems/logstash-core-5.1.1.snapshot2-java/lib/logstash/util.rb:150:in `Util'
  /Users/ph/es/logstash-plugins/logstash-input-stdin/vendor/jruby/1.9/gems/logstash-core-5.1.1.snapshot2-java/lib/logstash/util.rb:5:in `(root)'
  org/jruby/RubyKernel.java:1040:in `require'
  /Users/ph/es/logstash-plugins/logstash-input-stdin/vendor/jruby/1.9/gems/logstash-core-5.1.1.snapshot2-java/lib/logstash/util/byte_value.rb:1:in `(root)'

[ph]

If we take a step back and look at a previous failling in a plugin travis job

  file:/home/travis/.rvm/rubies/jruby-1.7.25/lib/jruby.jar!/jruby/kernel19/kernel.rb:13:in `require_relative'
  /home/travis/build/logstash-plugins/logstash-output-elasticsearch/vendor/bundle/jruby/1.9/gems/logstash-devutils-1.2.0-java/lib/logstash/devutils/rspec/spec_helper.rb:15:in `(root)'
  org/jruby/RubyKernel.java:1059:in `load'

We see in the trace something related to devutils, at line 15 we require the logging file, we do not require anything else from Logstash before that point.

# from spec_helper
require "logstash/logging"

At that time in the test the logstash-core_jars is not required, making the jar_import fails?

[ph]

@colinsurprenant You are right, jar dependencies should have picked the dependencies, like log4j and we clearly see it in the bundle install log. Adding *.jar in the gemspec is still required to make sure the gem contains our custom logstash-core.jar which is not declared on in the jar-dependencies requirements and also not on maven.

[colinsurprenant]

@ph agree, but we only need to include our generated jar, not the dependencies which are installed by jar-dependencies

[ph]

fixed in the latest snapshot 5.1.2 of the gem.

[cpiment]

Hi @ph ,

Is there any workaround in 5.1.1? I tried to reference the gem snapshot in my gemspec but it requires 'logstash-core-event-java (= 5.1.2.snapshot1) java' which is not deployed in rubygems

[ph]

@cpiment Can you try again? we have released a new version of the core gems 5.1.1.1?

[cpiment]

It worked

Thanks!

[MichaelRuhwedel]

Hi,
sorry for commenting on a closed issue, but it seems as if the logstash-core 5.1.2 depends on logstash-core-queue-jruby 5.1.2, which is not released. Could you please release "logstash-core-queue-jruby" 5.1.2?

[jsvd]

@MichaelRuhwedel thanks for the heads up, https://rubygems.org/gems/logstash-core-queue-jruby/versions/5.1.2-java has been published

[rnowif]

Hi,

I am using logstash-core-plugin-api 2.1.22 which depends on logstash-core 5.1.1.1 (https://rubygems.org/gems/logstash-core-plugin-api) and thus I have the same error when trying to run my tests.

Do you know when logstash-core-plugin-api will be released on RubyGems with a dependency to a newer logstash-core ?

[danielmitd]

@rnowif : To get rid of

NameError: missing class name (`org.apache.logging.log4j.Level')
# and/or
You have already activated jar-dependencies 0.3.9, but your Gemfile requires jar-dependencies 0.3.10.

Explicitly specifying this:

-s.add_development_dependency "logstash-devutils", 
+s.add_development_dependency "logstash-devutils", "~> 1.3", ">= 1.3.1"

worked for me.

ref: https://github.com/elastic/logstash-devutils/commit/75b8e934349529b9bb98ebf9aadfe3a97f622c62

[rnowif]

Wow, thank you so much for this :-)