Logstash: Logstash 5 alpha 5 kafka plugin errors

[olafhartong]

- Version: 5alpha5 (repo install)
- Operating System: Centos 7
- Config File
very elementary

input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
}
}

output {
kafka {
topic_id => CUSTOMER2
compression_type => gzip
}
}

- Steps to Reproduce:

• log output>
  {:timestamp=>"2016-08-18T11:34:46.920000+0000", :message=>"fetched an invalid config", :config=>"input {\n tcp {\n port => 5000\n type => syslog\n }\n udp {\n port => 5000\n type => syslog\n }\n}\n\noutput {\n kafka {\n topic_id => CUSTOMER2\n compression_type => gzip\n }\n}\n\n", :reason=>"Couldn't find any output plugin named 'kafka'. Are you sure this is correct? Trying to load the kafka output plugin resulted in this error: Problems loading the requested plugin named kafka of type output.", :level=>:error}
• First of all the plugin is there;
• Tried to reinstall the plugin in several versions via the update mechanism as well as manually downloading the gem, both fail;
  bin/logstash-plugin install --version 5.0.0 logstash-output-kafka
  Validating logstash-output-kafka-5.0.0
  Installing logstash-output-kafka
  Error Bundler::InstallError, retrying 1/10
  An error occurred while installing logstash-core-event-java (5.0.0.pre.alpha5), and Bundler cannot continue.
  Make sure that gem install logstash-core-event-java -v '5.0.0.pre.alpha5' succeeds before bundling.
  WARNING: SSLSocket#session= is not supported
  Error Bundler::InstallError, retrying 2/10
  An error occurred while installing logstash-core-event-java (5.0.0.pre.alpha5), and Bundler cannot continue.
  Make sure that gem install logstash-core-event-java -v '5.0.0.pre.alpha5' succeeds before bundling.
  WARNING: SSLSocket#session= is not supported
  Error Bundler::InstallError, retrying 3/10
  An error occurred while installing logstash-core-event-java (5.0.0.pre.alpha5), and Bundler cannot continue.
  Make sure that gem install logstash-core-event-java -v '5.0.0.pre.alpha5' succeeds before bundling.
  WARNING: SSLSocket#session= is not supported
  Error Bundler::InstallError, retrying 4/10
  An error occurred while installing logstash-core-event-java (5.0.0.pre.alpha5), and Bundler cannot continue.
  Make sure that gem install logstash-core-event-java -v '5.0.0.pre.alpha5' succeeds before bundling.
  WARNING: SSLSocket#session= is not supported
  Error Bundler::InstallError, retrying 5/10
  An error occurred while installing logstash-core-event-java (5.0.0.pre.alpha5), and Bundler cannot continue.
  Make sure that gem install logstash-core-event-java -v '5.0.0.pre.alpha5' succeeds before bundling.
  WARNING: SSLSocket#session= is not supported
  Error Bundler::InstallError, retrying 6/10
  An error occurred while installing logstash-core-event-java (5.0.0.pre.alpha5), and Bundler cannot continue.
  Make sure that gem install logstash-core-event-java -v '5.0.0.pre.alpha5' succeeds before bundling.
  WARNING: SSLSocket#session= is not supported
  Error Bundler::InstallError, retrying 7/10
  An error occurred while installing logstash-core-event-java (5.0.0.pre.alpha5), and Bundler cannot continue.
  Make sure that gem install logstash-core-event-java -v '5.0.0.pre.alpha5' succeeds before bundling.
  WARNING: SSLSocket#session= is not supported
  Error Bundler::InstallError, retrying 8/10
  An error occurred while installing logstash-core-event-java (5.0.0.pre.alpha5), and Bundler cannot continue.
  Make sure that gem install logstash-core-event-java -v '5.0.0.pre.alpha5' succeeds before bundling.
  WARNING: SSLSocket#session= is not supported
  WARNING: SSLSocket#session= is not supported
• Afterwards the plugin list breaks due to a failed update
  Bundler::GemNotFound: Could not find gem 'logstash-output-kafka (= 5.0.0) java' in any of the gem sources listed in your Gemfile or installed on this machine.
  verify_gemfile_dependencies_are_found! at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:328
  each at org/jruby/RubyArray.java:1613
  verify_gemfile_dependencies_are_found! at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:307
  start at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:199
  resolve at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:182
  resolve at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:192
  specs at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:132
  specs_for at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:177
  requested_specs at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:166
  requested_specs at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/environment.rb:18
  setup at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/runtime.rb:13
  setup at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler.rb:122
  setup! at /usr/share/logstash/lib/bootstrap/bundler.rb:64
  execute at /usr/share/logstash/lib/pluginmanager/list.rb:17
  run at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
  execute at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11
  run at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
  run at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132
  (root) at /usr/share/logstash/lib/pluginmanager/main.rb:43

[suyograo]

@splitbrains thanks for filing this issue. We've fixed this in subsequent Kafka fixes and will also be fixed in our next pre-release. Until then, you can do:

bin/logstash-plugin update logstash-input-kafka

[olafhartong]

@suyograo thanks for your response. I guess i'll have to wait for the next pre-release, since I am unable to update (as mentioned in the above post)
Is there a date set for the next pre-release already?

[joekiller]

@splitbrains what command did you run to update? It isn't listed in your bullets. 5.0.4 is already released so you should be able to to either download the gem and manually install it or run the command suyograo mentioned.

https://rubygems.org/gems/logstash-input-kafka/versions/5.0.4

https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html#installing-local-plugins

[carnalim]

I get the following:

/usr/share/logstash/bin/logstash-plugin install logstash-output-kafka-5.0.4.gem
Validating logstash-output-kafka-5.0.4.gem
Installing logstash-output-kafka
Error Bundler::InstallError, retrying 1/10
An error occurred while installing logstash-core-event-java (5.0.0.pre.alpha5), and Bundler cannot continue.
Make sure that `gem install logstash-core-event-java -v '5.0.0.pre.alpha5'` succeeds before bundling.

[dcode]

I was able to get this to at least function by manually downloading the plugins (both input and output) and do an offline install. The problem is that the install hangs (and requires javac?). After canceling it after it hangs, logstash seems to operate just fine.

With a clean install of logstash 5 alpha 5, logstash tells me it can't find a plugin for my kafka input stanza.

[robert-blankenship]

Same issue

[shoggeh]

@dcode out of curiousity, how did you manage to install this plugin offline?

I'm battling with similar issue in https://github.com/elastic/logstash/issues/5966 and can't find working solution.

[dcode]

@shoggeh I just brute forced my way through and it happened to work. So, the plugin includes the necessary JAR files to access Kafka. I assume this is why it's running javac, to recompile for the local platform. It stalls for a bit and I cancelled it by hitting Ctrl+c. I checked that the plugin was installed in /usr/share/logstash/vendor/local_gems/, and then it just worked.

Obviously sketchy, but I was able to reproduce that 4 times successfully, with only one failure when I cancelled too early.

[shoggeh]

That's completely insane, but seems to be working for other plugins too.

[ph]

The related kafka bug seems to be fixed in the latest releases so do th eoffline issue.