Logstash: Official way to add a Gem inside Logstash for use with Ruby Filter?

[damienalexandre]

At the moment there is no official way to add a Gem on Logstash to make it available for all plugins. As I didn't want to write my own plugin for this (because there is a lot of boilerplate, releases to do... and I only need 10 lines of ruby) here is what I did.

I added this to the root Gemfile (/usr/share/logstash/Gemfile):

gem "device_detector", "~> 1.0"

I ran this to force Logstash updating the gems:

/usr/share/logstash/bin/logstash-plugin update logstash-filter-grok

Then I can see my Gem in /usr/share/logstash/vendor/bundle/jruby/1.9/gems/device_detector-1.0.0/ :+1: and use it inside a ruby filter for example:

filter {
  if [user_agent] =~ /.+/ {
    ruby {
      init => 'require "device_detector"'
      code => 'ua = event.get("user_agent")

        client = DeviceDetector.new(ua)

        if (client.known?)
          event.set("[ua_detect][os][name]", client.os_name)
          event.set("[ua_detect][os][version]", client.os_full_version)
          event.set("[ua_detect][client][name]", client.name)
          event.set("[ua_detect][device][type]", client.device_type)
          event.set("[ua_detect][device][model]", client.device_name)
        end'
    }
  }
}

It works but it feels like a hack...

• is it supposed to work or is there a better way?
• the bundler update is not exposed and I had to use a bin/logstash-plugin update call, maybe it should be added as a proper command in the bin directory?

Thanks!
_This way done with Logstash 5.3.1._

[andrewvc]

The real smell here IMHO is the complexity in creating a plugin. What can we do to make that easier? At the minimum we will need that plugin to be a separate gem packaged up separately. Did you read https://www.elastic.co/guide/en/logstash/current/_how_to_write_a_logstash_filter_plugin.html ?

The problem here is that adding gems + ruby snippets ad-hoc is not maintainable. It may work as a one off in your case, but it's very hard for other users to reuse your code. The plugin packaging system has tools to let plugin authors guarantee logstash core version compatibility make upgrading plugins easy, etc.

[andrewvc]

To be clear, we appreciate you opening this issue, and I'm sorry that you haven't had the ideal experience for your specific use case. I'm trying to ask, looking at Logstash as a whole, would this sort of functionality be a net positive or negative.

[damienalexandre]

Thanks for the feedback,
so yeah I did look at the documentation about creating a plugin for Logstash and there is nothing wrong with it (maybe the part about needing a github account is too much but ok).

I think building a plugin is the right way, be it public or not, it's just more robust, can be tested, etc.

But it's also a more strict and time consuming process - and that's what I like a lot about the "ruby" filter: it allow me to build proof of concept filters very quickly, directly in my logs pipeline, without having to compile anything for example. I can iterate, index some logs with my code, rinse and repeat fast.
But it lack this ability to have dependencies, like in my snippet.

I don't know if this is a common request from other users, or if plugin developers can already "live edit" a Gem plugin as easily as a ruby filter in a config file?

[jsvd]

allowing users to add dependencies is correctly supported in the creation of a custom plugin.
On the other hands, bunder is the underlying dependency manager so editing the Gemfile works online as bundler will download the remaining gem.

Between these two I don't see what else can be provided.
Editing the is discouraged as you pretty much can do whatever you want, like delete dependencies, introduce new ones, put version restrictions on gems, etc. At that point you are on your own as it's hard to know if something won't break.

This actually brings up a good point: that logstash should not present the Gemfile upfront in root of the tar.gz/zip uncompressed folder since it's a ruby specific detail.

[ionutzp]

i actually had to to the same thing, install a ruby gem for use with the ruby filter plugin so it is a valid need @damienalexandre

[Fustzecutor]

Seconded. If possible, the mechanism to add additional ruby gems/plugins should be easier. This is one of the reasons why Elastic is not the goto platform because data enrichment is somewhat difficult without ruby devs.