Logstash: [meta] defaults plugins migration to new Event interface and plugin-api 2.0

[colinsurprenant]

This is a meta issue to track progress for refactoring the defaults plugins. Core plugins migration has been completed per #5170. Related #5140.

This migration recipe goals are:

• avoid publishing any core plugins for the refactor
• be able to test refactored plugins within their refactor branch in Travis
• be able to test/build logstash using the refactored plugins within their branches

Plugin Migration Recipe

• create branch feature/plugin-api-2_0
• this branch should be published upstream in the logstash-plugins repo.

• .travis.yml should be updated to

  sudo: false
  jdk:
  - oraclejdk8
  language: ruby
  cache: bundler
  rvm:
  - jruby-1.7.25
  before_install:
  - git clone -b feature/event_interface https://github.com/elastic/logstash
  script:
  - bundle exec rspec spec
  

• minimally these gems need to be added to the plugin Gemfile

  # this is temporary for the feature/plugin-api-2_0 branch and is meant for travis testing
  gem "logstash-core", :path => "./logstash/logstash-core"
  gem "logstash-core-plugin-api", :path => "./logstash/logstash-core-plugin-api"
  gem "logstash-core-event-java", :path => "./logstash/logstash-core-event-java"
  gem "logstash-devutils", :github => "elastic/logstash-devutils", :branch => "feature/plugin-api-2_0"
  

• add any other plugins in the gemspec required by this plugin, for example, if logstash-codec-plain is a required plugin add this:

  gem "logstash-codec-plain", :github => "logstash-plugins/logstash-codec-plain", :branch => "feature/plugin-api-2_0"
  

• modify the gemspec to bump the plugin-api version constrain to 2.0:

  s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
  

• and finally, refactor code and specs for new Event interface

  Running local specs

• if you want to test locally you will have to create a symbolic link to the logstash code base with the feature/event_interface shared branch

  ln -s /you/path/to/logstash .
  bundle
  bundle exec rspec spec
  

  Helpful Scripts

@ph has added this ruby script to automate much of the initial work: https://gist.github.com/e28dddccfbe32c3985ab51d6f9b9ec41

Plugins for core specs

• [x] https://github.com/elastic/logstash-devutils/pull/46
• [x] https://github.com/logstash-plugins/logstash-filter-clone/pull/9 |
• [x] https://github.com/logstash-plugins/logstash-patterns-core/pull/149 |
• [x] https://github.com/logstash-plugins/logstash-filter-grok/pull/81 |
• [x] https://github.com/logstash-plugins/logstash-filter-mutate/pull/72 |
• [x] https://github.com/logstash-plugins/logstash-filter-multiline/pull/27 |
• [x] https://github.com/logstash-plugins/logstash-codec-plain/pull/2 |
• [x] https://github.com/logstash-plugins/logstash-input-generator/pull/13 |
• [x] https://github.com/logstash-plugins/logstash-codec-line/pull/9 |
• [x] https://github.com/logstash-plugins/logstash-codec-json/pull/27 |
• [x] https://github.com/logstash-plugins/logstash-codec-json_lines/pull/26 |
• [x] https://github.com/logstash-plugins/logstash-input-stdin/pull/10 |
• [x] https://github.com/logstash-plugins/logstash-input-tcp/pull/52 |
• [x] https://github.com/logstash-plugins/logstash-output-stdout/pull/8 |
• [x] [logstash-input-heartbeat](https://github.com/logstash-plugins/logstash-input-heartbeat) | @suyograo
• [x] [logstash-output-zeromq](https://github.com/logstash-plugins/logstash-output-zeromq) | @suyograo
• [x] [logstash-codec-collectd](https://github.com/logstash-plugins/logstash-codec-collectd) | @suyograo
• [x] [logstash-output-xmpp](https://github.com/logstash-plugins/logstash-output-xmpp) | @suyograo
• [x] [logstash-codec-dots](https://github.com/logstash-plugins/logstash-codec-dots) | @suyograo
• [x] [logstash-codec-edn](https://github.com/logstash-plugins/logstash-codec-edn) | @suyograo
• [x] [logstash-codec-edn_lines](https://github.com/logstash-plugins/logstash-codec-edn_lines) | @suyograo
• [x] [logstash-codec-fluent](https://github.com/logstash-plugins/logstash-codec-fluent) | @suyograo
• [x] [logstash-codec-es_bulk](https://github.com/logstash-plugins/logstash-codec-es_bulk) | @suyograo
• [x] [logstash-codec-graphite](https://github.com/logstash-plugins/logstash-codec-graphite) | @suyograo
• [x] [logstash-codec-msgpack](https://github.com/logstash-plugins/logstash-codec-msgpack) | @suyograo
• [x] [logstash-codec-multiline](https://github.com/logstash-plugins/logstash-codec-multiline) | @suyograo
• [x] [logstash-codec-netflow](https://github.com/logstash-plugins/logstash-codec-netflow) | @suyograo
• [x] [logstash-codec-oldlogstashjson](https://github.com/logstash-plugins/logstash-codec-oldlogstashjson) | @suyograo
• [x] [logstash-codec-rubydebug](https://github.com/logstash-plugins/logstash-codec-rubydebug) | @suyograo
• [x] [logstash-filter-anonymize](https://github.com/logstash-plugins/logstash-filter-anonymize) | @suyograo
• [x] [logstash-filter-checksum](https://github.com/logstash-plugins/logstash-filter-checksum) | @suyograo
• [x] [logstash-filter-clone](https://github.com/logstash-plugins/logstash-filter-clone) | @suyograo
• [x] [logstash-filter-csv](https://github.com/logstash-plugins/logstash-filter-csv) | @suyograo
• [x] [logstash-filter-date](https://github.com/logstash-plugins/logstash-filter-date) | @suyograo
• [x] [logstash-filter-dns](https://github.com/logstash-plugins/logstash-filter-dns) | @suyograo
• [x] [logstash-filter-drop](https://github.com/logstash-plugins/logstash-filter-drop) | @suyograo
• [x] [logstash-filter-fingerprint](https://github.com/logstash-plugins/logstash-filter-fingerprint) | @suyograo
• [x] [logstash-filter-geoip](https://github.com/logstash-plugins/logstash-filter-geoip) | @suyograo
• [x] [logstash-filter-json](https://github.com/logstash-plugins/logstash-filter-json) | @suyograo
• [x] [logstash-filter-kv](https://github.com/logstash-plugins/logstash-filter-kv) | @suyograo
• [x] [logstash-filter-metrics](https://github.com/logstash-plugins/logstash-filter-metrics) | @suyograo
• [x] [logstash-filter-ruby](https://github.com/logstash-plugins/logstash-filter-ruby) | @suyograo
• [x] [logstash-filter-sleep](https://github.com/logstash-plugins/logstash-filter-sleep) | @suyograo
• [x] [logstash-filter-split](https://github.com/logstash-plugins/logstash-filter-split) | @suyograo
• [x] [logstash-filter-syslog_pri](https://github.com/logstash-plugins/logstash-filter-syslog_pri) | @suyograo
• [x] https://github.com/logstash-plugins/logstash-filter-throttle/pull/11 | @ph
• [x] https://github.com/logstash-plugins/logstash-filter-urldecode/pull/7 | @ph
• [x] https://github.com/logstash-plugins/logstash-filter-useragent/pull/29 | @ph
• [x] https://github.com/logstash-plugins/logstash-filter-uuid/pull/5 | @ph
• [x] https://github.com/logstash-plugins/logstash-filter-xml/pull/29 | @ph
• [x] [logstash-input-couchdb_changes](https://github.com/logstash-plugins/logstash-input-couchdb_changes) | @ph
• [x] [logstash-input-elasticsearch](https://github.com/logstash-plugins/logstash-input-elasticsearch) | @ph
• [x] [logstash-input-eventlog](https://github.com/logstash-plugins/logstash-input-eventlog) | @ph
• [x] [logstash-input-exec](https://github.com/logstash-plugins/logstash-input-exec) | @ph
• [x] [logstash-input-file](https://github.com/logstash-plugins/logstash-input-file) | @ph
• [x] [logstash-input-ganglia](https://github.com/logstash-plugins/logstash-input-ganglia) | @ph
• [x] [logstash-input-gelf](https://github.com/logstash-plugins/logstash-input-gelf) | @ph
• [x] [logstash-input-graphite](https://github.com/logstash-plugins/logstash-input-graphite) | @ph
• [x] [logstash-input-http](https://github.com/logstash-plugins/logstash-input-http) | @ph
• [x] [logstash-input-http_poller](https://github.com/logstash-plugins/logstash-input-http_poller) | @ph
• [x] [logstash-input-imap](https://github.com/logstash-plugins/logstash-input-imap) | @ph
• [x] [logstash-input-irc](https://github.com/logstash-plugins/logstash-input-irc) | @ph
• [x] [logstash-input-jdbc](https://github.com/logstash-plugins/logstash-input-jdbc) | @ph
• [x] [logstash-input-log4j](https://github.com/logstash-plugins/logstash-input-log4j) | @ph
• [x] [logstash-input-lumberjack](https://github.com/logstash-plugins/logstash-input-lumberjack) | @ph
• [x] [logstash-input-pipe](https://github.com/logstash-plugins/logstash-input-pipe) | @ph
• [x] [logstash-input-rabbitmq](https://github.com/logstash-plugins/logstash-input-rabbitmq) | @ph
• [x] [logstash-input-redis](https://github.com/logstash-plugins/logstash-input-redis) | @ph
• [x] [logstash-input-s3](https://github.com/logstash-plugins/logstash-input-s3) | @ph
• [x] [logstash-input-snmptrap](https://github.com/logstash-plugins/logstash-input-snmptrap) | @ph
• [x] [logstash-input-sqs](https://github.com/logstash-plugins/logstash-input-sqs) | @ph
• [x] [logstash-input-syslog](https://github.com/logstash-plugins/logstash-input-syslog) | @ph
• [x] [logstash-input-twitter](https://github.com/logstash-plugins/logstash-input-twitter) | @ph
• [x] [logstash-input-udp](https://github.com/logstash-plugins/logstash-input-udp) | @ph
• [x] [logstash-input-unix](https://github.com/logstash-plugins/logstash-input-unix) | @ph
• [x] [logstash-input-xmpp](https://github.com/logstash-plugins/logstash-input-xmpp) | @guyboertje
• [x] [logstash-input-zeromq](https://github.com/logstash-plugins/logstash-input-zeromq) | @talevy
• [x] [logstash-input-kafka](https://github.com/logstash-plugins/logstash-input-kafka) | @talevy
• [x] [logstash-input-beats](https://github.com/logstash-plugins/logstash-input-beats) | @talevy
• [x] [logstash-output-cloudwatch](https://github.com/logstash-plugins/logstash-output-cloudwatch) | @talevy
• [x] [logstash-output-csv](https://github.com/logstash-plugins/logstash-output-csv) | @talevy
• [x] [logstash-output-elasticsearch](https://github.com/logstash-plugins/logstash-output-elasticsearch) | @talevy
• [x] [logstash-output-email](https://github.com/logstash-plugins/logstash-output-email) | @talevy
• [x] [logstash-output-exec](https://github.com/logstash-plugins/logstash-output-exec) | @talevy
• [x] [logstash-output-file](https://github.com/logstash-plugins/logstash-output-file) | @talevy
• [x] [logstash-output-ganglia](https://github.com/logstash-plugins/logstash-output-ganglia) | @talevy
• [x] [logstash-output-gelf](https://github.com/logstash-plugins/logstash-output-gelf) | @talevy
• [x] [logstash-output-graphite](https://github.com/logstash-plugins/logstash-output-graphite) | @talevy
• [x] [logstash-output-hipchat](https://github.com/logstash-plugins/logstash-output-hipchat) | @talevy
• [x] [logstash-output-http](https://github.com/logstash-plugins/logstash-output-http) | @talevy
• [x] [logstash-output-irc](https://github.com/logstash-plugins/logstash-output-irc) | @talevy
• [x] [logstash-output-juggernaut](https://github.com/logstash-plugins/logstash-output-juggernaut) | @talevy
• [x] [logstash-output-lumberjack](https://github.com/logstash-plugins/logstash-output-lumberjack) | @talevy
• [x] [logstash-output-nagios](https://github.com/logstash-plugins/logstash-output-nagios) | @talevy
• [x] [logstash-output-nagios_nsca](https://github.com/logstash-plugins/logstash-output-nagios_nsca) | @talevy
• [x] [logstash-output-null](https://github.com/logstash-plugins/logstash-output-null) | @talevy
• [x] [logstash-output-opentsdb](https://github.com/logstash-plugins/logstash-output-opentsdb) | @talevy
• [x] [logstash-output-pagerduty](https://github.com/logstash-plugins/logstash-output-pagerduty) | @talevy
• [x] [logstash-output-pipe](https://github.com/logstash-plugins/logstash-output-pipe) | @talevy
• [x] [logstash-output-rabbitmq](https://github.com/logstash-plugins/logstash-output-rabbitmq) | @talevy
• [x] [logstash-output-redis](https://github.com/logstash-plugins/logstash-output-redis) | @talevy
• [x] [logstash-output-s3](https://github.com/logstash-plugins/logstash-output-s3) | @talevy
• [x] [logstash-output-sns](https://github.com/logstash-plugins/logstash-output-sns) | @talevy
• [x] [logstash-output-sqs](https://github.com/logstash-plugins/logstash-output-sqs) | @talevy
• [x] [logstash-output-statsd](https://github.com/logstash-plugins/logstash-output-statsd) | @talevy
• [x] [logstash-output-tcp](https://github.com/logstash-plugins/logstash-output-tcp) | @talevy
• [x] [logstash-output-udp](https://github.com/logstash-plugins/logstash-output-udp) | @talevy
• [x] [logstash-output-kafka](https://github.com/logstash-plugins/logstash-output-kafka) | @talevy
• [x] [logstash-mixin-aws](https://github.com/logstash-plugins/logstash-mixin-aws) @talevy
• [x] [logstash-mixin-http_client](https://github.com/logstash-plugins/logstash-mixin-http_client/pull/21) | @ph
• [x] https://github.com/logstash-plugins/logstash-mixin-rabbitmq_connection/pull/21 | @ph

[ph]

We can automate all of the heavy work minus the actual code/spec?
Are we fine doing that?

[suyograo]

@ph +1 for automating..

[colinsurprenant]

it might be tricky to automate the addition of dependent plugins from the gemspec but the basic block

# this is temporary for the feature/plugin-api-2_0 branch and is meant for travis testing
gem "logstash-core", :path => "./logstash/logstash-core"
gem "logstash-core-plugin-api", :path => "./logstash/logstash-core-plugin-api"
gem "logstash-core-event-java", :path => "./logstash/logstash-core-event-java"
gem "logstash-devutils", :github => "elastic/logstash-devutils", :branch => "feature/plugin-api-2_0"

can be easily automated

[colinsurprenant]

also I've seen a bit different .travis.yml, with the integration tests flag?

[suyograo]

Btw, all plugins should have the integration flag set if its not already there..

[colinsurprenant]

I added the list of plugins done so far for passing core specs.

[ph]

@suyograo logstash-codec-oldlogstashjson is marked as completed but I can't find the PR.

[suyograo]

Here's the merge plan:

• [x] Review PRs for all plugins (@tal, @ph, @suyograo)
• [ ] merge https://github.com/elastic/logstash/pull/5170 to master and 5.0 after cleaning up temp stuff in there (@colinsurprenant)
• [ ] release new 5.0.0-beta1.snapshot1 core plugins (@suyograo)
• [x] Create new branches (plugin-api-v1) off master in all plugins repo for 2.x development (@ph)
• [x] Delete core-api-v2 (@ph)
• [ ] Script merge of feature/plugin-api-2_0 to master in all plugins (@ph)
• [ ] Update plugins .travis.yml, Gemfile via mass update (@ph)
• [ ] Bump major version for all default plugins, add CHANGELOG entry (@ph)
• [ ] Make sure tests are green for all plugins (@ph)
• [ ] and mass publish (@ph)
• [ ] Generate new lock file for master & 5.0 (@colinsurprenant)
• [ ] Generate new lock file for 5.0 with newly published plugins, and release new 5.0.0-beta1.snapshot1 packages for testing (@suyograo)

[colinsurprenant]

@suyograo

• the plugins branches off master will be 2.x ?
• we need to revert both .travis.yml and Gemfile in mass update
• you mean new 5.0.0-beta1.snapshot2 ?

[suyograo]

@colinsurprenant updated.

[colinsurprenant]

@suyograo @ph also, .travis.yml should not be updated to _old state_, we certainly want to keep the Java 8 selection?

[colinsurprenant]

@suyograo @ph core-api-v1 (not v2) branch for LS 2.x development.

[ph]

@colinsurprenant We already have v1, it was before the shutdown change, and used to target LS 1.5?

[colinsurprenant]

@ph how is that possible? we only recently introduced the plugin-core-api gem recently
ah yes, ok, I think I remember - it's core-api, we need to create plugin-api-v1

[suyograo]

@colinsurprenant core-api-v1 branch already exists and was created to continue development of plugins under v1 core API (pre shutdown changes, < LS 2.0). core-api-v2 once branched will match v2 of plugin APIs and continue development for them (pre Java Event, < LS 2.0).

Also, here is a .travis.yml we can mass update with: https://gist.github.com/suyograo/331f6384e143b147c09db51a854540c4

Includes Java 8 and --integration tags

[ph]

Also, here is a .travis.yml we can mass update with: https://gist.github.com/suyograo/331f6384e143b147c09db51a854540c4

I am not sure we should mass update with the bundle exec rspec --tag integration, since integration usually mean that we need to add some steps in the travis file to make sure the environment is ready to be used for integration testing. Like installing redis, rabbitmq or supporting libraries.

But this could be a good way to find which plugins are failling and change their travis.yml manually.

@colinsurprenant @suyograo core-api-v2 I agree is a weird name, another option would be to make it explicit and rename them logstash-1.5.x and logstash-2.x ;)

[ph]

@suyograo After discussing on slack zoom I am ok to move ahead with the core-api-v2, its only a branch name, if we change our mind in the future we can always rename it/them.

@colinsurprenant Can we move forward?

[colinsurprenant]

@ph @suyograo it is still confused.

• core-api-v1 branch was introduced for supporting the pre-shutdown changes for the 1.5 series. At this point we had no notion of plugin-api, we came up with that ad-hoc branch name at the time.
• we need a branch to be able to continue to support le LS versions that are using the plugin-api v1 which are all pre-5.0 releases. if we use the name "v2" in that branch name it will be extremely confusing.
• since we now have a concept of plugin-api formal versioning I think we need to use that concept to name branches and use corresponding names and versions numbers.
• in that respect, the problem here is the original "core-api-v1" branch name which does not make sense with the new plugin-api concept.

For this I suggest we simply leave the core-api-v1 branch name as-is, for legacy reasons, which anyway will not be needed anymore, and introduce branches with the plugin-api name and versions.

[suyograo]

For this I suggest we simply leave the core-api-v1 branch name as-is, for legacy reasons, which anyway will not be needed anymore, and introduce branches with the plugin-api name and versions.

@colinsurprenant @ph I am fine with naming the branch based on plugin API version. Good with plugin-api-v1

[colinsurprenant]

@suyograo, @ph and myself just reviewed the tasks sequence and Houston, all engines go!