Logstash: Mass update plugins for 5.0.0 release

Created on 17 Mar 2016 · 47Comments · Source: elastic/logstash

mass update and publish to all plugins needed because of the constraints in the gemspec for plugins.

See for example: https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/logstash-output-elasticsearch.gemspec#L27

s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"

ain't gonna cut it for 5.0

blocker v5.0.0 v5.0.0-alpha1

Source

suyograo

Most helpful comment

After discussing this over on zoom with the team we will try the following.

[x] Yank the gems that had a < 6.0.0.alpha1 dependency, since they made the build fail.
[ ] Create a gem that will match the plugin api, the gem will be named logstash-core-plugin-api and will be empty for now.
[ ] This gem will hold the constraints to the actual logstash-core
[x] Create one plugin that depends on that gem and test it in logstash 2.X and next 5.X release.
[ ] Mass update the plugin with the new dependency on logstash-core-plugin-api and the updated changelog.

ph on 21 Mar 2016

👍2

All 47 comments

we had spoken about creating a level of indirection between logstash-core and logstash-plugins with a logstash-core-api gem, that we could keep at a lower minor version and change at a slower pace..I guess we completely forgot this during 2.x

jsvd on 17 Mar 2016

@jsvd I remember this discussion, the only downside it will make plugins incompatible with older version and we will need to backport stuff.

Related to this discussion. https://github.com/elastic/logstash/issues/4236

ph on 17 Mar 2016

Discussing with @jsvd on that we are seeing two differents way of making this work.

update the logstash-core constrains to be "> 2.0.0.alpha1" and "< 6.0.0.alpha1"
OR, create a new gems called logstash-core-api|logstash-core-contract and keep the version outside of the normal logstash release versioning, make the plugins depends on it. This gems should have a dependency on "logstash-core >= 2.0.0.alpha1" to make the plugins on 2.0. And the logstash itself depends on a specific version of logstash-core-contract

ph on 17 Mar 2016

@jsvd and I created a graph to brainstorm around the constraints we think that will make it work and result in less mass update.
dependencies

ph on 17 Mar 2016

@colinsurprenant we would like your feedback on ^

ph on 17 Mar 2016

small errors in the graph updated with a middle version of ES.

ph on 17 Mar 2016

@ph @jsvd since we are running short of time for alpha1, can we do this:
update the logstash-core constrains to be "> 2.0.0" and "< 6.0.0"

not sure why we need the alphas?

suyograo on 18 Mar 2016

just a quick heads up, while discussing this this morning, I mentioned that the original intent of the logstash-core gem was to actually be the _contract_ and to achieve this we only need to stop sync'ing the logstash-core gem version with the logstash release version. A major version bump in logstash-core will only be necessary to signal incompatible core API changes.

In practical terms, this would mean that the different logstash-core-* gem versions would change independently from the logstash package release. For example a logstash 5.0 release could contain a logstash-core 2.8, logstash-core-event-java 2.6.3, etc.

I think everyone agreed to the main idea. Are there any caveat we should consider before moving forward with this?

colinsurprenant on 18 Mar 2016

After doing another discussion with @jsvd @andrewvc we decided to do a mass update with the following constraints "> 2.0.0" and "< 6.0.0.alpha1" for alpha1 only so we can move forward AND do the following for alpha 2

Do more testing with making the version of the gems not following Logstash version
Decide on the scheme we will be using for the 2.X releases we still need to support these version and release logstash-core gems for them.
Depending of the results from 1, yank the gems if we need to prevent people from installing theses versions. (I will create a script for that)

@colinsurprenant Should we create an issue for the logstash-core versioning?
Specifically how we will be handling the scenario in in 2.X One idea was do align the minor version with the major of logstash.

ph on 18 Mar 2016

@ph I am not sure I understand why this is needed, we could release logstash 5.0.0.apha1 but keep the current logstash-core version? Is this only to buy more time for testing? I think that moving forward with such a change is ok in an alpha context?

colinsurprenant on 18 Mar 2016

We can't just use logstash-core's major version as the "contract-api" gem, because logstash-core also has code that changes and is different for each major version of logstash.

This means that we also need to use the minor version of logstash-core to synchronize with logstash "product" major release.

So, for example, logstash-core 2.2.x would be used for logstash 2.x and logstash-core 2.5.x would be used for logstash 5.x

With this logic, we'd still need to create a new release of logstash-core (2.5.0) but avoid updating the plugins.

This strategy seems sane and what @ph and @andrewvc and I agreed is that we're still not sure about all the potential edge cases of this versioning scheme, so for now we'd follow the current plan, we can still change things until beta1

jsvd on 18 Mar 2016

@jsvd @colinsurprenant it need to be done for alpha2

ph on 18 Mar 2016

the thing is that we'll forever have a logstash-core gem in rubygems.org using 5.0.0.alpha1
not a big deal, but can be annoying when we break the api 3 more times :D

jsvd on 18 Mar 2016

Well we can still yank it.. Since its an alpha no?

ph on 18 Mar 2016

So the problem we really have here is the dual purpose of logstash-core has being the plugins API and the core logstash functionality which can move at different pace.

What if we extract the plugins related API code into it's own gem? (let's call it logstash-core-plugins)

This way plugins would only depend on the logstash-core-plugins version?

colinsurprenant on 18 Mar 2016

that was our initial purpose with a "logstash-contract" gem in the second comment. while the proposal describes this gem as being empty, the idea was to gradually gravitate the contract api code to it.

jsvd on 19 Mar 2016

@jsvd ok, but I'd go ahead and move the plugins api related code in it. Also, I suggest we call it something else than contract or api, maybe logstash-core-plugins ?

I think we should bite the bullet and move forward with this, otherwise we'll carry this dependency problem for a while in the 5.x cycle.

Colin

colinsurprenant on 21 Mar 2016

I think we should bite the bullet and move forward with this, otherwise we'll carry this dependency problem for a while in the 5.x cycle.

If I understand correct @colinsurprenant you are suggesting that we go ahead with the graph that @jsvd and I proposed? (minus a better name)

ph on 21 Mar 2016

@ph what I suggest is to move the plugins API code in a gem called logstash-core-plugins (any better name?) so that the version constrain related to the core plugins API can move at a different pace than the logstash-core version.

colinsurprenant on 21 Mar 2016

.. and also moving the actual boundary code to the gem.

I think "logstash-plugin-api" or contract makes sense. imo a "logstash-core-plugins" gem is misleading (doesn't actually contain core plugins)

jsvd on 21 Mar 2016

@jsvd what do you mean by «moving boundary code» do you see something different than the "plugins API code" I mentioned?

colinsurprenant on 21 Mar 2016

my comment was meant to be a followup to @ph's, completing his comment with your intention

jsvd on 21 Mar 2016

we'll find a way to understand each other :grin: :mega: :mega:

colinsurprenant on 21 Mar 2016

We certainly will, :laughing:

ph on 21 Mar 2016

ahah <3

colinsurprenant on 21 Mar 2016

recap of the impact of dependency restrictions in the choosing of prerelease gems:

using .a in minimum version allows pre released gems:

% cat Gemfile
source "https://rubygems.org"
gem "logstash-core-event", ">= 2.0.0.a", "< 6.0.0"
% bundle update
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using logstash-core-event 2.3.0.snapshot2
Using bundler 1.11.2
Bundle updated!

Using only X.Y.Z does not use prerelease gems:

% cat Gemfile
source "https://rubygems.org"
gem "logstash-core-event", ">= 2.0.0", "< 6.0.0"
% bundle update
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using logstash-core-event 2.2.2 (was 2.3.0.snapshot2)
Using bundler 1.11.2
Bundle updated!

Using .a in maximum version allows pre releases:

% cat Gemfile
source "https://rubygems.org"
gem "logstash-core-event", ">= 2.0.0", "< 6.0.0.a"
% bundle update
Fetching gem metadata from https://rubygems.org/..
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using logstash-core-event 2.3.0.snapshot2 (was 2.2.2)
Using bundler 1.11.2
Bundle updated!

Bonus:
(predictably) using X.999.0 does not allow prerelease gems:

% cat Gemfile
source "https://rubygems.org"
gem "logstash-core-event", ">= 2.0.0", "< 5.999.0"
% bundle update
Resolving dependencies...
Using logstash-core-event 2.2.2 (was 2.3.0.snapshot2)
Using bundler 1.11.2
Bundle updated!

jsvd on 21 Mar 2016

👍2

After discussing this over on zoom with the team we will try the following.

[x] Yank the gems that had a < 6.0.0.alpha1 dependency, since they made the build fail.
[ ] Create a gem that will match the plugin api, the gem will be named logstash-core-plugin-api and will be empty for now.
[ ] This gem will hold the constraints to the actual logstash-core
[x] Create one plugin that depends on that gem and test it in logstash 2.X and next 5.X release.
[ ] Mass update the plugin with the new dependency on logstash-core-plugin-api and the updated changelog.

ph on 21 Mar 2016

👍2

@jsvd I think we can use this in the logstash-core-plugin-api, Yes its ugly but it will be only used in this gem. WDYT?

gem "logstash-core", ">= 2.0.0", "< 5.999.0"

ph on 21 Mar 2016

@ph @jsvd but how will that work for building a pre-release logstash package? it will include a pre-release version of logstash-core and logstash-core-plugin-api need to resolve that too?

colinsurprenant on 21 Mar 2016

ACTUALLY, do we need to establish a dependency to logstash-core in logstash-core-plugin-api gemspec?! the logstash Gemfile will list what versions of the two gems are necessary then when installing plugins, dependency resolution will be done using the logstash-core-plugin-api version no?

colinsurprenant on 21 Mar 2016

@colinsurprenant I will update this issue with the usecase we have talked on zoom.

ph on 21 Mar 2016

We need to define a constraints on the logstash-core-plugin-api because we are removing the constraints from the plugin itself. Without that limit an incompatible plugin could be installed in previous version of logstash 1.5 or logstash 2.0. This is the only way to make sure we can't install it.

Let me walk through how the versions will work on different side.

Logstash 2.X

We dont need to change or add dependencies, it should pickup the last it can install, I will test this.(The last in this case is the last logstash-core-plugin-api that match the contraints)
plugins depends on logstash-core-plugin-api ~> 1.0
logstash-core-plugin-api depends on "logstash-core", ">= 2.0.0", "< 5.999.0"
plugins can be installed on 2.X and 5.0.0.alpha1

Logstash 5.0.0.alpha1, The api didn't not change.

plugins depends on logstash-core-plugin-api ~> 1.0
logstash-core-plugin-api depends on "logstash-core", ">= 2.0.0", "< 5.999.0"
plugins can be installed on 2.X and 5.0.0.alpha1

Logstash 5.0.0.alpha2, the api changes

plugins depends on logstash-core-plugin-api ~> 2.0
logstash-core-plugin-api depends on "logstash-core", ">= 2.999.0", "< 5.999.0"
plugins can only be installed on 5.0+
logstash 5.0.alpha2 depends on logstash-core-plugin-api "2.0"

Logstash 5.0.0, the api didnt change from the alpha2

plugins depends on logstash-core-plugin-api ~> 2.0
logstash-core-plugin-api depends on "logstash-core", ">= 2.999.0", "< 5.999.0"
plugins can only be installed on 5.0+
logstash 5.0 depends on logstash-core-plugin-api "2.0"

Logstash 6.0.0, the api didn't changes

plugins depends on logstash-core-plugin-api ~> 2.0
release a 2.1 version of logstash-core-plugin-api depends on "logstash-core", ">= 2.999.0", "< 7.999.0"
plugins can only be installed on 5.X and 6.X**
logstash 6.0 depends on logstash-core-plugin-api "2.1"

ph on 21 Mar 2016

@ph but for Logstash 2.X, all plugins will be mass updated to now depend on logstash-core-plugin-api ~> 1 too? so logstash-core-plugin-api will need to depends on "logstash-core", what version?

colinsurprenant on 21 Mar 2016

Same as alpha1, since they need to be installed/work in both environment.

logstash-core-plugin-api depends on "logstash-core", ">= 2.0.0", "< 5.999.0"

ph on 21 Mar 2016

@ph ok, can you also include the logstash-core-plugin-api actual version in your example above?

colinsurprenant on 21 Mar 2016

updated.

ph on 21 Mar 2016

yeah, I think we have a solution right there :heart: I'll re-read a bit later once I performed some GC to re-assess!

colinsurprenant on 21 Mar 2016

Have a working test case, looks good so far

ph on 22 Mar 2016

Have to use --full-index to make bundler works, the dependencies api is still borked :~/

ph on 22 Mar 2016

I have created the followings gems:
https://rubygems.org/gems/ph-core-plugin-api (test of a contract gem)
https://rubygems.org/gems/logstash-input-thisisonlyatest (a plugin)

With different mix of dependencies.

| version | 1.5.5 | 2.2.2 | 5.0 | 6.0 |
| --- | --- | --- | --- | --- |
| plugin 1.X, core-plugin 1.X | :x: | :white_check_mark: | :x: | :x: |
| plugin 2.X, core 2.X | :x: | :x: | :white_check_mark: | :white_check_mark: |

logstash 1.5.5

~/e/s/logstash-1.5.5 ❯❯❯ bin/plugin list --verbose onlyatest
ERROR: No plugins found
~/e/s/logstash-1.5.5 ❯❯❯  bin/plugin install logstash-input-thisisonlyatest                                                                                                                                   ⏎
Validating logstash-input-thisisonlyatest
Installing logstash-input-thisisonlyatest
Error Bundler::InstallError, retrying 1/10
An error occurred while installing logstash-input-thisisonlyatest (1.0.1), and Bundler cannot continue.
Make sure that `gem install logstash-input-thisisonlyatest -v '1.0.1'` succeeds before bundling.
WARNING: SSLSocket#session= is not supported
^C%

logstash 2.2.2.

~/e/s/logstash-2.2.2 ❯❯❯ bin/plugin install logstash-input-thisisonlyatest                                                                                                                                                                                                                                                ⏎
Validating logstash-input-thisisonlyatest
Installing logstash-input-thisisonlyatest
Installation successful
~/e/s/logstash-2.2.2 ❯❯❯ bin/plugin list --verbose logstash-input-thisisonlyatest
logstash-input-thisisonlyatest (1.0.3)
~/e/s/logstash-2.2.2 ❯❯❯

logstash 5.0

With this patch applied https://gist.github.com/9ce087619055b3bf5a74

/t/logstash-5.0 git:master ❯❯❯ bin/plugin install logstash-input-thisisonlyatest                                                                                                                                                                                                                                        ⏎ ✱
Validating logstash-input-thisisonlyatest
Installing logstash-input-thisisonlyatest
WARNING: SSLSocket#session= is not supported
WARNING: SSLSocket#session= is not supported
Installation successful
/t/logstash-5.0 git:master ❯❯❯ bin/plugin list --verbose logstash-input-thisisonlyatest                                                                                                                                                                                                                                 ⏎ ✱
logstash-input-thisisonlyatest (2.0.2)
/t/logstash-5.0 git:master ❯❯❯

logstash 6.0

with gist https://gist.github.com/06e77042e684ff1a4e68

This build could be faster, please consider using the Gradle Daemon: https://docs.gradle.org/2.8/userguide/gradle_daemon.html
/t/logstash git:master ❯❯❯ bin/plugin install logstash-input-thisisonlyatest                                                                                                                                                                                                                                              ✱
Validating logstash-input-thisisonlyatest
Installing logstash-input-thisisonlyatest
Installation successful
/t/logstash git:master ❯❯❯ bin/plugin list --verbose logstash-input-thisisonlyatest                                                                                                                                                                                                                                       ✱
logstash-input-thisisonlyatest (2.0.1)
/t/logstash git:master ❯❯❯

ph on 22 Mar 2016

fun fact, previous versions of logstash will pickup the latest version of the plugins they can handle with the defined dependencies. The user doesn't need to give a version number to install a plugin.

ph on 22 Mar 2016

I tested to explicitely set the ph-core-plugin-api gem to 2.1 with the 6.0 release and it worked like a charm.

ph on 22 Mar 2016

@jsvd / @colinsurprenant Can you take a look at my test and proposal and the following PR https://github.com/elastic/logstash/pull/4866 ?

ph on 22 Mar 2016

After discussing with @jsvd I had a bit more thought on that to make it work with preleases.
All the logic need to be handled with the logstash-core-plugin-api constraints. I am wondering if this workflow would be better.

Scenario we don't break api, Logstash 5.X to Logstash 6.0

We didn't not introduce any breaking changes of the API, so both will use a 2.X release of the logstash-core-plugin-api. Logstash starting with 5.0, the software will depends on a specific version of the logstash-core-plugin-api, this version will support the current logstash-core that the package is shipped with.

During development

The logstash-core-plugin-api is a local gem only and need to have constraints defined to support the current logstash. This version need to be in the 2.X range, so the plugins can be installed in that installation. So in a development branch, master, 2.X, 5.X It will match the current version of the core.

LOGSTASH_CORE_VERSION = "6.0.0.dev"
Gem::Requirement.new(">5.0.0.a", "<= #{LOGSTASH_CORE_VERSION").satisfied_by?(Gem::Version.new("6.0.0.dev"))

When generating a pre-release.

We create a logstash-core-plugin-api 2.X and we do a minor or patch increment, this release need to have the right constraints to support the pre-release logstash. The prerelease version of logstash depends on it. This is exactly what we have done before on 150 plugins.

LOGSTASH_CORE_VERSION = "6.0.0.alpha1"
Gem::Requirement.new(">5.0.0.a", "<= #{LOGSTASH_CORE_VERSION}").satisfied_by?(Gem::Version.new("6.0.0.alpha1"))

We relax the _upper_ bound of the constraint

Contracts for devs.

Plugins developers define the constraint on the logstash-core-plugin-api as ~> 2.0, this mean when you do a bundler install inside the plugin directory it will pickup the last version of it, which will pickup the latest version of the logstash-core gem, including all the other gems, I believe this is an _OK_ behavior

ph on 22 Mar 2016

In a zoom call with @suyograo @colinsurprenant and myself we have decided to go for the simple route and add use the current flow:

Plugins depends on a logstash-core-plugin-api with this pessimistic constraint ~> 2.0 (2.X.X)
On every release of logstash, including prerelease we will release a new version of the logstash-core-plugin-api, it will call rubygems to get the latest released version and will increase the minor version.
The logstash-core-plugin-api will have a strict dependency on a specific release of logstash-core.
Logstash itself will have a strict dependency on logstash logstash-core-plugin-api