Elasticsearch-rails: ES 6.x support

[peter-hank]

Does the gem support Elasticsearch 6.x? If not, do you plan to support it anytime soon?

[karmi]

Hi @peter-hank, the libraries should _in general_ support Elasticsearch 6.x, but it might require some additional configuration, namely when it comes to removal of types. Eg. the elasticsearch-model in the default configuration will try to use multiple types per model class, so a user configuration which makes it use a separate index for each model class might be needed.

Also, the tests and examples haven't been yet updated to 6.x, which is something needed... We do plan to support 6.x as soon as possible in the default configuration of the gems, but I don't have a concrete date.

[sly7-7]

@karmi In order to test this, I guess I must use the master branch ? (Also, maybe you want some help to update the tests and examples ?)

[Aesthetikx]

Just anecdotal evidence, I have been seeing HTTP 406 errors on 6.x, haven't dug into why though.

Failure/Error: model.__elasticsearch__.create_index!(force: true)

     Elasticsearch::Transport::Transport::Errors::NotAcceptable:
       [406] {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

[tenaz3]

How can I have the elasticsearch 6 with elasticsearch-rails 5 ?

```Bundler could not find compatible versions for gem "elasticsearch":
In Gemfile:
elasticsearch (~> 6.0)

elasticsearch-model (>= 5.0.2, ~> 5.0) was resolved to 5.0.2, which depends on
  elasticsearch (~> 5)

elasticsearch-model (>= 5.0.2, ~> 5.0) was resolved to 5.0.2, which depends on
  elasticsearch (~> 5)

```

[karmi]

@tenaz3, so what is exactly the problem with 6 and 5? The elasticsearch-model has a very relaxed dependency on the Elasticsearch client. I've just generated a new Rails 5, and when added the following to the Gemfile:

gem 'elasticsearch', '~> 6'
gem 'elasticsearch-model', '~> 5'

this is what I see after running bundle install:

$ grep 'elasticsearch' Gemfile.lock 
  elasticsearch (6.0.0)
    elasticsearch-api (= 6.0.0)
    elasticsearch-transport (= 6.0.0)
  elasticsearch-api (6.0.0)
  elasticsearch-model (5.0.0)
    elasticsearch (> 1)
  elasticsearch-transport (6.0.0)
 elasticsearch (~> 6)
 elasticsearch-model (~> 5)

which seems to satisfy the requirement?

[karmi]

@Aesthetikx, this would indicate that you're using an older elasticsearch gem, can you check that you have upgraded the dependency for elasticsearch to something like ~> 6?

[lulessa]

@karmi Your suggestion works. However, since v5.0.1, elasticsearch-model has depended on 'elasticsearch', '~> 5'.

https://github.com/elastic/elasticsearch-rails/blob/3b6f27d371c848d04488bf77a294fd1ff25929e4/elasticsearch-model/elasticsearch-model.gemspec#L26

Following your suggestion to add this to Gemfile

gem 'elasticsearch', '~> 6'
gem 'elasticsearch-model', '~> 5'

Note that it will install elasticsearch-model 5.0.0. I suppose this is good enough (we don't get the patch versions) because we are really looking forward to version 6.

[surjay]

Are there any updates on the timeline for this? It looks like master is pointing towards 6.0alpha now? ES is officially at 6.1.3 as of January.

[kaspergrubbe]

It would be wonderful to have an ETA, are we looking at Q2 or Q3?

[karmi]

Right! We are looking at investing some significant effort into the elasticsearch-rails family of gems in couple of months, so realistically, Q3 would be the answer here. (Again, please note that the dependencies are very relaxed and the -rails gems should be freely mixable with -ruby gems, and the main practical issue I see is the configuration for index_name and dcoument_type, so the gem doesn't try to mash multiple models into one index.)

[jalberto]

it doesn't works for elasticsearch-persistence

Bundler could not find compatible versions for gem "elasticsearch":
  In Gemfile:
    elasticsearch (~> 6.0)

    elasticsearch-persistence (~> 5.0) was resolved to 5.0.2, which depends on
      elasticsearch (~> 5)

[karmi]

@jalberto, that's really strange, can you post your Gemfile (and possibly a Ruby version), so I can run it?

[jalberto]

@karmi

ruby 2.4.1

My Gemfile is quite long but this is the relevant part:

group :default, :elk do
  gem 'elasticsearch' , "~> 6.0"
  gem "elasticsearch-persistence" , "~> 5.0"
end

This is lockfile relevant part with versions set to 5.x:

    elasticsearch (5.0.4)
      elasticsearch-api (= 5.0.4)
      elasticsearch-transport (= 5.0.4)
    elasticsearch-api (5.0.4)
      multi_json
    elasticsearch-model (5.0.2)
      activesupport (> 3)
      elasticsearch (~> 5)
      hashie
    elasticsearch-persistence (5.0.2)
      activemodel (> 4)
      activesupport (> 4)
      elasticsearch (~> 5)
      elasticsearch-model (~> 5)
      hashie
      virtus
    elasticsearch-transport (5.0.4)
      faraday
      multi_json

[leifcr]

This requirement elasticsearch (~> 5) corresponds to elasticsearch (~> 5.0) which will be the same as elasticsearch (> 6.0, <=5.0)

In order to support both 5.x and 6.x, this requirement must be changed to >= 5.0

[karmi]

Sorry for the silence here. @leifcr's suggestion sounds right, would that work for you. As I wrote in the other tickets, I still need to focus on other responsibilities at the moment, but we're planning a significant attention to the elasticsearch-rails group of libraries soon.

[jalberto]

@karmi I still need to test that solution, but if it works is good to go.

[karmi]

@jalberto , let me know please. I'll find a time to try a Gemfile on my side if it doesn't work.

[pctj101]

As a note for the curious, for example this kind of parent-child type does not work:
elasticsearch-model/test/integration/active_record_associations_parent_child_test.rb

Where it says something like:

  mapping _parent: { type: 'question' }, _routing: { required: true } do

This is the kind of document/test which would need to be updated and operate fairly differently than previous ES versions.

In my case, I may need to do a gradual migration between ES vOld and ES v6, so it would be nice for class names to be entirely different so a ESv2 gem could be run simultaneously with a ESv6 gem during transition. But not sure if that makes sense for the maintainers.

[leifcr]

PR #797 is missing fixing the parent relationship tests to use join mapping instead of _parent.

[estolfo]

Hi all
We are working towards a 6.x release of the gem in a few weeks. Currently, the work and testing against ES 6.x is in the 6.x branch.

Is there any reason you think we should keep this issue open seeing as we have active development now in the 6.x branch?

[kaspergrubbe]

@estolfo Please keep this issue open until support for 6.0 is shipped. It will be a nice indicator how things are progressing for everyone interested.

[estolfo]

ok, no problem

[kaspergrubbe]

@estolfo How come the label for this issue is waiting-for-user, what do you need from us to continue?

[estolfo]

Sorry @kaspergrubbe, I put that label when I originally asked if I could close the issue. I'll take that label off now.

[estolfo]

Hello, I wanted to let you all know that there are a 6.0.0.pre versions available of the elasticsearch-rails, elasticsearch-model, and elasticsearch-persistence gems, if you'd like to try them out in your testing environments.
Please open a GitHub issue if you'd like to report anything. Thanks!

[estolfo]

I'm pinging this thread again to remind everyone that a 6.0.0.pre version is available for the rails integration gems. Please open an issue if you find any.

[estolfo]

6.0.0 of the rails integration gems is released.

[jalberto]

@estolfo looks like Elasticsearch::Persistence.client= is not present in version 6. Is there any Changelog with breaking changes? (Changelog file seems quite old)

[estolfo]

Hi @jalberto
You will have to set a client on the Repository class or instance instead.
You may find this documentation useful.
Please see the CHANGELOG on the 6.x branch.
And this blog post has in-depth explanations of how to use the Repository pattern. Let me know if you have any specific questions!

[jalberto]

thanks @estolfo still I cannot see a "breaking changes" list, will be very useful for a migration from 5.x to 6.x.

Also will be great to know why those breaking changes happened, in particular if not deprecation warnings has been implemented. I am sure there are good reasons behind that decisions, still a good practice.

[estolfo]

Hi @jalberto, I've created entries for the two releases (6.0.0.pre and 6.0.0) and put the list of changes there. I've also included a list of breaking changes in the details for 6.0.0. Please let me know if there's anything else I can do to make the change log clearer for these versions.
I'll continue using the Releases list in the future so that it's easier to parse what changes go into each version and so that I can communicate breaking changes more effectively. Thanks for your suggestions!

[jalberto]

thanks @estolfo that is what I needed.Great work.
Right now the Changelog and release notes contains different information so it's bit confusing.

[estolfo]

@jalberto
The CHANGELOG is automatically generated as part of a rake task when we release the gems. I'm going to leave it as-is for now. I'm considering removing it and converting all the release notes to the GitHub releases so that there's only one source of the change log.

[estolfo]

I'm going to close this issue, as the gems supporting Elasticsearch 6.0 have been released for over a week.