Elasticsearch-rails: ES 7.0.0 support?

[albertosilletti]

We moved from ES 6.7.0 to 7.0.0
We are unable to execute Expansion.__elasticsearch__.create_index! where Expansion is a very simple ActiveRecord model without anything particular.

class Expansion < ApplicationRecord
  require 'elasticsearch/model'
  include Elasticsearch::Model
end

2.6.1 :041 > Expansion.__elasticsearch__.create_index!
Elasticsearch::Transport::Transport::Errors::BadRequest ([400] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."}],"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."},"status":400})

Gems:
elasticsearch-model (6.0.0)
elasticsearch-transport (7.0.0.pre)
elasticsearch (~> 7.0.0.pre)
elasticsearch-api (= 7.0.0.pre)
elasticsearch-transport (= 7.0.0.pre)

Thank you

[karmi]

Hi! It looks like you're using the 6.x version of the Rails integrations, ie. elasticsearch-model (6.0.0) — can you upgrade that to 7.x as well?

[albertosilletti]

Unfortunately 6.0.0 (Feb 2018) is the latest version. We were able to update all the other gems to 7.0.0pre, except for elasticsearch-model

[estolfo]

@albertosilletti elasticsearch-model has not been released for Elasticsearch 7.0 yet but it will be released soon. Thanks for your patience.

[albertosilletti]

Thanks a lot! Any eta.....? is this planned "in the weeks" or "months"?

[estolfo]

A pre version will most likely be available early next week and then a GA soon after that.

[skukx]

Linking the Pull Request for this:
https://github.com/elastic/elasticsearch-rails/pull/875

[estolfo]

@albertosilletti @skukx elasticsearch-rails v7.0.0.pre gems are now available: https://rubygems.org/gems/elasticsearch-rails/versions/7.0.0.pre
Please try them in your testing environments and report any issues. This is a pre release so not recommended for production use. Thanks!

[skukx]

Hi @estolfo I haven't seen any issues with elasticsearch-rails v7.0.0.pre and Elasticsearch 7.0.0. However, I am curious if elasticsearch-rails v7.0.0 gem will be backward compatible with Elasticsearch 6.7. I ask this since it will determine our upgrade strategy.

Thank you for all your work!

[estolfo]

The gems supporting ES 7.0 are not currently being tested against ES 6.x but I would encourage you to:

• Review the release notes for 7.0.0.pre.
• Read the info here on the removal of types, in particular see the option include_type_name and its uses.
• Review your code to see where you might want to include that option or adjust the code for the removal of types.
• Test the rails 7.0.0.pre gems in your testing environment. Changes in the gem for 7.0 were made backwards compatible, when possible. For example, this one.

If you test 7.0.0.pre against ES 6.x and find an issue, would you please open it?

[reyanshmishra]

Any progress on this when will this support ES v7.0.0 ? I am testing it with

elasticsearch (7.1.0)
elasticsearch-api (7.1.0)
elasticsearch-model (7.0.0.pre)
elasticsearch-transport (7.1.0)

and still getting

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [jixee_documentsection : {dynamic=false, properties={title={analyzer=english, type=text, index_options=offsets}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [jixee_documentsection : {dynamic=false, properties={title={analyzer=english, type=text, index_options=offsets}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [jixee_documentsection : {dynamic=false, properties={title={analyzer=english, type=text, index_options=offsets}}}]"}},"status":400}

with create_index!(force: true)

I think may be it is because mapping types removed.
Hey @skukx you didn't get this error with create_index! method?

[estolfo]

@reyanshmishra Did you get this error when you tried to create the index for a repository?
If so, would you mind sharing the output of calling #mapping on the repository object? Thanks

[reyanshmishra]

@estolfo I am sorry that was my stupidity. I had document_type in there.

[estolfo]

No worries! Glad you were able to resolve it.

[estolfo]

By the way, I was hoping to release the elasticsearch-rails gems v7.0 including support for Rails 6 because the timing seemed to line up. Rails 6 still hasn't been released but I'm going to wait a little bit longer before investigating other options. For example, I might add support for Rails 6 in a future minor 7.x release.

[fakenine]

Hello @estolfo,

I tried to update to the v7.2 and followed the master branch of this gem that has 7.X, but I'm still getting this error when creating an index for any repository:

Elasticsearch::Transport::Transport::Errors::BadRequest ([400] {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [server : {properties={text={analyzer=snowball, type=text}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [server : {properties={text={analyzer=snowball, type=text}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [server : {properties={text={analyzer=snowball, type=text}}}]"}},"status":400})

I used the same mapping as in the readme:

      settings number_of_shards: 1 do
        mapping do
          indexes :text, analyzer: 'snowball'
        end
      end

If I'm not mistaken, snowball does not exists anymore in 7.x. What could be a good fix?

[estolfo]

@fakenine Do you have a document_type defined for your repositories?

[fakenine]

@estolfo Indeed! My bad, I misread @reyanshmishra's message.
I works when I delete the document_type.

Thank you

[estolfo]

@fakenine Glad to hear it!

[stephenhmarsh]

I'm trying to copy a Rails app on Heroku to another instance because we need an extra environment, and our Elasticsearch provider (Bonsai) provisioned Elasticsearch 7.2.0 on the new app, which introduced me to this lovely error. (Original app is on 6.something)

Fortunately I googled and found this issue, but even with the latest gems, the app connected to the 7.2.0 instance errors out.

Gemfile:

    elasticsearch (7.2.0)
      elasticsearch-api (= 7.2.0)
      elasticsearch-transport (= 7.2.0)
    elasticsearch-api (7.2.0)
      multi_json
    elasticsearch-extensions (0.0.31)
      ansi
      elasticsearch
    elasticsearch-model (6.0.0)
      activesupport (> 3)
      elasticsearch (> 1)
      hashie
    elasticsearch-rails (6.0.0)
    elasticsearch-transport (7.2.0)
      faraday
      multi_json

Example mapping (1 of 3 models I need to import from ActiveRecord):

irb(main):013:0> pp Characteristic.mapping
#<Elasticsearch::Model::Indexing::Mappings:0x0000559a232c8a68
 @mapping=
  {:id=>{:type=>:long},
   :catchy_name=>
    {:type=>:text,
     :analyzer=>"ngram_analyzer",
     :search_analyzer=>"whitespace_analyzer"},
   :scale_low_text=>
    {:type=>:text,
     :analyzer=>"ngram_analyzer",
     :search_analyzer=>"whitespace_analyzer"},
   :scale_high_text=>
    {:type=>:text,
     :analyzer=>"ngram_analyzer",
     :search_analyzer=>"whitespace_analyzer"},
   :description=>
    {:type=>:text,
     :analyzer=>"ngram_analyzer",
     :search_analyzer=>"whitespace_analyzer"}},
 @options={:dynamic=>"false"},
 @type="_doc">
=> #<Elasticsearch::Model::Indexing::Mappings:0x0000559a232c8a68 @type="_doc", @options={:dynamic=>"false"}, @mapping={:id=>{:type=>:long}, :catchy_name=>{:type=>:text, :analyzer=>"ngram_analyzer", :search_analyzer=>"whitespace_analyzer"}, :scale_low_text=>{:type=>:text, :analyzer=>"ngram_analyzer", :search_analyzer=>"whitespace_analyzer"}, :scale_high_text=>{:type=>:text, :analyzer=>"ngram_analyzer", :search_analyzer=>"whitespace_analyzer"}, :description=>{:type=>:text, :analyzer=>"ngram_analyzer", :search_analyzer=>"whitespace_analyzer"}}>

Model source code:

mappings dynamic: 'false' do
    indexes :id, type: :long
    [:catchy_name, :scale_low_text, :scale_high_text, :description].each do |text_field|
       indexes text_field, type: :text, analyzer: 'ngram_analyzer', search_analyzer: 'whitespace_analyzer'
    end
end

When I run create_index! I always get same error, regardless if I pass include_type_name: true or not)

irb(main):001:0> Characteristic.__elasticsearch__.create_index!(include_type_name: true)
2019-07-21 02:01:25 +0000: HEAD https://r31cz7375p:***********@pine-70517907.us-east-1.bonsaisearch.net:443/characteristics [status:404, request:0.020s, query:N/A]
2019-07-21 02:01:25 +0000: < 
2019-07-21 02:01:25 +0000: [404] 
2019-07-21 02:01:25 +0000: PUT https://r31cz7375p:***********@pine-70517907.us-east-1.bonsaisearch.net:443/characteristics [status:400, request:0.018s, query:N/A]
2019-07-21 02:01:25 +0000: > {"settings":{"analysis":{"filter":{"ngram_filter":{"type":"nGram","min_gram":2,"max_gram":20}},"analyzer":{"ngram_analyzer":{"type":"custom","tokenizer":"standard","filter":["lowercase","asciifolding","ngram_filter"]},"whitespace_analyzer":{"type":"custom","tokenizer":"whitespace","filter":["lowercase","asciifolding"]}}}},"mappings":{"_doc":{"dynamic":"false","properties":{"id":{"type":"long"},"catchy_name":{"type":"text","analyzer":"ngram_analyzer","search_analyzer":"whitespace_analyzer"},"scale_low_text":{"type":"text","analyzer":"ngram_analyzer","search_analyzer":"whitespace_analyzer"},"scale_high_text":{"type":"text","analyzer":"ngram_analyzer","search_analyzer":"whitespace_analyzer"},"description":{"type":"text","analyzer":"ngram_analyzer","search_analyzer":"whitespace_analyzer"}}}}}
2019-07-21 02:01:25 +0000: < {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."}],"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."},"status":400}
2019-07-21 02:01:25 +0000: [400] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."}],"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."},"status":400}
Traceback (most recent call last):
        1: from (irb):1
Elasticsearch::Transport::Transport::Errors::BadRequest ([400] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."}],"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."},"status":400})

help pls?

[estolfo]

Hi @stephenhmarsh Thanks for your question.
The problem here is that you are using the elasticsearch-rails gems 6.x series with Elasticsearch server version 7.x and you're specifying a mapping type, which has been deprecated. The older gem is sending a mapping config in a format no longer supported by Elasticsearch 7.0.

We were holding off a bit on the release of the rails gems supporting Elasticsearch 7 because we were hoping that Rails 6 would be released and we could support both in the new major version. Nevertheless, it's uncertain as to when Rails 6 will be available so we will move forward releasing the gems supporting Elasticsearch 7.

That said, we still need to fix a few things before releasing. There is a pre version of the elasticsearch-rails gem available (7.0.0.pre) if you'd like to test with it.

I've also opened a pull request here against the 6.x version of the gems to allow arbitrary options to be sent to the create_index! method. This way, you can pass the include_type_name parameter to the create_index! command using the 6.0.0 gem version connected to Elasticsearch 7.x. If we release 6.1.0 including that arbitrary option support, you can at least continue to use the 6.x gem versions with Elasticsearch 7.x until we release the 7.0.0 gems.

Let me know if you have any questions.

[stephenhmarsh]

Thank you @estolfo that clarifies for me what's going on.

[jon-at-jobber]

Hi! We are running into similar issues (not looking for help, docs and examples above are enough) and I see you've decided to move forward with release rather than wait for Rails6. Any insight into when that will drop?

[estolfo]

The elasticsearch rails gems versions 7.0 have been released, supporting Rails 6 and Elasticsearch 7

[vbrazo]

+1 in the document_type issue. Had to remove to make it work. Thanks 👍

[TomaszKot11]

Indeed in order to use with elasticsearch 7.0 you have to have version 7.0 of a gem :)

[Bachdz]

@estolfo Indeed! My bad, I misread @reyanshmishra's message.
I works when I delete the document_type.

Thank you

Hi. Where can I find and define the document_type ? How did you fix it? Thank you in advance

[slavajacobson]

Same issue here. Followed README instructions to setup the gem on Rails 6, created a basic model, then ran MyModel.__elasticsearch__.create_index! and got the same error...

[slavajacobson]

Fixed with:

gem 'elasticsearch-model', github: 'elastic/elasticsearch-rails', tag: 'v7.0.0'
gem 'elasticsearch-rails', github: 'elastic/elasticsearch-rails', tag: 'v7.0.0'

then running bundle update
and ensuring you don't have document_type in your Model

[enthusiasmus]

Why it is even possible to use document_type with the elasticsearch-model >= v7.0.0 gem?
When I use it with document_type "vendor", I always receive the following error:

Elasticsearch::Transport::Transport::Errors::BadRequest:
  [400] {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [**vendor** : {dynamic=false, properties={...}],"status":400}

Gem-Version: 7.0.0
Elasticsearch-Version: 7.5.2

[skukx]

@enthusiasmus The document type was used in older versions of elasticsearch. The reason the gem still supports this would probably be due to backward compatibility and granting older gem users a path to migrate to the new version.

https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#_schedule_for_removal_of_mapping_types

[HumayunNaseer]

I'm facing this issue on Heroku , elasticsearch working fine locally but on Heroku with bonsai, I'm facing this error anyone has solution? @estolfo ?

[picandocodigo]

Hi @HumayunNaseer, since this issue has been closed for a while and the previous problems been solved, could you please open a new issue? Make sure you include as much detail as possible: client version, Elasticsearch version, what is the exact error you're facing, etc. Thanks!

[HumayunNaseer]

heroku run bundle exec rake environment elasticsearch:import:model CLASS='PickupLocations::PickupLocation' FORCE=true

i was running this command from 2 days and got the same error above and not it is accidentally successful I don't know how?
@picandocodigo