Elasticsearch-rails: Mapping definition for [title_suggest] has unsupported parameters: [payloads : true]

[regularlady]

@karmi thank you for all of your incredible work on Elastic. We happily converted our repositories to Elastic a few months ago and are giving a talk in our community about why we did so.

I'm eager to add in autocomplete. I read over your excellent completion suggester example here.

Here is my (abbreviated) code:

class Page
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  settings index: { number_of_shards: 1 } do
    mappings dynamic: 'false' do
      indexes :title, analyzer: 'english', index_options: 'offsets'
      indexes :title_suggest, type: 'completion', payloads: true
      indexes :description, analyzer: 'english'
      indexes :site_id, analyzer: 'english'
    end
  end


  def as_indexed_json(options={})
    as_json.merge \
    title_suggest: {
          input:  title,
          output: title,
          payload: { url: "/pages/#{id}" }
      }
  end
end

For importing, I tried both:

Page.__elasticsearch__.client.indices.delete index: Page.index_name rescue nil
Page.__elasticsearch__.client.indices.create \
  index: Page.index_name,
  body: { settings: Page.settings.to_hash, mappings: Page.mappings.to_hash }
Page.import query: -> { where(status_id: 100, indexable: true, class_name: [nil, '']) }, batch_size: 100

and

Page.__elasticsearch__.create_index! force: true
Page.__elasticsearch__.refresh_index!
Page.import query: -> { where(status_id: 100, indexable: true, class_name: [nil, '']) }, batch_size: 100}

Unfortunately, I'm consistently getting:

Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Mapping definition for [title_suggest] has unsupported parameters: [payloads : true]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [page]: Mapping definition for [title_suggest] has unsupported parameters: [payloads : true]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [title_suggest] has unsupported parameters: [payloads : true]"}},"status":400}

I'm using elasticsearch-5.1.1 with elasticsearch-model and elasticsearch-rails gems installed on Rails 4.2.7. Is it possible that payload was deprecated? I appreciate any guidance on how to map the Page.title to the title_suggest field so that I can call:

 Elasticsearch::Model.client.suggest(index: "pages",
                                                     body: {
                                                         suggestion: {
                                                             text: params[:q].to_s,
                                                             completion: {
                                                                 field: 'title_suggest'
                                                             }
                                                         }
                                                     })

Thanks!

[bublik]

https://www.elastic.co/guide/en/elasticsearch/reference/5.2/search-suggesters-completion.html

[karmi]

Hi, thank you for such kind words!

The example has been changed, actually, to support the new syntax in Elasticsearch 5.x, which removed the support for the payload options: https://github.com/elastic/elasticsearch-rails/blob/16ef3934b4408de950e6c676fac0c88f4eb5ff7b/elasticsearch-model/examples/activerecord_mapping_completion.rb#L22

Now, instead of returning the payload, the whole hit is returned from Elasticsearch, so you can use any regular property of the document. In order to save bandwidth and memory, it makes sense to return only properties which you care about, by using the "_source filtering" feature (see _source_include and _source_exclude features.

I must say that I liked the _payload_ support, since it was a nice way how to decouple regular document properties from the "suggest" properties, but removing the support is a good technical improvement in Elasticsearch and also makes everything more consistent. In fact, in one of the example applications available in this repository, I've "faked" the payload option like this:

attribute :suggest, Hashie::Mash, mapping: {
    type: 'object',
    properties: {
      title: {
        type: 'object',
        properties: {
          input:   { type: 'completion' },
          output:  { type: 'keyword', index: false },
          payload: { type: 'object', enabled: false }
        }
      }
    }

In this way, whatever you want as your output is duplicated in suggest.title.output, which is not indexed (to save space in the inverted index), and the additional data are stored in suggest.title.payload, where the indexing is also disabled.

The full source is here: https://github.com/elastic/elasticsearch-rails/blob/master/elasticsearch-persistence/examples/music/album.rb#L33-L43

Of course, the structure must be taken into account when indexing, see an example here: https://github.com/elastic/elasticsearch-rails/blob/master/elasticsearch-persistence/examples/music/index_manager.rb#L49-L56

Then, when you search, you use _source: ['suggest.*'] to filter only those fields. Full example here: https://github.com/elastic/elasticsearch-rails/blob/master/elasticsearch-persistence/examples/music/suggester.rb

But take this only as a hint that the separation of "search" data and "suggest" data can still be achieved -- I imagine that in most cases, using regular source filtering is enough.

One last hint: instead of having the title_suggest field, it might make more sense to use the "multi fields" feature, and "embed" the suggest field into the title, somewhat like this:

mappings dynamic: 'false' do
  indexes :title, analyzer: 'english', index_options: 'offsets' do
    indexes :suggest, type: 'completion'
    # ...
  end
  #...
end

Then, you can reference the field as title.suggest in your queries. This pattern is really useful for situations where you want to index a property like last_name one way for searching (eg. with the english analyzer) and another way for sorting or aggregations (as the keyword type).

Actually, I've just changed the example in question into the format I'm suggesting, see the commit attached to this issue!

Please let me know if you would continue to struggle with that!

[regularlady]

That is exactly what I needed. Thank you x a million @karmi! 🎉