Elasticsearch-rails: Searching multiple indices/types (equivalent to "Tire.search")

[spodlecki]

Unfortunately, I've spent possibly too much time looking through this new gem looking for a top level search method. Is there a way to search without using a specific model? (Like Tire.search used to do)

For the most part, I'm using the base client methods to do my searching, however, I think I'd like to handle most of the response parts through this gem. (Such as Kaminari)

Possibly, is there a way to search via Elasticsearch::Model.client.search(search_hash) and use this gem's result formatting?

I'm partially new to this lower level of ES (coming from Tire) so hopefully you know what I'm looking for. Any help is much appreciated!!

Other than having somewhat of a learning curve, LOVING these new gems for ES

[karmi]

Hi, there's no support for "multi model searches" a la Tire... Given the modularity of the gem, it should be quite easy to re-use some of the results decorations etc, I'll have a look into that.

[karmi]

I've spent couple of hours with this, and seems like _some_ multi-model support would be easy to add.

Of course, only when we talk about response.results, not supporting response.records -- thus evading complicated magic in adapters, which would fetch the right records from right tables, re-order them by score, etc. Getting the records should be trivial (included in the demo below.)

I'm quoting a working example below:

# Multi Model Searches
#
# ruby -I lib tmp/multi_model_search.rb
#

require 'pry'
Pry.config.history.file = File.expand_path('../../tmp/multimodel.pry', __FILE__)

require 'logger'
require 'ansi/core'
require 'active_record'
require 'kaminari'

require 'elasticsearch/model'

ActiveRecord::Base.logger = ActiveSupport::Logger.new(STDOUT)
ActiveRecord::Base.establish_connection( adapter: 'sqlite3', database: ":memory:" )

ActiveRecord::Schema.define(version: 1) do
  create_table :people do |t|
    t.string :first_name, :last_name
    t.timestamps
  end
  create_table :companies do |t|
    t.string :title
    t.timestamps
  end
end

Kaminari::Hooks.init

class Person < ActiveRecord::Base
  include Elasticsearch::Model
end

class Company < ActiveRecord::Base
  include Elasticsearch::Model
end

Person.create first_name: 'John',   last_name: 'Smith'
Person.create first_name: 'Robert', last_name: 'Smith'

Company.create title: "John 's Bakery"
Company.create title: "Robert 's Carpentry"

Person.import(force: true)
Company.import(force: true)

Elasticsearch::Model.client.indices.refresh

puts '', '-'*Pry::Terminal.width!

# -------------------------------------------------------------------------------------------------

class MultipleModels < Array
  def client
    Elasticsearch::Model.client
  end

  def ancestors
    []
  end

  def default_per_page
    10
  end

  def inspect
    "MultipleModels: #{super}"
  end
end

module Elasticsearch::Model
  # Search multiple models
  #
  def search(query_or_payload, models=[], options={})
    if models.empty?
      models = Object.constants
      .select { |c| Kernel.const_get(c).respond_to?(:__elasticsearch__) }
      .map    { |c| c.is_a?(Class) ? c : Kernel.const_get(c) }
    end

    models = MultipleModels.new(models)

    index_names    = models.map { |c| c.index_name }
    document_types = models.map { |c| c.document_type }

    search   = Searching::SearchRequest.new(
                 models,
                 query_or_payload,
                 {index: index_names, type: document_types}.merge(options)
               )

    Response::Response.new(models, search)
  end

  module_function :search
end

# -------------------------------------------------------------------------------------------------

response = Elasticsearch::Model.search 'John';

puts "Total:", response.results.total, '-'*80

puts "Results:", response.to_a.map(&:to_hash), '-'*80

puts "Records:", response.results.map { |r| r['_type'].classify.constantize.where(id: r['_id']) }.inspect, '-'*80

puts "Page 2:", response.page(2).to_a.inspect, '-'*80

Pry.start(binding, prompt: lambda { |obj, nest_level, _| '> ' }, quiet: true)

[spodlecki]

Thank you! I ended up writing my "own" gem for this. I'm still using this gem for indexing, but we def required a multiple model / type search.

I did yank a lot of your response code (sorry). I referenced this gem a lot in the readme & code itself. The main purpose of the gem is faceted navigation & basic search though, https://github.com/viperdezigns/elastic-engine

Probably not the best code, but its my first stab at using the new ES

[karmi]

I guess I don't understand why you need to copy & paste code from the elasticsearch-model gem, instead of just using it as a dependency and building "decorations" on top of that. That is even more striking when you say that the "main purpose" of your gem is faceted navigation and search.

Would you be so kind and provided some reasoning and/or examples?

[spodlecki]

Not sure what examples you are looking for... but the reasoning is simple.

I'm more of a front-end than a back end developer-- so correct me if I'm wrong here. If I used elasticsearch-model as a dependency, wouldn't that be loading all of your modules & classes? Your response code handler (minus facet methods) from ElasticSearch was exactly what I needed to parse out facets & process them. All I was aiming for was to not reinvent the wheel.
( https://github.com/viperdezigns/elastic-engine/tree/master/lib/elastic_engine/response )

Honestly-- I'm not proficient enough in the backend RoR world to properly use your Response code via a dependency. I'd be happy to discuss further on a chat (skype / icq). If you'd like to show me how to use your Gem's Result & Results; I'm all for it.

(only letters) skype: c \ r \ az \ yvi \ pa
(only #s) 60_89 \287_80

[karmi]

Understood and no worries.

Basically, you could only add the elasticsearch-model as a dependency, and then use, extend or monkeypatch whatever classes or modules you would need. That applies to response, or e. g. to your ElasticEngine::Search::Faceted class.

Closing the ticket.

[samqiu]

Nice feature, needed + 1

For building some kind of real-time search (with Twitter Typeahead.js)

like spotlight search on Mac OS

[nono]

It's also a feature I'd like to use on my projects. So +1 too.

[karmi]

Guys, please see my reasoning why this is not a good idea, regarding how the gem loads the records.

It would be possible, though, to add some Elasticsearch::Model.search, which would return the results collection (documents) -- that's why I added the working code above.

When it comes to "magic behaviour" like this, I must admit I'm quite cautious. If you have ideas or code sketches, please put them in!

[nono]

I'm interested by the Elasticsearch::Model.search, which returns the results collection.

[karmi]

@nono Then steal the code above, try it in your use case and give feedback, please! :)

[nono]

@karmi done, it's here -> https://github.com/nono/linuxfr.org/commit/d2fb186b033ea3092c2dabe5ad508d588785fbe5

[karmi]

Nice, have left some comments at the commit. I'll try to come up with some implementation of multi-model search which wouldn't impair usability...

[halloffame]

Can you reopen this issue since you closed #30 and #50 in favor of this one. This will also make it easier for people to find when they are looking through open issues.

[jwoertink]

@karmi That example you posted above, how does it know to call the _msearch api? Would it just be better to manually implement http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-multi-search.html ?

[daino3]

Hello,

Just a quick thought (and I might be missing something here). If you need this functionality, rather than monkey patching Elasticsearch::Model, since Karmi's implementation above inevitably hits elasticsearch-api#actions/search. Why not simply initialize a client w/

c = Elasticsearch::Client.new
c.search q: 'frisky kittens', body: {highlight: {fields: {...} } }

If no indexes are passed, it searches all indexes. Then you can create your own helper methods to convert them into AR objects, etc. Would eliminate the need the ES-model altogether and maintaining monkey-patched code?

[rossdakin]

:+1: @daino3

I just implemented multi-model search this way (shared index, different types).

I referenced the client through Elasticsearch::Model just for grins (assuming default values for logging, etc. are easier to keep consistent that way):

index = 'my-index'
body = {
  query: { ... },
  highlight: { ... },
  suggest: { ... }
}

@response = Elasticsearch::Model.client.search(index: index, body: body)

EDIT: this, of course, assumes that you're only interested in results rather than instantiated records, as @karmi has mentioned above.

[micahlmartin]

@rossdakin That works perfectly for me too. I don't need the direct models integration. I think this should be documented in a more prominent place.

[karmi]

Hi all, thanks for the discussion and the examples! I'll be travelling in the coming days, but I'll keep the issue in mind and will return to it as soon as possible (mainly finishing the work in the dsl branch of elasticsearch-ruby)

[webattitude]

Hi, what about single model, but multiple time-based indices ?

[webattitude]

is wildcard in index_name allowed ? or a list ?

[solaleye]

Late to this convo, but really informative!!! Thanks everyone. 👍