Searchkick: Multilingual search best practices

[Midnighter]

I'm wondering if there are any best practices I should be aware of when it comes to search in a multilingual application? What I'm envisioning at the moment goes something like this: I have an Item class which has descriptions in ten different languages.

class Item < ActiveRecord::Base
    searchkick
    has_many :descriptions
end

class Description < ActiveRecord::Base
    belongs_to :item
    belongs_to :language
end

The description has a field title and a longer text field, let's call it full. What I'm imagining right now is to generate a separate index for each language and use its own stemmer. I would then search over all indexes.

Is this a sound strategy? What do you prefer?

[ankane]

@Midnighter I haven't tried something like this, but maybe someone else can weigh in.

[cireficc]

I'll note that Elasticsearch seems to be smart enough to choose the correct stemmer when it indexes content, so you might not have to explicitly set the stemmer. For example, I have a Film model whose titles are in any of 8 languages; the title field is indexed, but the stemmer was never explicitly set. Search is accurate in all languages, including Japanese, Arabic, Chinese, and Russian, all of which have their own unique stemming issues. Elasticsearch just gets it right :+1:

[fredriksundstrom]

I’d love to verify that Elasticsearch does automatic stemming for me, but I’m not sure how. Is it possible to, i.e., get the stemmed searchstring to verify it? Sort of like this: http://9ol.es/porter_js_demo.html

I, for example, am creating a Swedish page. I get one result locally with just Elasticsearch and another with Searchly/Searchbox or Bonsai via Heroku, no matter if I set the language to Swedish or not (and re-indexing). So, would be glad to know if Searchly/Searchbox and Bonsai just does automatic stemming – or no swedish stemming at all, no matter the config. :)

I know there’s an "analyze" endpoint in Elasticsearch, but I don’t really know how to use it from searchkick or manually.

[ankane]

Hey @fredriksundstrom, see the debugging queries section for how to check tokenization and stemming: https://github.com/ankane/searchkick#debugging-queries

[cireficc]

@ankane I looked through that section and tried some debugging on my own Film model to determine what stemmer was being chosen in my search queries, but no stemmer information was given in the output (just the analyzer).

I also looked through the Elasticsearch documentation for a specific CURL request that might provide that information, but I couldn't find anything useful. Is stemmer information even exposed in ES? I see that you can override stemming rules, but I haven't found anything relating to showing which stemmer is chosen when indexing or querying.

[ankane]

Cleaning up stale issues