Searchkick: Auto - reindex? analyzer [searchkick_search] not found

[jakcharlton]

This error is a bit of a pain for me, as reindex needs to be run whenever searchkick is upgraded

We have partitioned out ES so that each tenant in our multi-tenant app has their own index per model, I will have to write a Rake task to be run after deploy that loops through each tenant and reindexes each model

Could Searchkick not automatically reindex post upgrade? Or even just automatically if it gets this error?

[ankane]

What does your reindex code currently look like? I'm not sure Searchkick supports this use case at the moment.

[jakcharlton]

The rake task is just a modification of the built in one

But this requires that after a deploy, if Searchkick has been upgraded, or methods have changed, that this is run - while it is being run, all ES queries via Searchkick will exception - and downtime for clients

Not sure where/how you could do it - but current way is a little flaky, even though it is an infrequent event

namespace :searchkick do
  namespace :reindex do
    desc "reindex all machines"
    task :all_machines => :environment do
      puts "Reindex loading"
      Rails.application.eager_load!
      Machine.each do |machine|
        puts "-" * 80
        puts "Reindexing MACHINE : #{machine.name}"
        puts "-" * 80
        Machine.current = machine
        (Searchkick::Reindex.instance_variable_get(:@descendents) || []).each do |model|
          puts "Reindexing #{model.name}..."
          model.reindex
        end
        puts "-" * 80
        puts "Reindex complete"
        puts "-" * 80
        puts ""
      end
    end
  end
end

[ankane]

It's hard to tell without knowing more about how Machine works, but here's the general case that causes the analyzer [searchkick_search] not found error:

A record is indexed (via save!, etc.) without its model being indexed. This causes Elasticsearch to create a new index without any of the Searchkick settings (analyzers, etc.). Basically:

Product.searchkick_index.delete
Product.first.reindex
Product.search "test"
# error since Product.reindex is never called

[jakcharlton]

So for example here ... use a Basecamp style "Project" in place of Machine - same concept (a group of data)

I was under the impression - possibly mistaken - that indexes were created as needed by Searchkick

So a user creates a new account and a new Machine (or Project) - do we need to get Searchkick to reindex at that point, or will it do so when it finds no index at all?

  searchkick({index_name: -> { "#{self.name.tableize}_#{Machine.current.database_name}" } })

[ankane]

Searchkick won't reindex automatically since this can have performance implications. If you reindex when a new Machine is created, it should take care of the problem.

[ankane]

You could also add application level logic to reindex when you encounter that error, although the other approach is better IMO.

[jakcharlton]

Ok - so that's a new problem I didn't realise we had ... cheers :)

On Tue, Apr 8, 2014 at 3:24 PM, Andrew Kane [email protected]:

Searchkick won't reindex automatically since this can have performance
implications. If you reindex when a new Machine is created, it should take
care of the problem.

Reply to this email directly or view it on GitHubhttps://github.com/ankane/searchkick/issues/176#issuecomment-39812834
.

[jakcharlton]

The logic to reindex upon that error is I guess what I think Searchkick
should do, or at least have config to allow it to do so, so that the logic
doesnt get distributed through an app

On Tue, Apr 8, 2014 at 3:26 PM, Andrew Kane [email protected]:

You could also add application level logic to reindex when you encounter
that error, although the other approach is better IMO.

Reply to this email directly or view it on GitHubhttps://github.com/ankane/searchkick/issues/176#issuecomment-39812904
.

[ankane]

Hmm, just realized a problem with the setup. Searchkick will reindex all records into that index, not a subset. Think my original thought was correct that Searchkick doesn't support this case at the moment.

[jakcharlton]

I'm not certain it can ...

I think you are saying that asking Searchkick to reindex will grab all "Contacts' for example, and index them all into each index?

That's not possible in my app ... as the models are partitioned too (hence Machine.current)

At least I'm pretty sure it isn't possible ... (Contact.all would get all for the current Machine)

class Contact
  include Mongoid::Document
  include Mongoid::Timestamps
  store_in database: ->{ Machine.current.database_name }

  searchkick({index_name: -> { "#{self.name.tableize}_#{Machine.current.database_name}" } })

...

[ankane]

Ah, gotcha. It should work in that case.

I'm still hesitant to add an option for this. I'm not sure how to explain exactly what this option does but that it's dangerous and not recommended for 99.9% of use cases.

[jakcharlton]

The conditions where your documentation says a reindex is required would
seem to be fairly common, and until the reindex is done, I presume
exceptions will occur (most apps wont be catching them) - unless your
documentation is just being cautious?

On Tue, Apr 8, 2014 at 3:48 PM, Andrew Kane [email protected]:

Ah, gotcha. It should work in that case.

I'm still hesitant to add an option for this. I'm not sure how to explain
exactly what this option does but that it's dangerous and not recommended
for 99.9% of use cases.

Reply to this email directly or view it on GitHubhttps://github.com/ankane/searchkick/issues/176#issuecomment-39813897
.

[ankane]

The documentation is cautious, but a reindex does not cause this issue / downtime.

This issue is caused by not calling Model.reindex in the first place (or in your case, not calling Model.reindex for each Machine)

[jakcharlton]

OK- I'll take your word for it then, have seen this a fair few times, but
I'm on a dev box so may have done something odd to break it :)

Thanks a lot for looking into it

On Tue, Apr 8, 2014 at 3:55 PM, Andrew Kane [email protected]:

The documentation is cautious, but a reindex does not cause this issue /
downtime.

This issue is caused by not calling Model.reindex in the first place (or
in your case, not calling Model.reindex for each Machine)

Reply to this email directly or view it on GitHubhttps://github.com/ankane/searchkick/issues/176#issuecomment-39814177
.

[ankane]

Yeah, no problem.

[ankane]

Cleaning up issues

[cmrichards]

So I've suddenly got this error on our production searchkick server :

QueryParsingException[[comments_production] [match] analyzer [searchkick_search] not found];

I've no clue how it happened - I haven't done any major upgrades or changes. The only way that searchkick is updated is via the searchkick gem I use in some of my models.

how has this happened and how can I fix it? thanks.

[cmrichards]

I've tried re-indexing everything by running "RAILS_ENV=production rake searchkick:reindex:all" but I get another error :

Reindexing Comment...
rake aborted!
400 : {"error":"InvalidAliasNameException[[comments_production_20140602165906429] Invalid alias name [comments_production], an index exists with the same name as the alias]","status":400}
/home/sal/apps/rails/project/vendor/bundle/ruby/2.1.0/gems/searchkick-0.3.4/lib/searchkick/reindex.rb:28:in `reindex'

[cmrichards]

Ok fixed it.

Ran :+1:
curl -XDELETE 'hostname:9200/comments_production'
RAILS_ENV=production rake searchkick:reindex:all

Somehow only the comments_production index was affected.