Searchkick: Incremental indexing vs bulk indexing?

[rajatgarg79]

Now that we have 17 million products, indexing takes a long time and if it gets interrupted for any reason, our users suffer.

Is there a way to do say, Product.reindex(:id>100000) - such that data is added to same index vs creating a new index for it?

Any pointers will be greatly appreciated.

[ankane]

Hey @rajatgarg79, you can do:

Product.where("id > 100000").find_in_batches do |batch|
  Product.searchkick_index.import(batch)
end

Much better for larger indices.

[dferrazm]

@ankane I think such an important feature should be documented somewhere in Readme. We had a hard time trying to figure out why something like some_company.products.reindex was deleting the entire index and creating another just with some_company's products.

If you want I can go ahead and update that for you.

[Chocksy]

@rajatgarg79 Have a look at this gist also: https://gist.github.com/Chocksy/6efa4833989f51e46fca

[ankane]

For future readers, there's now a resume option if your reindex is interrupted.