Searchkick: Add option to reindex with multiple threads

[ankane]

Use multiple threads to send data to Elasticsearch to speed up reindex for larger clusters.

Product.reindex(concurrency: 4)

https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#_use_multiple_workers_threads_to_send_data_to_elasticsearch

[ankane]

I added this in the threads branch. However, in the benchmarks on my machine, it's not faster. Still trying to identify the bottleneck, but happy if others want to help.

git clone https://github.com/ankane/searchkick.git
cd searchkick
git checkout threads
BUNDLE_GEMFILE=benchmark/Gemfile bundle
BUNDLE_GEMFILE=benchmark/Gemfile bundle exec rake benchmark

[ankane]

Think it has to do with thread scheduling. New direction: will try to do parallel requests with Faraday + Typhoeus.

[ankane]

In performance benchmarks, Ruby code to query the DB and generate search data is the biggest bottleneck. Using ActiveJob to parallelize this across multiple processes / machines (not just threads) is best, but requires some orchestration.

[ankane]

^ Done. Check out https://github.com/ankane/searchkick#large-data-sets

[bbugh]

That link is now https://github.com/ankane/searchkick#parallel-reindexing