Searchkick: Reindex of 40,000+ documents getting time-out error [Faraday::TimeoutError: Net::ReadTimeout]

[dsandber]

I'm reindexing a table and getting a TimeoutError. The indexing is working (I can see the document count increasing until the error occurs). The error happens 5-30 minutes into the re-indexing. I've tried changing Searchkick.timeout to 120 but it doesn't help.

irb(main):025:0> Job.reindex
Faraday::TimeoutError: Net::ReadTimeout
from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:158:in rescue in rbuf_fill' from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:152:inrbuf_fill'
from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:134:in readuntil' from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:144:inreadline'
from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:39:in read_status_line' from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:28:inread_new'
from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1408:in block in transport_request' from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1405:incatch'
from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1405:in transport_request' from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1378:inrequest'
from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1371:in block in request' from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:853:instart'
from /app/vendor/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1369:in request' from /app/vendor/bundle/ruby/2.1.0/gems/faraday-0.9.0/lib/faraday/adapter/net_http.rb:80:inperform_request'
from /app/vendor/bundle/ruby/2.1.0/gems/faraday-0.9.0/lib/faraday/adapter/net_http.rb:39:in call' from /app/vendor/bundle/ruby/2.1.0/gems/faraday-0.9.0/lib/faraday/rack_builder.rb:139:inbuild_response'
... 11 levels...
from /app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/notifications.rb:159:in block in instrument' from /app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/notifications/instrumenter.rb:20:ininstrument'
from /app/vendor/bundle/ruby/2.1.0/gems/activesupport-4.0.5/lib/active_support/notifications.rb:159:in instrument' from /app/vendor/bundle/ruby/2.1.0/gems/searchkick-0.8.3/lib/searchkick/logging.rb:46:inimport_with_instrumentation'
from /app/vendor/bundle/ruby/2.1.0/gems/searchkick-0.8.3/lib/searchkick/reindex.rb:64:in block in searchkick_import' from /app/vendor/bundle/ruby/2.1.0/gems/activerecord-4.0.5/lib/active_record/relation/batches.rb:76:infind_in_batches'
from /app/vendor/bundle/ruby/2.1.0/gems/activerecord-deprecated_finders-1.0.3/lib/active_record/deprecated_finders/relation.rb:70:in find_in_batches' from /app/vendor/bundle/ruby/2.1.0/gems/activerecord-4.0.5/lib/active_record/querying.rb:8:infind_in_batches'
from /app/vendor/bundle/ruby/2.1.0/gems/searchkick-0.8.3/lib/searchkick/reindex.rb:63:in searchkick_import' from /app/vendor/bundle/ruby/2.1.0/gems/searchkick-0.8.3/lib/searchkick/reindex.rb:32:inreindex'
from (irb):25
from /app/vendor/bundle/ruby/2.1.0/gems/railties-4.0.5/lib/rails/commands/console.rb:90:in start' from /app/vendor/bundle/ruby/2.1.0/gems/railties-4.0.5/lib/rails/commands/console.rb:9:instart'
from /app/vendor/bundle/ruby/2.1.0/gems/railties-4.0.5/lib/rails/commands.rb:62:in <top (required)>' from /app/bin/rails:4:inrequire'

[ankane]

The Elasticsearch server is most likely overloaded or running out of memory. Try giving it more resources.

[dsandber]

Hmm, probably right. I’m using a Heroku add-on service for it, but yeah, you’re right, I should take it up with them. Thanks.

[dsandber]

I now have had this same problem with two different Heroku Elasticsearch providers -- SearchBox and Bonsai. I'm going to do the re-index manually by saving each record and updating a flag to keep track of the last re-index of each record.

[petriewong]

I am also facing similar problem. My app was installed in Heroku and using Bonsai (Business). When I tring to reindex a index with more than 1M record (about 10GB) using a detached PX dyno, the reindex process would just exit after few hours. Now, I try to reindex one by one recode.

I suggest that add range reindex function into searchkick. :)

[dsandber]

I’m guessing that the elastisearch server times out eventually. If reindex could take a batch-size it would probably work fine.

[zealot128]

You might try to add a smaller batch size to the searchkick call:

class Product < ActiveRecord::Base
  searchkick batch_size: 200 # defaults to 1000
end

[gbarillot]

Thanks, solved my problem using a RDS postgres instance on AWS

[mninaut]

@zealot128 ty

[ankane]

Cleaning up issues.

[diogolmenezes]

I had this problem and when my calculate indices manually, everything went back to work

rake searchkick:reindex:all

or if you want item by item

Product.reindex
User.reindex

[aliabbas-2012]

Reindexing Project...
rake aborted!
Faraday::TimeoutError: Net::ReadTimeout

Net::ReadTimeout: Net::ReadTimeout

IO::EAGAINWaitReadable: Resource temporarily unavailable - read would block

Tasks: TOP => searchkick:reindex:all
(See full trace by running task with --trace)