Searchkick: Reindex of Collection gives Elasticsearch Transport error

[Ravi1729]

Hello,

I have a model with name Actor.And the model looks like the following snippet

class Actor 
      include Mongoid::Document  
      field :name, type: String 
      field :birthName, as: :birth_name, type: String 
      field :date_of_birth, type: Date 
      field :height, type: Measurement 
      field :bio, type: String 
    searchkick
end

when I try to reindex my model using Actor.reindex. I am getting the following error which shows the root cause as Index already exists.

irb(main):001:0> Actor.reindex
Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"index_already_exists_exception","reason":"index [actors_development_20170209175219593/TKOPkMY1TsGNEYmWlVCKo
A] already exists","index_uuid":"TKOPkMY1TsGNEYmWlVCKoA","index":"actors_development_20170209175219593"}],"type":"index_already_exists_exception","reason":"index [actors_development_20170209175219593/TKO
PkMY1TsGNEYmWlVCKoA] already exists","index_uuid":"TKOPkMY1TsGNEYmWlVCKoA","index":"actors_development_20170209175219593"},"status":400}
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/elasticsearch-transport-5.0.1/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/elasticsearch-transport-5.0.1/lib/elasticsearch/transport/transport/base.rb:318:in `perform_request'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/elasticsearch-transport-5.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/elasticsearch-transport-5.0.1/lib/elasticsearch/transport/client.rb:128:in `perform_request'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/elasticsearch-api-5.0.1/lib/elasticsearch/api/namespace/common.rb:21:in `perform_request'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/elasticsearch-api-5.0.1/lib/elasticsearch/api/actions/indices/create.rb:86:in `create'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/searchkick-2.1.1/lib/searchkick/index.rb:14:in `create'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/searchkick-2.1.1/lib/searchkick/index.rb:177:in `create_index'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/searchkick-2.1.1/lib/searchkick/index.rb:226:in `reindex_scope'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/searchkick-2.1.1/lib/searchkick/model.rb:70:in `searchkick_reindex'
        from (irb):1
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /home/ravi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'from bin/rails:4:in `require'

Since the error is showing Index already exists, then I deleted the index from elasticsearch and tried to run again.But, it is giving me the same error index already exists

Can anyone please help to find the reason behind this error

Thanks,
Ravi.

[ankane]

Hey @Ravi1729, I haven't seen this before. Can you run:

p Time.now.to_i
p Actor.searchkick_index.all_indices
Actor.reindex
p Time.now.to_i
p Actor.searchkick_index.all_indices

and paste the output?

[Ravi1729]

Hey ankane,

I found out the reason behind that.

I am using Searchkick v2.1.1 with Elasticsearch v.5.0.1...which is giving me the transport error.

Then I replaced Elasticsearch v5.0.1 with v2.4.3 .Then it started indexing properly. I guess there is a compatibility issue between searchkick v2.1.1 and Elasticsearch v5.0.1

[ankane]

Hey @Ravi1729, Searchkick works fine with Elasticsearch 5.0.1. Anyways, glad you found something that worked.

[davemate23]

Hi @ankane, I'm having the same issue with the following error:

$ rake searchkick:reindex CLASS=Course
rake aborted!
Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"index_already_exists_exception","reason":"index [courses_development_20170228124357519/4V4nAQ0cQeWPkW8RcfB5ug] already exists","index_uuid":"4V4nAQ0cQeWPkW8RcfB5ug","index":"courses_development_20170228124357519"}],"type":"index_already_exists_exception","reason":"index [courses_development_20170228124357519/4V4nAQ0cQeWPkW8RcfB5ug] already exists","index_uuid":"4V4nAQ0cQeWPkW8RcfB5ug","index":"courses_development_20170228124357519"},"status":400}

I'm using ElasticSearch 5.2.1, but before I revert to 2.4.3, I thought I would send you the reports you asked for from @Ravi1729 to see if there was anything useful in them:

2.3.3 :001 > p Time.now.to_i
1488286198
 => 1488286198 
2.3.3 :002 > p Course.searchkick_index.all_indices
["courses_development_20170228124357519"]
 => ["courses_development_20170228124357519"] 
2.3.3 :003 > Course.reindex
  Course Load (142.5ms)  SELECT  "courses".* FROM "courses" ORDER BY "courses"."id" ASC LIMIT $1  [["LIMIT", 1000]]
 => true 
2.3.3 :004 > p Time.now.to_i
1488286254
 => 1488286254 
2.3.3 :005 > p Course.searchkick_index.all_indices
["courses_development_20170228125033152"]
 => ["courses_development_20170228125033152"]

[Ravi1729]

Hi @davemate23 ,

Recently I changed my elasticsearch version from v2.4.3 to v5.2.1 and searchkick is able to reindex the data without any problem. I guess the problem is not related to version compatibility.

[chrishough]

@davemate23 I am having the same issue, how did you solve yours?

[marcelokanzaki]

Same issue here

[saurabhbhatia]

I have this issue on Circle CI. I'm using Elasticsearch 5.2 for my development and Circle CI has 2.4.3. This errors randomly pops up once in a while.

[benreyn]

I am experiencing the same issue on Circle CI, but only intermittently. Should this issue be re-opened?

[kakra]

I'm seeing this, too... Running "rake searchkick:reindex:all" always produces this issue with one of the models, not necessarily the same on successive runs. Running "Model.reindex" in a rails console almost never produces this issue for me. It seems like there's some timing or locking issue involved, and such a race condition should be fixed.

This happened only after I was forced to upgrade ElasticSearch. So the different behavior is probably in ElasticSearch but I'm confident it's still a bug that should be fixed in Searchkick because it should not depend on specific behavior of very specific ES versions.

Reindexing User...
rake aborted!
Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"index_already_exists_exception","reason":"index [sbcms_development_users_20171201121205965/-xEONhgRRAmsAUYuW4423g] already exists","index_uuid":"-xEONhgRRAmsAUYuW4423g","index":"sbcms_development_users_20171201121205965"}],"type":"index_already_exists_exception","reason":"index [sbcms_development_users_20171201121205965/-xEONhgRRAmsAUYuW4423g] already exists","index_uuid":"-xEONhgRRAmsAUYuW4423g","index":"sbcms_development_users_20171201121205965"},"status":400}
/home/kakra/.rvm/gems/ruby-2.2.5/gems/elasticsearch-transport-6.0.0/lib/elasticsearch/transport/transport/base.rb:202:in `__raise_transport_error'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/elasticsearch-transport-6.0.0/lib/elasticsearch/transport/transport/base.rb:319:in `perform_request'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/elasticsearch-transport-6.0.0/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/elasticsearch-transport-6.0.0/lib/elasticsearch/transport/client.rb:131:in `perform_request'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/elasticsearch-api-6.0.0/lib/elasticsearch/api/namespace/common.rb:21:in `perform_request'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/elasticsearch-api-6.0.0/lib/elasticsearch/api/actions/indices/create.rb:86:in `create'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/searchkick-1.5.1/lib/searchkick/index.rb:13:in `create'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/searchkick-1.5.1/lib/searchkick/index.rb:151:in `create_index'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/searchkick-1.5.1/lib/searchkick/index.rb:201:in `reindex_scope'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/searchkick-1.5.1/lib/searchkick/model.rb:58:in `searchkick_reindex'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/searchkick-1.5.1/lib/searchkick/tasks.rb:26:in `block (4 levels) in <top (required)>'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/searchkick-1.5.1/lib/searchkick/tasks.rb:24:in `each'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/searchkick-1.5.1/lib/searchkick/tasks.rb:24:in `block (3 levels) in <top (required)>'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/cli/exec.rb:75:in `load'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/cli/exec.rb:75:in `kernel_load'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/cli/exec.rb:28:in `run'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/cli.rb:424:in `exec'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/cli.rb:27:in `dispatch'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/cli.rb:18:in `start'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/exe/bundle:30:in `block in <top (required)>'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/lib/bundler/friendly_errors.rb:122:in `with_friendly_errors'
/home/kakra/.rvm/gems/ruby-2.2.5/gems/bundler-1.16.0/exe/bundle:22:in `<top (required)>'
/home/kakra/.rvm/gems/ruby-2.2.5/bin/bundle:23:in `load'
/home/kakra/.rvm/gems/ruby-2.2.5/bin/bundle:23:in `<main>'
/home/kakra/.rvm/gems/ruby-2.2.5/bin/ruby_executable_hooks:15:in `eval'
/home/kakra/.rvm/gems/ruby-2.2.5/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => searchkick:reindex:all
(See full trace by running task with --trace)

[weilandia]

I disabled shard allocation and forgot to re-enable it (during an upgrade), and I was getting the same errors as above. Ya'll might be running into the same problem.

How to resolve:

curl -XPUT 'localhost:9200/_cluster/settings' -d '{
    "transient" : {
        "cluster.routing.allocation.enable" : "all"
    }
}'

[kakra]

I'm still seeing this error. And one clue I currently have: While the application is running through passenger (even when idle), it's more likely to throw this error. When stopping passenger, it's more likely to not happen. This is fine in development. But the whole purpose of deployment is to not stop the application while migration the database (and this is where I also run reindexing when index-related changes were introduced).

[dyerc]

I received a similar Elasticsearch::Transport::Transport::Errors::BadGateway error this morning. The re-indexing crashed with this error, when I re-ran searchkick:reindex:all it worked fine but seemed to end up with multiple indexes and duplicate results.

Running the following manually for each model seemed to sort the duplicate results issue

Model.searchkick_index.delete
Model.reindex

I'm also using passenger and searchkick version 2.3.0

[motsat]

I also got the same error in 2.5.0.
It is time to take a time-consuming case (for example, specifying a huge number of lines of synonym file) when creating index with PUT request.

In elasticsearch-ruby, host_unreachable_exceptions occurred and retry processing was done.

https://github.com/elastic/elasticsearch-ruby/blob/master/elasticsearch-transport/lib/elasticsearch/transport/transport/base.rb#L293

rescue * host_unreachable_exceptions => e
        logger.error "[# {e.class}] # {e.message} # {connection.host.inspect}" if logger

        connection.dead!

        if @options [: reload_on_failure] and tries <connections.all.size
          logger.warn "[# {e.class}] Reloading connections (attempt # {tries} of # {connections.all.size})" if logger
          reload_connections! and retry
        end

I avoided it by extending the timeout.
SearchkickTimeout = 120

[kakra]

Oh wow @motsat ... Good find!

I wonder why I never saw the log message which should've been logged according to this code. Anyway, the first few tries with adjusted timeout were successful now.

I guess Searchkick should maybe internally use different timeouts for index creation than other stuff. @ankane What do you think?