Searchkick: How to remove Records in search index do not exist in database

[vishalzambre]

First
Looking to remove records in search index do not exist in database

Describe the bug
Sometimes on destroy failed to trigger a callback or if we use delete_all or delete it failed to remove from Elasticsearch, problem is when we use count, it shows wrong count than exist in DB

To reproduce
Create record and use delete_all

require "bundler/inline"

gemfile do
  source "https://rubygems.org"

  gem "activerecord", require: "active_record"
  gem "activejob", require: "active_job"
  gem "sqlite3"
  gem "searchkick", git: "https://github.com/ankane/searchkick.git"
end

require "minitest/autorun"

puts "Searchkick version: #{Searchkick::VERSION}"
puts "Elasticsearch version: #{Searchkick.server_version}"

ActiveRecord::Base.establish_connection adapter: "sqlite3", database: ":memory:"
ActiveJob::Base.queue_adapter = :inline

ActiveRecord::Migration.create_table :products do |t|
  t.string :name
end

class Product < ActiveRecord::Base
  searchkick
end

class ProductTest < Minitest::Test
  def setup
    Product.reindex
    Product.create!(name: "Test")
    Product.create!(name: "Test 2")
    Product.search_index.refresh
  end

  def test_count_before_delete
    assert_equal Product.search("test", fields: [:name]).total_count, 2
  end

  def test_count_after_delete
    Product.all.delete_all
    assert_equal Product.search("test", fields: [:name]).total_count, 0
  end

  def test_count_reindex_after_delete
    Product.all.delete_all
    Product.reindex
    assert_equal Product.search("test", fields: [:name]).total_count, 0
  end
end

* Output *

✗ ruby test.rb
Searchkick version: 4.4.1
Elasticsearch version: 6.8.1
-- create_table(:products)
   -> 0.0040s
Run options: --seed 35012

# Running:

warning: 299 Elasticsearch-6.8.1-1fad4e1 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template", 299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in create index requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions."
Fwarning: 299 Elasticsearch-6.8.1-1fad4e1 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template", 299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in create index requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions."
warning: 299 Elasticsearch-6.8.1-1fad4e1 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template", 299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in create index requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions."
.warning: 299 Elasticsearch-6.8.1-1fad4e1 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template", 299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in create index requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions."
.

Finished in 1.228398s, 2.4422 runs/s, 2.4422 assertions/s.

  1) Failure:
ProductTest#test_count_after_delete [test.rb:42]:
Expected: 2
  Actual: 0

3 runs, 3 assertions, 1 failures, 0 errors, 0 skips

Additional context
Add any other context.

[ankane]

Hey @vishalzambre, check out https://stackoverflow.com/questions/48631505/searchkick-remove-documents-when-unable-to-find-original-record.

[vishalzambre]

@ankane Thanks for notice it.

I'm doing same way right now, but problem is, It needs to do a manual job, I'm looking for something proper hook or method or job which can remove results when it founds like on with_hits method or results method.

I saw in results method it prints missing ids, and was just thinking some patch there?

[ankane]

Added a missing_ids method on master you can use to automate cleanup.

[ankane]

Changed to missing_hits so it's useful for multi-index searches.

[ankane]

Thinking about it more, the hits don't seem super useful one their own, so changed to missing_records, which returns the id and model. That should make it easy to use the approach in the link.

[vishalzambre]

@ankane Thank you so much, I'll refer master