Elasticsearch-rails: erroneous DocumentMissingException

[githorse]

I've been using Elasticsearch for months but just started getting hit by constant errors in the index; I'm not sure what changed. One place I'm getting them is logging out of the app, when trying to write to the current_user's remember_digest attribute. My logout code is vanilla rails, basically directly from the latest Hartl RoR tutorial:

# sessions_controller.rb
  def destroy
    sign_out if signed_in?
    redirect_to root_path
  end

  def sign_out
    forget current_user
    session.delete(:user_id)
    @current_user = nil
  end

  def forget user
    user.forget
    cookies.delete(:user_id)
    cookies.delete(:remember_token)
  end

# user.rb
  def forget
    update_attribute :remember_digest, nil   # error thrown here
  end

The update_attribute line throws this error:

[404] {"error":"DocumentMissingException[[users][4] [user][22]: document missing]","status":404}

Apart from this, sign out works fine. In particular, the current_user (22) is still in the database when the error occurs (and remains so); the record is not being destroyed. Running User.import force: true resolves the problem (until the next logout).

My user model is set up along the lines of the feature extraction pattern from the docs:

module Searchable
  extend ActiveSupport::Concern

  included do
    include Elasticsearch::Model
    include Elasticsearch::Model::Callbacks

    class << self

     def search(*args)
         # Detail omitted. My issues don't seem to be 
         # a problem with the actual search, but with the callbacks above. 
     end
  end
end

Actually searching with Elasticsearch works fine, but I get errors all the time in the index. Sometimes they seem to occur when creating or deleting a model; however, in this case, no User is being created or deleted, so I'm really not sure what might cause this.

[karmi]

Hmm, my instinct tells me this might be about needing to refresh the index, but the error indicates you're usign the "Get Document" API, which is 100% realtime... Can you enable the log and see what is actually being sent between ES and Ruby?

[githorse]

@karmi -- here's an example (just happened). This is all I get in my Rails logs:

Started GET "/sign_out" for ::1 at 2016-05-10 15:18:17 -0400
Processing by SessionsController#destroy as HTML
  User Load (0.2ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = ? LIMIT 1  [["id", 2]]
   (0.1ms)  begin transaction
   (0.1ms)  commit transaction
ElasticsearchPolice: intercepted unknown Elasticsearch error on /sign_out:
  [404] {"error":{"root_cause":[{"type":"document_missing_exception","reason":"[user][2]: document missing","shard":"2","index":"users"}],"type":"document_missing_exception","reason":"[user][2]: document missing","shard":"2","index":"users"},"status":404} ([404] {"error":{"root_cause":[{"type":"document_missing_exception","reason":"[user][2]: document missing","shard":"2","index":"users"}],"type":"document_missing_exception","reason":"[user][2]: document missing","shard":"2","index":"users"},"status":404}):

ElasticsearchPolice: ignoring this error and continuing current request.

Completed 200 OK in 86ms (ActiveRecord: 0.4ms)

ElasticsearchPolice is my (woefully ineffective) error-handling module, which for now just logs the error.

The Elasticsearch process at localhost:9200 showed _no_ activity here in the default INFO-level log. I just enabled DEBUG level logs so will have to wait for the next error to see what they say.

[deepakm-webonise]

Any updates on this issue I am also facing the same issue

Elasticsearch::Transport::Transport::Errors::NotFound (
  [404] {
    "error":{
      "root_cause":[
        {
          "type":"document_missing_exception",
          "reason":"[user][1200000]: document missing",
          "shard":"0",
          "index":"development_users"
        }
      ],
      "type":"document_missing_exception",
      "reason":"[user][1200000]: document missing",
      "shard":"0",
      "index":"development_users"
    },
    "status":404
  }):
 app/controllers/users/sessions_controller.rb:26:in `destroy'

app/controllers/users/sessions_controller.rb

24. def destroy
25.   redirect_path = after_sign_out_path_for(resource_name)
26.   signed_out = Devise.sign_out_all_scopes ? sign_out : sign_out(
27.     resource_name)
28.   respond_to do |format|
29.     format.all { head :no_content }
30.     format.any(*navigational_formats) { redirect_to redirect_path }
31.   end
32. end

[anlek]

Any news on this? I'm seeing a similar issue in my code.

[reneweteling]

Hey guys,

Im experiencing the same issue, and i have a case to replicate the problem.

1. create a model (User) and link it up to elastic as shown in the docs
2. create the index User.__elasticsearch__.create_index!(force: true)
3. modify the record, User.update(name: 'Joe')

Error

{
  "error": {
    "root_cause": [
      {
        "type": "document_missing_exception",
        "reason": "[user][2080730]: document missing",
        "index_uuid": "2YbpMSZlTey2EF1s0yIOWQ",
        "shard": "3",
        "index": "users"
      }
    ],
    "type": "document_missing_exception",
    "reason": "[user][2080730]: document missing",
    "index_uuid": "2YbpMSZlTey2EF1s0yIOWQ",
    "shard": "3",
    "index": "users"
  },
  "status": 404
}

But if you index the model

User.import

and then try to update it, its ok, so the gem somehow assumes the model lives in Elastic. The operation isnt idempotent

Any suggestions would be much appreciated!

Thanks!!

[TopdiR]

I have exactly same issue as @reneweteling ! Rails and gems are up to date. Please somebody explain this behavior. Thanks!

[ddavisgraphics]

Same Issue on update and new items created after the import was made.

Elasticsearch::Transport::Transport::Errors::NotFound: [404] {"error":{"root_cause":[{"type":"document_missing_exception","reason":"[employee][201]: document missing","index_uuid":"FQ8V9nniS_2Nmxo-5DGiOw","shard":"3","index":"users_development"}],"type":"document_missing_exception","reason":"[employee][201]: document missing","index_uuid":"FQ8V9nniS_2Nmxo-5DGiOw","shard":"3","index":"users_development"},"status":404}
from /usr/local/bundle/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:202:in `__raise_transport_error'

[ddavisgraphics]

To fix the issue that I had, I destroyed the indexes and reimported each model separately from IRB. This helped and has removed all issues. Not sure what the problem was or why it appeared.

[multiplegeorges]

I was getting the same error and it's a callback order issue.

I have Shrine set up for file attachments on my User model and if I try and assign the Shrine file inside the create operation, the model saves/commits then the Shrine callback updates the user attachment data before ES has received the POST to create the document in the index.

The solution is to split up the User creation operation.

This doesn't work:

User.create!(
  email: Faker::Internet.email,
  profile: File.new('path/to/file' # This line changes the callback ordering and breaks the ES document indexing
)

ES tries to update the document for the profile attributes before the actual User document exists.

This works:

user = User.create!(email: Faker::Internet.email)
# ES gets a new document in the callbacks here
user.profile = File.new('path/to/file' 
user.save!
# And now there's a document in ES for the update call

YMMV but this solved it for me.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.