Elasticsearch-rails: How Could I Remove Transport::Errors::Conflict

[xuxiangyang]

Hi, I have a problem,
I use elasticsearch-model like this

include Elasticsearch::Model
include Elasticsearch::Model::Callbacks

When update concurrency in production model I got the error

Elasticsearch::Transport::Transport::Errors::Conflict

like this

[409] {"error":{"root_cause":[{"type":"version_conflict_engine_exception","reason":"[swipe_configs][f55a8333-ee71-4473-b575-7d63b89de8a0]: version conflict, current [2], provided [1]","shard":"1","index":"inchat_release"}],"type":"version_conflict_engine_exception","reason":"[swipe_configs][f55a8333-ee71-4473-b575-7d63b89de8a0]: version conflict, current [2], provided [1]","shard":"1","index":"inchat_release"},"status":409}

How could I avoid this error?

thank you

[karmi]

So, the error indicates that you're passing a version number 1, which doesn't match the version in the index. Have you read through https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning ?

[xuxiangyang]

yes, I know es. I just use save, update method. Has not passed the version in option.Does this gem pass it automatly?

[karmi]

No, the gem should not pass any version, so it's not clear where it's coming from. Can you check out the examples for ActiveModel in this repository, or the full example applications, and see whether they work for you? Then you can check your code.

[olivierbuffon]

I got the exact same problem. I'm doing a simple create (I tested save also, same problem).
I'm using Elasticsearch 5.2 with elasticsearch-persistence (0.1.9).

Here is my class:

class Transaction
  include Elasticsearch::Persistence::Model

  #
  #   Constants
  #

  CATEGORIES = %w(retail fnb).freeze

  #
  #   Attributes
  #

  attribute :club_id, Integer
  attribute :user_id, Integer
  attribute :id_ref, String
  attribute :amount, Float
  attribute :category, String
  attribute :occurred_at, Time, mapping: { type: 'date' }

  #
  #   Validations
  #

  validates :club_id, presence: true
  validates :user_id, presence: true
  validates :id_ref, presence: true
  validates :amount, presence: true
  validates :occurred_at, presence: true
  validates :category, presence: true, inclusion: { in: CATEGORIES }
  ...
end

and here is the operation that generate this error:

def persist_new_transactions(club_id, user_id, customer_id_ref, pos_api_client, category)
  transactions(customer_id_ref) do |transaction|
    Transaction.create(transaction_params(club_id, user_id, transaction, category))
  end
end

...

def transaction_params(club_id, user_id, transaction, category)
  {
    club_id: club_id,
    user_id: user_id,
    id_ref: transaction[:reference_ref],
    amount: transaction[:amount],
    occurred_at: transaction[:occurred_at],
    category: category
  }
end

Finally here is the error I get:

Elasticsearch::Transport::Transport::Errors::Conflict: [409] {"took":3,"timed_out":false,"total":1,"deleted":0,"batches":1,"version_conflicts":1,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[{"index":"house_account-transactions","type":"transaction","id":"AVpDdNl4xOVMbeyxCXXm","cause":{"type":"version_conflict_engine_exception","reason":"[transaction][AVpDdNl4xOVMbeyxCXXm]: version conflict, current version [2] is different than the one provided [1]","index_uuid":"JrGCiF_9QaO0XpTdYeqM8Q","shard":"1","index":"house_account-transactions"},"status":409}]}

The worst is this error is happening randomly! I'm going crazy with this, so any help would be very appreciated ^^

Any idea?

[olivierbuffon]

Update:
After hours of investigation it seems the error related to the usage of delete_by_query.
I still don't know why but here is the problematic line:

Elasticsearch::Persistence.client.perform_request 'POST', "#{index_name}/_delete_by_query", {}, builder.query

As I said in my previous post, this error appears randomly which makes it very difficult to fix. So if anyone have an idea...
Thanks for the help.

[karmi]

Hi, I'm not sure of the root cause here yet, _but_, the delete_by_query API has been recently fixed in the elasticsearch-api gem "master" branch (see https://github.com/elastic/elasticsearch-ruby/issues/387#issuecomment-274050959). So, what version of ES and the Ruby/Rails libraries are you running?

[karmi]

Just a pointer, when looking at the Elasticsearch "Delete By Query" API documentation:

_delete_by_query gets a snapshot of the index when it starts and deletes what it finds using internal versioning. That means that you’ll get a version conflict if the document changes between the time when the snapshot was taken and when the delete request is processed. When the versions match the document is deleted.

That corresponds with what you're seeing. What is probably hapenning in practical terms is:

1. You issue a delete_by_query API call. That creates a snapshot if the index in question, and starts processing the documents.
2. Meanwhile, your application updates one of those documents, incrementing that internal version.
3. The delete_by_query operation for this updated document fails.

[olivierbuffon]

Thanks @karmi ! I just found this information in the doc this morning too ^^.
I'm currently working on it but there's still one thing I don't understand. How an object "maked as deleted" could have been updated since the only operation I do here is create new objects after this delete_by_query operation.
To me, the only scenario that could explain this case, is that one of my newly created document get an already used _id value. Does is makes sense? Maybe I'm missing something.
I'm still investing on how this could happen but nothing new for now. Any idea?

EDIT:
FYI here is the gems and versions that I'm currently using:

...
Using elasticsearch-api 5.0.3
Using elasticsearch-extensions 0.0.23 from https://github.com/elastic/elasticsearch-ruby (at master@b1cc7b0)
Using elasticsearch-transport 5.0.3
Using elasticsearch 5.0.3
Using elasticsearch-model 0.1.9
Using elasticsearch-persistence 0.1.9
...

[karmi]

How an object "maked as deleted" could have been updated (...)

I'm not 100% sure, but the subtlety in the API description is that the operation "gets a snapshot of the index", hence, it "freezes" that index in time, including the versions, and starts processing the documents. (This is exactly the same thing as with the "Scroll" API.) But the index is kept "live", of course, and can happily receive changes, and any change increments the internal version -- in your example, the error says current version [2] is different than the one provided [1].

So, I don't think I was correct in the point "3" of my description, I don't know 100% what happens at this point, but if the index is connected eg. to a database, an ID can indeed be "reused"...

To debug it further, can you get from the logs if the "conflict" error happens for the "Update Document" operation or the "Delete By Query" operation? My guess is the former, but wanna be sure. In that case, for further debugging, I'd probably try to match the document ID for that operation with documents matching the "Delete By Query" query. I guess it's probably challening to do it on a live system in flux, but you can eg. issue a regular "Search" request before you issue the "Delete By Query" one, and dump the matching IDs somewhere, so you can later on match them...

Sorry for not having a straightforward explanation here... One last thing to double-check is whether the "Update Document" operation (from your callbacks) does or doesn't include the version (it _shouldn't_ by default).

[olivierbuffon]

Interesting, I' continue to investigate. FYI I tried two different way to manage the _id value.
First, I didn't specified any _id for the object creation, so I let ES set his own _id value.
After experiencing this 409 conflicts error I tried a different approach by manually specifying the _id value by doing this: _id: SecureRandom.uuid. Unfortunately, this results to the same issue.
I was very surprised this error happened in my second scenario because the chances SecureRandom.uuid regenerates the exact same id are extremely low.
I don't know if it helps but I found important that you know this information before going further.

[karmi]

Thanks for the detailed info. If the IDs are auto-generated, then it's indeed pretty weird. But the "general" source of the error must come from some clash between the "Delete By Query" operation and a regular document update (presumably triggered by a callback).

I know it's parallel to the issue, but I'll ask anyway: any pressing reason to use the "Delete By Query" operation at all? Normally, I would think twice before using it, but I understand there are scenarios/requirements where it's the right fit.

[olivierbuffon]

@karmi that's very valid question ^^! I would say that I need to delete some documents before create some new ones. I could use another way to do this deletion, if you have any recommandation.
But to be honest "Delete By Query" was the best way I found to do what I needed to do because I don't want to use a "one by one" delete operation since the amount of documents I need to delete could be pretty big :(

[karmi]

The "Delete By Query" operation is definitely easier to perform, than to delete documents one-by-one (even though, technically, ES performs the same behind the scenes).

I guess the main question is then what makes these documents "invalidated", to be deleted. Normally, we advise to create some sort of nomenclature based on indices, eg. for time-based data, and drop whole indices, eg. "delete everything older than 6 months". But I think in your case the requirements are different than this scenario.

[olivierbuffon]

I understand your point here, unfortunately I don't have a lot of choices. What I'm currently doing is pulling informations from an external API and persist those in ES. The main problem here, is this API endpoint could return a response (here an array of transactions) that could change over the time, e.g a transaction deleted on their side won't be returned by the API endpoint anymore. So as you can imagine, if I want my data to be up to date I alway need to fetch the entire liste of transactions, delete already persisted transactions and re-persist those in ES.
I'm thinking about a different approach, but this API is not very flexible and doesn't give me a lot of options :(
Anyway, I'll try do find a walkaround but I would be very interested to ear from you if something new raise on this issue.
Thanks for your help @karmi, very appreciated! :)

[karmi]

So, we never really clarified if the "409 conflict" error comes from the document update operation, or from the _delete_by_query operation -- I'd still be interested in that.

I'm afraid I still don't have a straightforward explanation or a fix, _but_, when thinking about the workflow you describe, there could be a way how to work around the "Delete By Query" approach, which could, arguably, be more transparent and scalable, especially for large workflows. The trouble I would be afraid of, when using "Delete By Query", is that the load on the cluster is less predictable and, _importantly_, less visible.

Another way would be to basically carry over these "housekeeping" operations into the application code: the documents, which are no longer valid (in terms of the external source), would be marked as invalid by property, something like invalid: true.

All queries would then use a must_not clause to filter these out from results (an index alias could be used for that, if that would make sense).

Then, periodically, a background process would query for these documents, getting their IDs. For each batch of IDs, a job (Sidekiq, Resque, ...) would be kicked off, which would delete them via the "Bulk delete operation.

Again, it's not necessarily something 100% applicable in your case, it's just that it might not only solve the problem at hand, but it might make the whole system more "transparent".

[olivierbuffon]

@karmi thanks for those suggestions. Since I'm only able to reproduce this case on my production environment (related to the amount of data I have in production), you can imagine it's very complicated to investigate through the logs. But what I did is extract the code I wrote to only run the _delete_by_query method without any other actions. In this case I had the exact same problem. I don't know exactly the behaviour of the code behind the _delete_by_query and what actions are triggered by this method, but those 409 conflits errors seems only related to this method usage.

After couple of tries here is "monkey patched" that worked in my code:

def delete_by_query(index_name, query)
  tries ||= 3
  Elasticsearch::Persistence.client.delete_by_query(index: index_name, body: query)
rescue Elasticsearch::Transport::Transport::Errors::Conflict => e
  raise e if (tries -= 1).zero?
  log_error('version conflict')
  sleep 30
  retry
end

This is clearly not a permanent solution but that the only way I found to make it work without redesign the entire code. I don't know if that helps but that's the point where I stopped for now.

[estolfo]

Closing as this does not require client changes.