Elasticsearch-php: Guzzle HTTP 409 Conflict

[r0mdau]

Hello,

I need approximatively 10 scripts to run separately and insert/delete/update documents in 1 elasticsearch server.

Sometimes 2 scripts or more are updating the same document in the same time.
Is there an other way instead of using try/catch in php ?

Thanks in advance for your support.

This is the error log :

PHP Fatal error:  Uncaught exception 'Guzzle\Http\Exception\ClientErrorResponseException' with message 'Client error response
[status code] 409
[reason phrase] Conflict
[url] http://192.168.1.200:9200/web/site/p0.storage.canalblog.com/_update' in /home/romain/Workspace/websearch/vendor/guzzle/http/Guzzle/Http/Exception/BadResponseException.php:43
Stack trace:
#0 /home/romain/Workspace/websearch/vendor/guzzle/http/Guzzle/Http/Message/Request.php(145): Guzzle\Http\Exception\BadResponseException::factory(Object(Guzzle\Http\Message\EntityEnclosingRequest), Object(Guzzle\Http\Message\Response))
#1 [internal function]: Guzzle\Http\Message\Request::onRequestError(Object(Guzzle\Common\Event), 'request.error', Object(Symfony\Component\EventDispatcher\EventDispatcher))
#2 /home/romain/Workspace/websearch/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/EventDispatcher.php(164): call_user_func(Array, Object(Guzzle\Common\Event), 'request.error', Object(Symfony\Component\EventDispatcher\EventDispatcher))
#3 /home/romain/W in /home/romain/Workspace/websearch/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/GuzzleConnection.php on line 262

[polyfractal]

The reason you are getting these exceptions is because Elasticsearch recognizes that the update would have modified a document that changed since you issued the update, potentially losing data. It rejects this update for safety reasons and lets your application decide what to do next.

Soo...it depends on the nature of the updates. If these are idempotent updates (e.g. it doesn't matter what order they are applied, such as incrementing a counter), you need to use the versioning support in Elasticsearch.

You can read more about versioning here: "Optimistic Concurrency Control" and here: "Partial Updates" (in particular "Dealing with Conflicts")

The short answer is that if the updates are idempotent, you should to tell Elasticsearch to retry those updates server-side in case of conflict:

$params['index'] = 'my_index';
$params['type']  = 'my_type';
$params['id']    = 'my_id';

// This needs to be set >= number of threads, since it might compete against all of them
// in worst case
$params['retry_on_conflict'] = 10;

$params['body']  = [
  'script' => 'ctx._source.views+=1',
  'upsert' => [
    'views' => 0
  ]
];
$ret = $client->update($params);

If your update is not idempotent (e.g. changing a name from 'fred' to 'tony'), that's where you will have to let your application catch the exception and deal with it manually, since there is no good automated way to deal with it. Elasticsearch won't know if 'fred' or 'tony' is the correct value, so the only automated solution is to clobber the old value.

[r0mdau]

Thank you for your answer.

The updates are idempotent.
I will try with the parameter retry_on_conflict and I will give you a feedback.

[r0mdau]

retry_on_conflict is a good option to solve my problem.

But take care, we should determine an acceptance level, maybe we have to use $params['retry_on_conflict'] >= number of threads * 2; to be quiet.