Elasticsearch-py: reindex method in class Elasticsearch timeout

[kuzhao]

When I'm using this method with body='{"source":{"index": ...}, "dest":{"index": ...}}', the same with REST api, I get a timeout error from urllib3 as follows:

File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 1278, in reindex
    params=params, body=body)
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 102, in perform_request
    raise ConnectionTimeout('TIMEOUT', str(e), e)
ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host=..., port=...): Read timed out. (read timeout=10))

However the reindex request did get passed to elasticsearch host and processed. It seems that the arguments of this method don't work as it's documented in either readthedocs or method description.

[HonzaKral]

This error means that you didn't get a response from elasticsearch in the allotted time (10 seconds by default). Which makes sense since the reindexing is a big operation.

To avoid this you need to set your timeout higher - either for the whole client (by passing timeout=N when creating the client) or for this individual request using the request_timeout=N parameter to the reindex call.

Hope this helps

[kuzhao]

@HonzaKral Thanks for your reply. I'm totally aware of how costly reindex's gonna be. So I added wait_for_completion=False but that didn't help out.
Meanwhile there's no 'request_timeout' but 'timeout' arg for reindex, according to the documentation. Furthermore, I found the method seems to pass 'param' arg to transport call in codes, and that looks different from documentation too.
Perhaps I've got it all wrong as I'm not much of a coder. If so, just blame me.

[HonzaKral]

request_timeout is available at all API calls - http://elasticsearch-py.readthedocs.io/en/master/api.html#timeout

The params arg is an internal way how we pass parameters that go into the query string around. In code all the parameters specified by the decorator are serialized and passed into the main API method inside the params dict.

[kuzhao]

So the method should be used in this way: body (src_idx, dst_idx) + longer timeout. And the doc section for reindex in class Elasticsearch should be modified to get aligned with global available request_timeout arg.
Still, the issue of wait_for_completion persists.
BTW I do hope this function could replace that in helper as the primitive way has the most efficiency.

[HonzaKral]

Thank you, I am closing this issue since the python side has been resolved. If the issue with wait_for_completion being ignored persists please open an issue with elasticsearch itself since there is nothing we can do.

Thank you!

as for the reindex helper - that will most likely stay to help move between clusters and also to provide more flexibility. But the reindex API is definitely the preferred way to do things and we will reflect that in the documentation.

[kuzhao]

Well noted and I appreciate your effort on this issue @HonzaKral
Have a nice day~