Elasticsearch-py: Inconsistent response to elasticSearch queries

[ravir4]

Hello, I am running very simple searches against ElasticSearch (from Ipython notebook) and I get mixed results.
Any help is greatly appreciated.

Here are some code snippets that I am using

This one works:

from elasticsearch import Elasticsearch
es = Elasticsearch('https:/........)

res = es.search(index = '', doc_type = "Devices')
print res['hits']['total']

---

This one doesn't work (I tried limiting the number of docs with matches thinking it maybe too many docs but same error)

---

from elasticsearch import Elasticsearch
es = Elasticsearch('https:/........)

res = es.search(index = '',body={"query": {"match_all": {}}})

When I run the above I get a Transport Error 400:

/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.pyc in _wrapped(_args, *_kwargs)
67 if p in kwargs:
68 params[p] = kwargs.pop(p)
---> 69 return func(_args, params=params, *_kwargs)
70 return _wrapped
71 return _wrapper

/usr/local/lib/python2.7/dist-packages/elasticsearch/client/init.pyc in search(self, index, doc_type, body, params)
525 index = '_all'
526 _, data = self.transport.perform_request('GET', _make_path(index,
--> 527 doc_type, '_search'), params=params, body=body)
528 return data
529

/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.pyc in perform_request(self, method, url, params, body)
305
306 try:
--> 307 status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
308
309 except TransportError as e:

/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.pyc in perform_request(self, method, url, params, body, timeout, ignore)
91 if not (200 <= response.status < 300) and response.status not in ignore:
92 self.log_request_fail(method, url, body, duration, response.status)
---> 93 self._raise_error(response.status, raw_data)
94
95 self.log_request_success(method, full_url, url, body, response.status,

/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.pyc in _raise_error(self, status_code, raw_data)
103 pass
104
--> 105 raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
106
107

RequestError: TransportError(400, u'')

[HonzaKral]

This is very weird error, could you please turn on logging and see what are the actuall messages being sent to elasticsearch?

import logging
logging.basicConfig(level=logging.DEBUG)

then repeat your experiment and paste the logging output.

Thanks!

[ravir4]

HonzaKral, Thank you for responding. Unfortunately I don't have access to the product Elastic Search system to make changes and enable logging. Let me see if I can reach out to the folks supporting it but if it needs a restart of the server it may be tougher to do I think. I will check on it Monday. Thanks again.

[HonzaKral]

The logging is on the python side, not the server

[ravir4]

Thank you. My bad, I didn't pay close attention to your request. Thanks again in advance...
Also forgot to add in my OP - this is what I am using

es = Elasticsearch(['https:/MYUSER: MYPASSWORD:@ESHOST'],ca_certs=certifi.where(),verify_certs=True,use_ssl=True,)

I do the following Imports:

import certifi
import logging
logging.basicConfig(level=logging.DEBUG)
from elasticsearch import Elasticsearch

Ok I added it and here is what I see when it fails:

Are you expecting more info in the debug output than this, it seemed like it didn't add much to the original. If so, I will go back and check.

/usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py:100: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning WARNING:elasticsearch:GET /MYINDEX/_search [status:400 request:0.111s]

--------------------------------------------------------------------------- RequestError Traceback (most recent call last) in () 1 logging.basicConfig(level=logging.DEBUG) ----> 2 res = es.search(index = ‘MYINDEX',body={"query": {"match_all": {}}}) /usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.pyc in _wrapped(_args, *_kwargs) 67 if p in kwargs: 68 params[p] = kwargs.pop(p) ---> 69 return func(_args, params=params, *_kwargs) 70 return _wrapped 71 return _wrapper /usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.pyc in search(self, index, doc_type, body, params) 525 index = '_all' 526 _, data = self.transport.perform_request('GET', _make_path(index, --> 527 doc_type, '_search'), params=params, body=body) 528 return data 529 /usr/local/lib/python2.7/dist-packages/elasticsearch/transport.pyc in perform_request(self, method, url, params, body) 305 306 try: --> 307 status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) 308 309 except TransportError as e: /usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.pyc in perform_request(self, method, url, params, body, timeout, ignore) 91 if not (200 <= response.status < 300) and response.status not in ignore: 92 self.log_request_fail(method, url, body, duration, response.status) ---> 93 self._raise_error(response.status, raw_data) 94 95 self.log_request_success(method, full_url, url, body, response.status, /usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.pyc in _raise_error(self, status_code, raw_data) 103 pass 104 --> 105 raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) 106 107 RequestError: TransportError(400, u'')

---

When it DOES work , here is all I see with the expected good JSON output

/usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py:100: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning

[HonzaKral]

You should also see the logging output, especially when it works, which should tell use what kind of requests happen.

Regardless, could you please try catching the exception and looking at it's .info:

try:
    es.search()
except TransportError as e:
    print(e.info)

Hopefully that should tell us something.
Thanks!

[ravir4]

Ok thanks. For some reason I couldn't get the debug data on my first VM (VM1) working. I was, in parallel, setting up a second VM (VM2) anyway to make sure there wasn't something wrong with VM1.

I was able to see python debug output on VM2 but TransportError became ConnectionError in this one (same code, same targets), not sure why. But I have the exact same scenario on VM2 as well.

Thanks again for your assistance

This code works and gets me the result I am looking for....

res = es.search(index = ‘MYINDEX', doc_type = ’TYPE')
print res['hits']['total']

gives me the same # I get from Postman or Sense

---

This one doesn't work

Code:

try:
res = es.search(index = ‘MYINDEX’,doc_type = 'TYPE',body={"query": {"match_all": {}}})
except requests.ConnectionError as e:
print(e.info)

Output:

WARNING:elasticsearch:GET http://ESHOST:443/MYINDEX/TYPE/_search [status:N/A request:0.342s]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 78, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, _kw)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 528, in urlopen
*_response_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 284, in from_httplib
*response_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 104, in __init__
self._body = self.read(decode_content=decode_content)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 182, in read
data = self._fp.read()
File "/usr/lib/python2.7/httplib.py", line 592, in read
s = self.fp.read()
File "/usr/lib/python2.7/socket.py", line 355, in read
data = self._sock.recv(rbufsize)
File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 200, in recv
return self.recv(_args, _kwargs)
File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 188, in recv
data = self.connection.recv(_args, *_kwargs)
ZeroReturnError
WARNING:elasticsearch:GET http://ESHOST:443/MYINDEX/TYPE/_search [status:N/A request:0.331s]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 78, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, *_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 528, in urlopen
*_response_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 284, in from_httplib
*response_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 104, in __init__
self._body = self.read(decode_content=decode_content)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 182, in read
data = self._fp.read()
File "/usr/lib/python2.7/httplib.py", line 592, in read
s = self.fp.read()
File "/usr/lib/python2.7/socket.py", line 355, in read
data = self._sock.recv(rbufsize)
File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 200, in recv
return self.recv(_args, _kwargs)
File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 188, in recv
data = self.connection.recv(_args, *_kwargs)
ZeroReturnError
WARNING:elasticsearch:GET http://ESHOST:443/MYINDEX/TYPE/_search [status:N/A request:0.345s]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 78, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, *_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 528, in urlopen
*_response_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 284, in from_httplib
*response_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 104, in __init__
self._body = self.read(decode_content=decode_content)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 182, in read
data = self._fp.read()
File "/usr/lib/python2.7/httplib.py", line 592, in read
s = self.fp.read()
File "/usr/lib/python2.7/socket.py", line 355, in read
data = self._sock.recv(rbufsize)
File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 188, in recv
data = self.connection.recv(_args, _kwargs)
ZeroReturnError
WARNING:elasticsearch:GET http://ESHOST:443/MYINDEX/TYPE/_search [status:N/A request:0.243s]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 78, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, *_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 528, in urlopen
*_response_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 284, in from_httplib
*response_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 104, in __init__
self._body = self.read(decode_content=decode_content)
File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 182, in read
data = self._fp.read()
File "/usr/lib/python2.7/httplib.py", line 592, in read
s = self.fp.read()
File "/usr/lib/python2.7/socket.py", line 355, in read
data = self._sock.recv(rbufsize)
File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 188, in recv
data = self.connection.recv(_args, *_kwargs)
ZeroReturnError

---

ConnectionError Traceback (most recent call last)
in ()
1 try:
----> 2 res = es.search(index = ‘MYINDEX’,doc_type = 'TYPE',body={"query": {"match_all": {}}})
3 except requests.ConnectionError as e:
4 print(e.info)

/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.pyc in _wrapped(_args, *_kwargs)
67 if p in kwargs:
68 params[p] = kwargs.pop(p)
---> 69 return func(_args, params=params, *_kwargs)
70 return _wrapped
71 return _wrapper

/usr/local/lib/python2.7/dist-packages/elasticsearch/client/init.pyc in search(self, index, doc_type, body, params)
525 index = '_all'
526 _, data = self.transport.perform_request('GET', _make_path(index,
--> 527 doc_type, '_search'), params=params, body=body)
528 return data
529

/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.pyc in perform_request(self, method, url, params, body)
305
306 try:
--> 307 status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
308
309 except TransportError as e:

/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.pyc in perform_request(self, method, url, params, body, timeout, ignore)
87 except Exception as e:
88 self.log_request_fail(method, full_url, body, time.time() - start, exception=e)
---> 89 raise ConnectionError('N/A', str(e), e)
90
91 if not (200 <= response.status < 300) and response.status not in ignore:

ConnectionError: ConnectionError() caused by: ZeroReturnError()

[HonzaKral]

Could it be that you are connecting through a proxy? It looks like something on your network is forbidding a GET request with a body.

Try to initialize your client with send_get_body_as='POST', it should help

[ravir4]

AWESOME THANK YOU !!!
Yes that was it, It is working now. Yes I am connecting through NGNIX. My bad I should have mentioned that earlier, didn't realize the impact it had. I am very new to ES and learning this real quick.

Thank you again.

[HonzaKral]

great, closing the ticket.

[ngobw]

Just wanted to say what @HonzaKral suggested is also the solution to my woes while developing on Google Endpoints. His suggestion has also been described in further details in under the Environment considerations :

In some environments (notably on Google App Engine) your http requests might be restricted so that GET requests won’t accept body. In that case use the send_get_body_as parameter of Transport to send all bodies via post: