Elasticsearch-py: Proxy settings

[svamet]

hi there,
i dont know if i am right here, but i have not found anything according to my problem in the web.
i have to use elasticsearch in python from behind a proxy server. how can i pass down the proxy setting to elasticsearch.
i tried something like that without success.

es = Elasticsearch([es_url], _proxy = "http://proxyurl:port", _proxy_headers = { 'basic_auth': 'USERNAME:PASSWORD' })
res = es.search(index=index, body=request, search_type="count")

any help would be very nice.
thanks

[HonzaKral]

Hi,

unfortunately our connection classes don't support proxy settings out of the box. The solution is to create your own and set it directly via urllib3 or requests.

Another, easier, option would be to just set the env variables (mentioned in http://www.python-requests.org/en/latest/user/advanced/#proxies) when using requests.

[svamet]

thanks for your fast answer.
sry this maybe goes too far, but im new to python and elasticsearch and any help would be nice..

how can i set my connection class via requests?
something like this wont work:

import requests
from elasticsearch import Elasticsearch

proxies = {'https': 'http://user:[email protected]:port'}
es = Elasticsearch([requests.get(es_url, proxies = proxies)])
res = es.search(index=index, body=request, search_type="count")

thanks

[svamet]

or do you mean something like?:

            proxies = {'https': 'http://user:[email protected]:port'}

            s = requests.Session()
            s.proxies = proxies

            es = Elasticsearch([es_url], connection_class = s)
            res = es.search(index=index, body=request, search_type="count")
            return res

Sorry i am realy desperate for a solution to this problem..

[HonzaKral]

this should work:

from elasticsearch import RequestsHttpConnection

class MyConnection(RequestsHttpConnection):
    def __init__(*args, **kwargs):
        proxies = kwargs.pop('proxies', {})
        super(MyConnection, self).__init__(*args, **kwargs)
        self.session.proxies = proxies

es = Elasticsearch([es_url], connection_class=MyConnection, proxies = {'https': 'http://user:[email protected]:port'})


print(es.info())

[svamet]

Thank you very much!! It works perfect!!

[ahmadkarim]

It gives an error "name error" self not defined

[ahmadkarim]

i want to do the same thing as svamet , but i think its because i am using python3.4 i am getting this error, but i cant get around it.

[HonzaKral]

Sorry, my bad, it should be:

from elasticsearch import RequestsHttpConnection

class MyConnection(RequestsHttpConnection):
    def __init__(self, *args, **kwargs):
        proxies = kwargs.pop('proxies', {})
        super(MyConnection, self).__init__(*args, **kwargs)
        self.session.proxies = proxies

es = Elasticsearch([es_url], connection_class=MyConnection, proxies = {'https': 'http://user:[email protected]:port'})

print(es.info())

I forgot the self in __init__.

[ahmadkarim]

Thankyou very much!! i should have figured it out :P

[ahmadkarim]

After setting up the proxy as you devised now when i do:
print ( es.info() )
I receive the following error:

C:\Python34>python.exe C:\Users\akh2rt\Desktop\PythonStuff\PythonGit\Kibana
\ES.py
GET http://xxxx.com:443/ [status:400 request:0.04
6s]
Traceback (most recent call last):
  File "C:\Users\akh2rt\Desktop\PythonStuff\PythonGit\Kibana\ES.py", line 7
8, in <module>
    print(  es.info() )
  File "C:\Python34\lib\site-packages\elasticsearch\client\utils.py", line
69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "C:\Python34\lib\site-packages\elasticsearch\client\__init__.py", li
ne 219, in info
    _, data = self.transport.perform_request('GET', '/', params=params)
  File "C:\Python34\lib\site-packages\elasticsearch\transport.py", line 329
, in perform_request
    status, headers, data = connection.perform_request(method, url, params,
 body, ignore=ignore, timeout=timeout)
  File "C:\Python34\lib\site-packages\elasticsearch\connection\http_request
s.py", line 84, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "C:\Python34\lib\site-packages\elasticsearch\connection\base.py", li
ne 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, err
or_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, '<html>\r\n<head
><title>400 The plain HTTP request was sent to HTTPS port</title></head>\r\
n<body bgcolor="white">\r\n<center><h1>400 Bad Request</h1></center>\r\n<ce
nter>The plain HTTP request was sent to HTTPS port</center>\r\n<hr><center>
nginx/1.9.6</center>\r\n</body>\r\n</html>\r\n')`

[HonzaKral]

You are trying to speak http to https port, could it be that you forgot use_ssl=True in your configuration?

[ahmadkarim]

yeah i thought that too but when i use_ssl= True then i get a lot of errors that latest being

raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPSConnectionPo
ol(host='xxxx.com', port=443): Max retries exceed
ed with url: / (Caused by NewConnectionError('<requests.packages.urllib3.co
nnection.VerifiedHTTPSConnection object at 0x000000000348FE80>: Failed to e
stablish a new connection: [Errno 11004] getaddrinfo failed',))) caused by:
 ConnectionError(HTTPSConnectionPool(host='xxxxx.c
om', port=443): Max retries exceeded with url: / (Caused by NewConnectionEr
ror('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object a
t 0x000000000348FE80>: Failed to establish a new connection: [Errno 11004]
getaddrinfo failed',)))

[HonzaKral]

and can you connect to the target just using requests? It looks like the host cannot resolve the address of the target or proxy.

Either way I am afraid this is out of scope of this library. First step would be to make your environment work with just plain requests from python using your proxy, then I can help you. Until then there is nothing I can do unfortunately.

[ahmadkarim]

I can connect to an *_http *_ server over a proxy and every thing works fine but when i do this:
r = requests.get('https://www.google.com',proxies=proxies, verify= False)
i get:

10, in <module>
    r = requests.get('https://www.google.com',proxies=proxies, verify= Fals
e)
  File "C:\Python34\lib\requests\api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python34\lib\requests\api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python34\lib\requests\sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python34\lib\requests\sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python34\lib\requests\adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.google.c
om', port=443): Max retries exceeded with url: / (Caused by NewConnectionEr
ror('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object a
t 0x0000000003384780>: Failed to establish a new connection: [Errno 11004]
getaddrinfo failed',))

but when i access a local https server on the intranet then request.get works fine without the proxy

[ahmadkarim]

for any one who is having the same problem with request library

https://lukasa.co.uk/2013/07/Python_Requests_And_Proxies/

[shenyan008]

and how to connect a cluster which behind nginx?
nginx proxy_pass to the internal es cluster

[HonzaKral]

@shenyan008 at that point all you have to do is specify connection parameters for that nginx - host and port should suffice

[shenyan008]

@HonzaKral Thank you for your reply, your rely is so quick.
but when I do es.count(), es connect to 127.0.0.1:9200, which is one internal node behind nginx, and failed to create http connection. 127.0.0.1:9200 is a node returned by /_nodes/_all/http

[shenyan008]

@HonzaKral It's my bad, I set the sniff_on_start=true when creating the es instance, but I do not know why this setting will cause the problem

[HonzaKral]

what sniffing does is it inspects the cluster and uses the lost of nodes to do load balancing. Since you don't have direct access to the nodes (and use nginx to connect to them indirectly instead) you cannot use that method. See http://elasticsearch-py.readthedocs.io/en/master/#environment-considerations for more information

[shenyan008]

@HonzaKral Thanks for your help, problem solved

[kanihal]

Shouldn't it be reading proxy settings form http_proxy and https_proxy system env variables, I had set my system proxies using

export http_proxy=http://10.129.3.103:1234
export https_proxy=http://10.129.3.103:1234

Ran
python dummy_using_es.py

but elasticsearch-py doesn't seem to pick proxy from system proxy settings, the request doesn't hit the proxy server.

I understand the temporary fix explained here but what is permanent fix for this?

[HonzaKral]

@kanihal if you just use the requests-based connection class it should pick up those variables automatically.