Elasticsearch-py: Basic Auth with special characters seems to not work

[pandaadb]

Hi

I have basic authentication enabled on my ES cluster and I am attempting to use the python client to connect to it. It appears that the hash character is not treated correctly with the authentication.

I have the following users set up in my test environment:

artur:m@m
artur:m@m#m

So, my tests: (Note i added some debugging code which is why it outputs what it reads

>>> es.Elasticsearch(["artur:m@m@localhost:9200"]).cluster.health()
//artur:m@m@localhost:9200
artur:m@m
{u'status': u'yellow', u'number_of_nodes': 1, u'unassigned_shards': 16, u'number_of_pending_tasks': 0, u'number_of_in_flight_fetch': 0, u'timed_out': False, u'active_primary_shards': 16, u'task_max_waiting_in_queue_millis': 0, u'cluster_name': u'elasticsearch', u'relocating_shards': 0, u'active_shards_percent_as_number': 50.0, u'active_shards': 16, u'initializing_shards': 0, u'number_of_data_nodes': 1, u'delayed_unassigned_shards': 0}
>>> 
>>> 
>>> es.Elasticsearch(["artur:m@m#m@localhost:9200"]).cluster.health()
//artur:m@m#m@localhost:9200
artur:m
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 71, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/cluster.py", line 33, in health
    'health', index), params=params)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 120, in perform_request
    raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f4f2cad6590>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f4f2cad6590>: Failed to establish a new connection: [Errno -2] Name or service not known)

The second attempt splits the URL wrong and therefore does not create a connection at all. The first one is successful.

With base64 encoding:

artur: YXJ0dXI=
m@m: bUBt
m@m#m: bUBtI20=

>>> es.Elasticsearch(["artur:bUBtI20=@localhost:9200"]).cluster.health()
//artur:bUBtI20=@localhost:9200
artur:bUBtI20=
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 71, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/cluster.py", line 33, in health
    'health', index), params=params)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 124, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 122, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.AuthenticationException: TransportError(401, u'Sorry, your request is forbidden.')
>>> es.Elasticsearch(["artur:bUBt@localhost:9200"]).cluster.health()
//artur:bUBt@localhost:9200
artur:bUBt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 71, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/cluster.py", line 33, in health
    'health', index), params=params)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 124, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 122, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.AuthenticationException: TransportError(401, u'Sorry, your request is forbidden.')
>>> es.Elasticsearch(["YXJ0dXI=:bUBtI20=@localhost:9200"]).cluster.health()
//YXJ0dXI=:bUBtI20=@localhost:9200
YXJ0dXI=:bUBtI20=
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 71, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/cluster.py", line 33, in health
    'health', index), params=params)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 124, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 122, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.AuthenticationException: TransportError(401, u'Sorry, your request is forbidden.')

>>> es.Elasticsearch(["artur:m%40m%23m@localhost:9200"]).cluster.health()
//artur:m%40m%23m@localhost:9200
artur:m%40m%23m
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 71, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/client/cluster.py", line 33, in health
    'health', index), params=params)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 124, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/artur/.local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 122, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.AuthenticationException: TransportError(401, u'Sorry, your request is forbidden.')

The last attempt is with url encoding instead of base64, this is because the curl alternative works with urlencoding.

None of my attempts seem to be doing the right thing here. All of them are unauthorized. And here is the curl alternative to this: (url encode instead of base64)

artur@pandaadb:~/dev/repo/es2csv$ curl 'artur:m%40m%23m@localhost:9200/_cat/indices?v'
health status index             pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   uptime-2016.08.10   5   1        160            0       86kb           86kb 
yellow open   .kibana             1   1          2            0      8.8kb          8.8kb 
yellow open   uptime-2016.09.26   5   1          2            0      9.5kb          9.5kb 
yellow open   uptime-2016.08.09   5   1        145            0     78.4kb         78.4kb 

So, I am not sure if this is a python issue, or if this is a client issue, or anything else, but I believe this should work?

Please let me know if I am using the client wrong..

Thanks!,

Artur

[HonzaKral]

Thanks for the report, this seems to be an issue in the underlying http library with the handling of the password in the url. Could you please try passing in the auth separately? es.Elasticsearch(["localhost:9200"], http_auth="artur:m@m").cluster.health() should work.

[pandaadb]

Hi,

yes I thought that this might have something to do with the python library. You are right, passing it separately does work:

>>> import elasticsearch as es
>>> es.Elasticsearch(["localhost:9200"], http_auth="artur:m@m").cluster.health()
//localhost:9200
{u'status': u'yellow', u'number_of_nodes': 1, u'unassigned_shards': 16, u'number_of_pending_tasks': 0, u'number_of_in_flight_fetch': 0, u'timed_out': False, u'active_primary_shards': 16, u'task_max_waiting_in_queue_millis': 0, u'cluster_name': u'elasticsearch', u'relocating_shards': 0, u'active_shards_percent_as_number': 50.0, u'active_shards': 16, u'initializing_shards': 0, u'number_of_data_nodes': 1, u'delayed_unassigned_shards': 0}
>>> es.Elasticsearch(["localhost:9200"], http_auth="artur:m@m#m").cluster.health()
//localhost:9200
{u'status': u'yellow', u'number_of_nodes': 1, u'unassigned_shards': 16, u'number_of_pending_tasks': 0, u'number_of_in_flight_fetch': 0, u'timed_out': False, u'active_primary_shards': 16, u'task_max_waiting_in_queue_millis': 0, u'cluster_name': u'elasticsearch', u'relocating_shards': 0, u'active_shards_percent_as_number': 50.0, u'active_shards': 16, u'initializing_shards': 0, u'number_of_data_nodes': 1, u'delayed_unassigned_shards': 0}

I thought of raising it anyway - maybe one could implement manual decoding so that it would accept urlencoded user/pass pairs for that.

My main issue is that I am trying to use es2csv which does only allow auth via URL not with the extra parameter. I might raise it with them to add this option explicitly.

[HonzaKral]

if you just pass in "http://artur:m@m@localhost:9200" it will also work as we parse that correctly. If you pass in the username:pass@host:port otherwise we don't try and parse it as an http url since it's lacking the schema.

My assumption was that people would pass in just host:port or be explicit by passing in either {'host': 'localhost', 'http_auth': 'user:pass'} or the full URI http://user:pass@localhost:9200/. This in-between was not expected.

[pandaadb]

Yep, the issue is the hash sign in the password that breaks the behaviour. For now, we created a new user with a more sane password :)

[HonzaKral]

I am sorry, but I have to close this issue, altering python's urlparse is out of scope of the work and there is a suitable workaround with passing in the auth information separately.

Thank you for raising it!

[pandaadb]

Not a problem at all - you are right, I am not expecting you to fix python libraries and there is a workaround with that :) My issue was with es2csv tooling and I submitted a fix for that using the workaround you suggested. All good - thank you!