Elasticsearch-py: "elasticsearch.exceptions.ImproperlyConfigured: Root certificates are missing..." even with certifi installed

[anfreita]

Creating an Elasticsearch client with use_ssl=True and verify_certs=True fails with elasticsearch.exceptions.ImproperlyConfigured: Root certificates are missing for certificate validation. Either pass them in using the ca_certs parameter or install certifi to use it automatically.

running this in python 2.7 with certifi installed:

$ python --version                                                                                                                                                                                              11:39:54
Python 2.7.12+
$ pip list                                                                                                                                                                                                      11:38:52
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
backports.shutil-get-terminal-size (1.0.0)
bcrypt (3.1.3)
boto3 (1.4.7)
botocore (1.7.43)
certifi (2017.11.5)
cffi (1.10.0)
chardet (3.0.4)
decorator (4.1.2)
Django (1.11.6)
django-crispy-forms (1.6.1)
docutils (0.14)
elasticsearch (6.0.0)
enum34 (1.1.6)
falcon (1.2.0)
futures (3.1.1)
idna (2.6)
ipython (5.5.0)
ipython-genutils (0.2.0)
jmespath (0.9.3)
lxml (3.8.0)
numpy (1.13.3)
olefile (0.44)
pandas (0.21.0)
pathlib2 (2.3.0)
pexpect (4.3.0)
pickleshare (0.7.4)
Pillow (4.2.1)
pip (9.0.1)
pkg-resources (0.0.0)
prompt-toolkit (1.0.15)
ptyprocess (0.5.2)
pycparser (2.18)
Pygments (2.2.0)
python-dateutil (2.6.1)
python-mimeparse (1.6.0)
pytz (2017.3)
requests (2.18.4)
requests-aws4auth (0.9)
requests-futures (0.9.7)
s3transfer (0.1.11)
scandir (1.6)
setuptools (36.0.1)
simplegeneric (0.8.1)
six (1.11.0)
SkySerpent (0.1.55)
traitlets (4.3.2)
trollius (2.1)
urllib3 (1.22)
wcwidth (0.1.7)
webkit2png (0.8.2)
wheel (0.30.0a0)

This appears to only affect version 6.0.0. I was able to successfully create the client in the same environment after backporting to 5.5.0

[HonzaKral]

Thanks, this is a bug cause by https://github.com/elastic/elasticsearch-py/pull/635#discussion_r134012922

The workaround is to pass in the ca_certs explicitly: Elasticsearch(use_ssl=True, ca_certs=certifi.where())

sorry about that!

[anfreita]

Thanks @HonzaKral for the quick reply. The workaround suffices!

[fxdgear]

@anfreita I'm having trouble recreating this error.

I have python 2.7.
I have certifi installed.

I'm creating an instance of Elasticsearch on elastic cloud that I created temporarily for this test.

In [1]: auth = ('elastic', 'my_password')

In [2]: from elasticsearch import Elasticsearch

In [3]: es = Elasticsearch("https://007ad05d9cd5a97ced39140436edc6d8.us-east-1.aws.found.io:9243/", http_auth=auth, use_ssl=True, verify_certs=True)

In [4]: es.info()
Out[4]: 
{u'cluster_name': u'007ad05d9cd5a97ced39140436edc6d8',
 u'cluster_uuid': u'W3KmtFS8TpGe6ztWzH107A',
 u'name': u'instance-0000000000',
 u'tagline': u'You Know, for Search',
 u'version': {u'build_date': u'2017-11-10T18:41:22.859Z',
  u'build_hash': u'8f0685b',
  u'build_snapshot': False,
  u'lucene_version': u'7.0.1',
  u'minimum_index_compatibility_version': u'5.0.0',
  u'minimum_wire_compatibility_version': u'5.6.0',
  u'number': u'6.0.0'}}

[anfreita]

@fxdgear apologies for the delay. The only difference I see between mine and your setup is that I'm still running v5.5 of Elasticsearch. I am still getting the error though:

>>> c = Elasticsearch(hosts=['******'], port=443, use_ssl=True, verify_certs=True, http_auth=('****', '****'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/andrew/src/Aquarium/venv/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 170, in __init__
    self.transport = transport_class(_normalize_hosts(hosts), **kwargs)
  File "/home/andrew/src/Aquarium/venv/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 108, in __init__
    self.set_connections(hosts)
  File "/home/andrew/src/Aquarium/venv/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 155, in set_connections
    connections = map(_create_connection, hosts)
  File "/home/andrew/src/Aquarium/venv/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 154, in _create_connection
    return self.connection_class(**kwargs)
  File "/home/andrew/src/Aquarium/venv/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 91, in __init__
    raise ImproperlyConfigured("Root certificates are missing for certificate "
elasticsearch.exceptions.ImproperlyConfigured: Root certificates are missing for certificate validation. Either pass them in using the ca_certs parameter or install certifi to use it automatically.
>>> c = Elasticsearch(hosts=['******'], port=443, use_ssl=True, verify_certs=True, http_auth=('****', '****'), ca_certs=certifi.where())
>>> c.info()
{u'cluster_name': u'pentairiot',
 u'cluster_uuid': u'jfN5gAlITUKYeoeCFNtgwg',
 u'name': u'46nZLyM',
 u'tagline': u'You Know, for Search',
 u'version': {u'build_date': u'2017-06-30T23:16:05.735Z',
  u'build_hash': u'260387d',
  u'build_snapshot': False,
  u'lucene_version': u'6.6.0',
  u'number': u'5.5.0'}}

[fxdgear]

fix here: https://github.com/elastic/elasticsearch-py/pull/689

[DeeeFOX]

es = Elasticsearch("https://host:port", http_auth=(user,pass), verify_certs=False)
worked

[marcelloromani]

@DeeeFOX that is not a fix, that is avoiding certificate validation altogether :-)

[raveenb]

this problem is showing up when accessing ES from Redis Worker Queue but not when accessed via Jupyter

the workaround works well though