Requests: Issue with Lets Encrypt certificates

Created on 30 Dec 2015 · 13Comments · Source: psf/requests

Having problems getting Requests to work with ssl certificates issued by lets encrypt. The error message is very unhelpful in narrowing down what the issue is exactly.

Lets see if I can post the right error this time:

File "E:anaconda27lib\site-packages\requestsapi.py", line 67, in get
return request('get', url, params=params, *_kwargs)
File "E:anaconda27lib\site-packages\requestsapi.py", line 53, in request
return session.request(method=method, url=url, *_kwargs)
File "E:anaconda27lib\site-packages\requests\sessions.py", line 468, in request
resp = self.send(prep, *_send_kwargs)
File "E:anaconda27lib\site-packages\requests\sessions.py", line 576, in send
r = adapter.send(request, *_kwargs)
File "E:anaconda27lib\site-packages\requestsadapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Source

yhorian

Most helpful comment

Well, the site in question is weird.

The validation failing here is actually for https://36monkeys.com/, which is where the original site redirects to. That site doesn't load in Chrome because it negotiates HTTP/2 or SPDY but doesn't actually define a secure-enough connection, which is moderately amusing, but it _does_ load in Safari.

But the problem is revealed by loading the site using OpenSSL. The server presents the following certificate chain:

Certificate chain
 0 s:/CN=36monkeys.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---

That is, it presents only the leaf certificate, without presenting the Let's Encrypt Authority X3 certificate. That is not enough: the reason Safari and friends can validate the certificate is because they ship with a list of intermediate certificates, but most operating systems and clients do not.

The Let's Encrypt docs say this too:

Please note, that you must use either chain.pem or fullchain.pem.

This server is not doing that: it's serving only cert.pem.

Please contact the server administrator and ask them to reconfigure their server.

Lukasa on 27 May 2016

👍4 ❤1 🎉1

All 13 comments

I previously submitted a post with error messages testing httplib2 and other libraries, which are having similar issues but aren't providing much more information on why they're returning this error when browser and a PHP script aren't having this problem.

yhorian on 30 Dec 2015

@yhorian1 Aha, there we go!

So, the problem here is almost certainly that your OpenSSL is too old. PHP is likely succeeding because it doesn't verify certificates at all or because it's not using OpenSSL.

OpenSSL versions less than 1.0.2 don't backtrack when trying to build up certificate chains, which means that if it encounters a cross-signed certificate it is unable to check whether or not it has a non-cross-signed version of that certificate in the trust store. That's what's happening in this case I suspect: Let's Encrypt is currently a cross-signed CA.

Do you mind providing a specific URL so I can validate my suspicion?

Lukasa on 30 Dec 2015

https://evil-source.co.uk/ is the site I was testing it.

yhorian on 30 Dec 2015

Thanks. So this is working fine on my OS X machine, using Python 2.7.11 with OpenSSL 1.0.2e. Can you let me know what version of requests you're using, where you got it from, and whether you have certifi installed?

Lukasa on 30 Dec 2015

Testing on python 3.4:
certifi 2015.11.20.1
requests 2.9.1

Testing on python 2.7:
certifi 2015.11.20.1
requests 2.9.1

built using anaconda on win10

yhorian on 30 Dec 2015

Interesting. Can you try using requests.get('https://evil-source.co.uk', verify=certifi.old_where())? (You'll need to import requests and certifi).

Lukasa on 30 Dec 2015

requests.get('https://evil-source.co.uk', verify=certifi.old_where())
File "E:anaconda27lib\site-packages\requestsapi.py", line 67, in get
return request('get', url, params=params, *_kwargs)
File "E:anaconda27lib\site-packages\requestsapi.py", line 53, in request
return session.request(method=method, url=url, *_kwargs)
File "E:anaconda27lib\site-packages\requests\sessions.py", line 468, in request
resp = self.send(prep, *_send_kwargs)
File "E:anaconda27lib\site-packages\requests\sessions.py", line 576, in send
r = adapter.send(request, *_kwargs)
File "E:anaconda27lib\site-packages\requestsadapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Process finished with exit code 1

No joy. Checked in python 3.4 and 3.5 as well, openssl is 1.0.2d

yhorian on 30 Dec 2015

Now that's _very_ interesting.

Do you have PyOpenSSL, ndg-httpsclient, and pyasn1 installed?

Can you find where anaconda has put the OpenSSL binary?

Lukasa on 30 Dec 2015

Ugh, so checked logs and figured it out.

There was a routing policy that was causing the ISP to re-route requests internally, causing the server to issue the wrong page. Hence the failure. Tried from a different location with similar setup - works fine. VPN switched on, works fine. This is possibly one of the more frustrating network policies I've ever encountered - even if the DNS says the site is in the same block they should be treating it as a public request, not a local!

Sorry for wasting your time!

yhorian on 30 Dec 2015

That's definitely a perplexing setup, well debugged on your end! No need to apologise: you didn't waste my time, you needed help to get past the possibility that your libraries were doing something wrong. I'm just glad you were able to come to a resolution!

Lukasa on 30 Dec 2015

I have similar problem, and it fails from two different locations and two different OS.
On my local machine (OSX), on fresh virtualenv:

Without certifi:

>>> r = requests.get('http://36apor.se')
Traceback (most recent call last):
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py", line 722, in _validate_conn
    conn.connect()
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connection.py", line 229, in connect
    ssl_version=resolved_ssl_version)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/util/ssl_.py", line 123, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 376, in wrap_socket
    _context=self)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 747, in __init__
    self.do_handshake()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 983, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 628, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py", line 543, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 595, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 595, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 189, in resolve_redirects
    allow_redirects=False,
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

With certifi:

>>> r = requests.get('http://36apor.se', verify=certifi.old_where())
Traceback (most recent call last):
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py", line 722, in _validate_conn
    conn.connect()
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connection.py", line 229, in connect
    ssl_version=resolved_ssl_version)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/util/ssl_.py", line 123, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 376, in wrap_socket
    _context=self)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 747, in __init__
    self.do_handshake()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 983, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 628, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/packages/urllib3/connectionpool.py", line 543, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 595, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 595, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 189, in resolve_redirects
    allow_redirects=False,
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/Users/mysz/Library/Python/2.7/lib/python/site-packages/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

And from other location with Ubuntu Xenial:

Without certifi:

>>> r = requests.get('http://36apor.se')
Traceback (most recent call last):
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 351, in _make_request
    self._validate_conn(conn)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 814, in _validate_conn
    conn.connect()
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 289, in connect
    ssl_version=resolved_ssl_version)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/util/ssl_.py", line 308, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 376, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 748, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 984, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 629, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 604, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 606, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 606, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 179, in resolve_redirects
    **adapter_kwargs
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

And without certifi:

>>> r = requests.get('http://36apor.se', verify=certifi.old_where())
Traceback (most recent call last):
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 351, in _make_request
    self._validate_conn(conn)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 814, in _validate_conn
    conn.connect()
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 289, in connect
    ssl_version=resolved_ssl_version)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/util/ssl_.py", line 308, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 376, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 748, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 984, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 629, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 604, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 606, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 606, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 179, in resolve_redirects
    **adapter_kwargs
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/home/mysz/aportest/venv-2/lib/python3.5/site-packages/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

Certificates on this domain is issued using letsencrypt.
On OSX, I have OpenSSL 0.9.8zh 14 Jan 2016, and on Ubuntu there is OpenSSL 1.0.2g-fips 1 Mar 2016.

msztolcman on 27 May 2016

Well, the site in question is weird.

But the problem is revealed by loading the site using OpenSSL. The server presents the following certificate chain:

Certificate chain
 0 s:/CN=36monkeys.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---

The Let's Encrypt docs say this too:

Please note, that you must use either chain.pem or fullchain.pem.

This server is not doing that: it's serving only cert.pem.

Please contact the server administrator and ask them to reconfigure their server.

Lukasa on 27 May 2016

👍4 ❤1 🎉1

@Lukasa You are absolutely right, server config was broken. I fixed it and now it works fine - thanks!

msztolcman on 27 May 2016

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Raise exception if status code doesn't equal 200

Matt3o12 · 3Comments

Bad error handling in requests in python 3 [During handling of the above exception, another exception occurred]

Gonzalliz · 3Comments

idna version pin does not allow version 2.7

jakul · 3Comments

`PUT` to Amazon S3 is failing

avinassh · 4Comments

Session baseUrl?

NoahCardoza · 4Comments