Requests: Certificate failure with Python 2.7.13 and current Requests on virtualized server

[PatrickDChampion]

I use to use Requests and Python 2.7.6 to access a server with a REST interface. After our organization update their servers, I got the "InsecurePlatformWarning: A true SSLContext object is not available" message and then a certificate failure at the bottom of the call stack.

I have now upgraded to Python 2.7.13 and tried. I stopped getting the InsecurePlatformWarning, but I still got the certificate failure at the end. So, then I updated using pip the Requests from version 2.7.0 to the current 2.13.0. I still get the same certificate failure (without the insecure warning). Below is the call stack. If I force the get() to ignore verification "r = s.get(url, params=addr, verify=False)", the transaction works fine, but I would like to maintain security and use the certificates. I am using Windows 7 Enterprise Service Pack 1 (64 bit) and 64bit Python.

Any idea on what is going wrong. Any suggestions?

Starting Session
Sending request
Traceback (most recent call last):
File "S:FY2015ChoiceCardDailyQcheckSoftAuthOld.py", line 20, in
r = s.get(url,params=addr)
File "C:Python27libsite-packagesrequestssessions.py", line 501, in get
return self.request('GET', url, *kwargs)
File "C:Python27libsite-packagesrequestssessions.py", line 488, in request
resp = self.send(prep, *send_kwargs)
File "C:Python27libsite-packagesrequestssessions.py", line 630, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:Python27libsite-packagesrequestssessions.py", line 190, in resolve_redirects
*adapter_kwargs
File "C:Python27libsite-packagesrequestssessions.py", line 609, in send
r = adapter.send(request, *kwargs)
File "C:Python27libsite-packagesrequestsadapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

[Lukasa]

Assuming the server used to work and now does not, it's most likely that they're serving an invalid certificate or certificate chain. Is the server publicly accessible? If so, can you tell me the host name?

[PatrickDChampion]

Thank you for responding. A closely related link has been working for another application. The server is borderline public. You may or may not have access to it. Can you see if the certificate or chain is valid without having the full REST api? I would prefer to not broadcast to everyone the url with subdirectory path. The top level url is https://www.va.gov but the VA has dozens of servers hanging off that front door. Can I email you the path?

[sigmavirus24]

@PatrickDChampion @Lukasa and I both have our emails on our profiles so you can do just that. =)

[Lukasa]

Yup. If you're really worried we both also have our GPG keys associated with keybase accounts, in case that's helpful.

[PatrickDChampion]

Just sent the full URL.

[PatrickDChampion]

Sorry. Clicked wrong button.

[awalker125]

Hi,

I'm seeing a similar issue in ansible/azure sdk with python 2.7.5 and requests 2.13.0

I've raised an issue in the azure sdk for it here but then found this which sounds similar.

https://github.com/Azure/azure-sdk-for-python/issues/1089

[myusuf3]

So this a pretty bad issue – I am unable to complete any OAuth requests because very call requests makes fails to verify SSL. I have updated my ca-certificates on boxes and I am using latest certifi

I am running Python 2.7.13, I am also suspecting that it works haphazardly, but will need to see it work again to confirm.

I also can't just disable checks, unless you have a global way to disable it in a Django project since packages I am using actually make the lower level calls.

[punkrokk]

I vote for allowing an option to validate an SSL cert with SHA1 root signature. This is the root of the problem here, right? @kennethreitz any ideas?

[myusuf3]

@punkrokk I am totally unsure, what is causing this issue, but verification needs to fail either quietly notifying the user, not blocking the call which is just counter productive to making network calls like so:

Authentication failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

[punkrokk]

Maybe I should open a separate issue, but I thought I followed this because I am having the issue with many APIs, where they have rootCA cert signed 1-2 years ago with SHA1, but requests doesn't like that. It was pretty hard to track down, and I can't seem to get requests to honor a cert bundle built by myself. So - in practice - what devs will do is set verify=False, just to get their https request to even work.

[myusuf3]

It's currently failing to verify a Google domain.

[punkrokk]

Here's an example cert that is failing for me: https://www.ssllabs.com/ssltest/analyze.html?d=tap-api-v2.proofpoint.com&latest (and you can see why in the report)

[Lukasa]

@myusuf3 @punkrokk Hey folks, it would have been better if the two of you had filed a new issue for this. However, yes, this is a 1024-bit root problem. In this case the fix is to change your requests call to requests.get(url, verify=certifi.old_where()). As to the idea of disabling cert validation, this is already possible by passing verify=False, as a quick look at the docs would have told you.

Requests will honor a cert bundle you build if you pass a path to the verify parameter. If this isn't working for you, please file a bug rather than suffer in silence.

[Lukasa]

Ok now that I'm at a computer let me answer you more directly and thoroughly.

I vote for allowing an option to validate an SSL cert with SHA1 root signature. This is the root of the problem here, right? @kennethreitz any ideas?

@punkrokk Requests by default already trusts the 1024-bit roots (this isn't a SHA1 concern, it's a 1024-bit concern). However, when it finds certifi in the environment it automatically updates to use the strong trust bundle present in certifi, which does not trust those roots.

verification needs to fail either quietly notifying the user, not blocking the call which is just counter productive to making network calls like so:

That's nonsensical. If the certificate verification fails, one possibility is that you are under active man-in-the-middle attack. Soft-failure of certificate validation is unacceptable: it's just not an option.

Maybe I should open a separate issue, but I thought I followed this because I am having the issue with many APIs, where they have rootCA cert signed 1-2 years ago with SHA1, but requests doesn't like that.

Requests does not police the signature algorithms used in a certificate chain.

It's currently failing to verify a Google domain.

This is because the Google certificate has two possible certificate chains it could build. One leads to a root certificate present in the certifi bundle that has a 2048-bit RSA key. The other leads to a root certificate not present in the certifi bundle that has a 1024-bit RSA key. These 1024-bit RSA keys are desperately unsafe, and so by default when Requests can find certifi it will not choose to trust them.

You can avoid this problem either by moving to a newer OpenSSL or by pointing Requests at certifi.old_where(). In the first case, this resolves OpenSSL's cert chain building problem to allow it to find the 2048-bit root that is actually present. In the second case, this will point to a version of certifi that contains the old, weak roots. I strongly recommend doing the first, rather than the second.

Here's an example cert that is failing for me: https://www.ssllabs.com/ssltest/analyze.html?d=tap-api-v2.proofpoint.com&latest (and you can see why in the report)

This validates just fine for me. Your issue isn't that there is a SHA-1 signature, though that is bad, but because like Google this site uses a cross-signed root certificate.

As an explainer of what this is, it occurred because Thawte had issued certificates for a long time that chained up to their Thawte Premium Server CA root certificate. This root certificate has a 1024-bit RSA key, which is extremely vulnerable to brute-force attacks to recover the private key. If the private key is recovered, it will be possible to create new certificates that appear to have been issued by the Thawte Premium Server CA. For this reason browsers have removed all 1024-bit RSA root certificates from their trust stores and CAs minted new 2048-bit-or-larger roots.

However, some older browsers may not have the ability to update their cert stores, and so may not know about the new 2048-bit roots. For this reason, CA's "cross-signed" them. That is, they created two versions of the certificate: one is self-signed, and distributed to trust stores; the other is signed by the old 1024-bit root. This is true of the root CA here, which is thawte Primary Root CA.

If you look back in your ssllabs log you'll see the remote server is sending a copy of this certificate over that is signed by Thawte Premium Server CA. However, if you expand the "certification paths" section of the report you'll see that the thawte Primary Root CA is being treated as a root, despite not being self-signed. What gives?

Well, SSLLabs has a cert chain builder that looks for roots in its trust store before it chains up to a cert sent by the remote server. This means that when it looks for a cert called "thawte Primary Root CA" it looks in its trust database (e.g. like certifi) for that root cert, and finds it. Thus, it has built a trusted chain.

Older OpenSSLs do not do this. They always build a cert chain first using the certs sent by the remote server. For this reason, OpenSSL will use the cross-signed root sent by the server, and so will look for Thawte Premium Server CA to root the chain. certifi doesn't have this cert (because it's vulnerable to attack), so OpenSSL gives up. Newer versions of OpenSSL are smarter about this, and so they have no trouble validating a cert chain like this (which is why I don't encounter the problem).

Your advice is the same as what I give to @myusuf3: either upgrade OpenSSL, or use the weak cert bundle. I strongly recommend the former.

[Lukasa]

By the way, you'll see that certifi contains thawte Primary Root CA. It also has Thawte Premium Server CA in a file labelled "old_root.pem", which is where the 1024-bit roots are stored in case they're needed.

[punkrokk]

@Lukasa Cory - thanks so much for this explanation. Very helpful. I agree with you. If I have problems with the cert bundle I will open a new ticket. But I suspect the internet will appreciate your explanation in the future. I couldn't figure this out easily. What version of OpenSSL do you recommend?

[myusuf3]

@Lukasa I have updated OpenSSL and has fixed my issue. Thanks. Documentation around these errors would be great. Your _effort_ is appreciated.

As for the suggestion to pass verified=False to the request call, that assumes I own all the code that uses requests a way to knowingly turn off that functionality would be appreciated.

Thanks again.

[Lukasa]

a way to knowingly turn off that functionality would be appreciated.

Requests very deliberately provides no such global switch: turning off certificate validation for all programs that use Requests very easily is a bad idea, and opens up real problems with securing a system. We provide environment variables that allow you to point us at custom cert bundles, which is the recommended solution: point us at your centralised CAs and we'll do the right thing.

[myusuf3]

@Lukasa this is turning its ugly head again. I have the latest update for OpenSSL. What could be the issue now? I also have the most recent certifi installed as well.

[Lukasa]

What OpenSSL are you using?

[myusuf3]

@Lukasa OpenSSL 1.0.1f 6 Jan 2014 I just upgraded to most recent and I am still getting the same issue. do I need to reinstall requests, or certifi

[Lukasa]

Neither should be required. OpenSSL 1.0.1f may still have chain building problems, depending on how your vendor provided it to you. Are you using certifi.old_where()?

[myusuf3]

I had to rebuild Python with the new OpenSSL that built from scratch. Software man. thanks @Lukasa