requests: Support SSL_CERT_FILE environment variable

[thoger]

Requests allows setting path to a CA certificate bundle that should be used instead of the default system one using the following environment variables: REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE.

https://github.com/kennethreitz/requests/blob/v2.8.1/requests/sessions.py#L621

Please make it also check SSL_CERT_FILE environment variable in the same way. This variable is recognized by OpenSSL and is documented in PEP 476 as a way to make Python standard library https clients use non-default CA bundle file:

https://www.python.org/dev/peps/pep-0476/#trust-database

It sounds reasonable to have a single environment variable that can be used for all Python scripts regardless of whether they are using requests or standard library httplib / urllib*.

In terms of the fix, a simple or os.environ.get('SSL_CERT_FILE') should do the trick.

[Lukasa]

This seems reasonable enough to me. I'd be happy to accept a patch that does this. We should also accept SSL_CERT_DIR.

[kennethreitz42]

This seems harmless enough.

[raultabacu]

Is this still an issue ? If so I would like to give it a shot.

[Lukasa]

@Keops92 The work has been done, sadly: see #2903.

[raultabacu]

Thanks !

[luv]

What about adding a new keyword (I don't know, let's say verify_native ) and if set to True, requests would respect SSL_CERT_FILE and SSL_CERT_DIR as well as try to use platform-native certificate storage? This shouldn't break backwards compatibility.

[Lukasa]

The Requests team are _strongly_ opposed to adding new keyword arguments, especially if they are redundant with, overlap with, or can be set to contradictory values with a pre-existing keyword argument.

[luv]

That makes sense.

Is there a plan to support system certificate storage in the next compatibility-breaking release?

[Lukasa]

@luv Currently the expectation is that a future urllib3 release will allow the passing of custom SSLContext objects to urllib3. That will allow construction of adapters that let you use the system certificate store on Linux. For OS X and Windows we have the beginnings of some code to do it, but it's a lot of very subtle work that needs a lot of bugs ironed out.

[luv]

Thanks for answer.

Actually, are you sure you need custom SSLContext to use system certificate store on Unix/Linux? It should be enough to pass correct path to "verify" keyword and you are sorted, even with the current version of requests.

I realize it must be really PITA working on supporting OS X, Windows, Unix and Linux system certificate store transparently.

Btw. from your previous comment it was not clear if you plan to limit this functionality to OS X and Windows only (and leave it up to the end-user to support unix and linux system certificates with custom SSLContext) or if you plan to support system certificates on Linux (and *BSD and friends :) ) as well.

Thanks!

[Lukasa]

It should be enough to pass correct path to "verify" keyword and you are sorted, even with the current version of requests.

I wish, but it is not. There is no consistent "correct" path that works across all Unices. There is no consistent "correct" path that works across all Linux distributions. There is not even a guarantee that there's a consistent "correct" path that works across all _versions_ of the same distribution. The reason the SSLContext approach is preferred is simply that it allows you to defer that choice to the distribution-provided OpenSSL, which is hopefully correctly configured.

Btw. from your previous comment it was not clear if you plan to limit this functionality to OS X and Windows only (and leave it up to the end-user to support unix and linux system certificates with custom SSLContext) or if you plan to support system certificates on Linux (and *BSD and friends :) ) as well.

The ideal end goal will be that there's a urllib3 flag you can hit to change its verification logic, and that requests will make it available in some way. That will work for all platforms.

[luv]

I see. Would you mind providing an example where curl's approach to finding the "correct path" does not work https://github.com/curl/curl/blob/master/acinclude.m4#L2627 ?

The reason the SSLContext approach is preferred is simply that it allows you to defer that choice to the distribution-provided OpenSSL, which is hopefully correctly configured.

Oh ... if this is going to depend on functions like X509_get_default_cert_dir - it might be, in the real world, more problematic than hard-coded paths :( ... http://lists.freebsd.org/pipermail/freebsd-python/2014-December/007809.html

[Lukasa]

Would you mind providing an example where curl's approach to finding the "correct path" does not work

Well, for one, in the FreeBSD post you just linked, which has the certs at /usr/local/etc/ssl/cert.pem and /usr/local/share/certs/ca-root-nss.crt, neither of which are in curl's list:

/etc/ssl/certs/ca-certificates.crt
/etc/pki/tls/certs/ca-bundle.crt
/usr/share/ssl/certs/ca-bundle.crt
/usr/local/share/certs/ca-root.crt
/etc/ssl/cert.pem

The general concerns here are the following set:

1. Requests does not want to have any quick-fire way of enabling this function unless it works across _all_ platforms we support uniformly. Right now, we cannot do that. This means we do not want to advertise an API for it. Any easy-access API that only works on some systems is a _bad_ API in this case, because it contributes to user confusion.
2. However, we are willing for users who have more specialised needs to overrule us. That is part of the reason the Transport Adapter API layer exists: it allows users to configure urllib3 directly in a way that enables them to support use-cases they know will work in their more general scenario. That includes this particular use-case.
3. Ultimately, there is no cross-platform way to locate SSL certificates, and if Unices ship OpenSSLs that do not _actually_ point at the correct certificate locations then there is _no_ automated solution available to us.

It's the wild west out here, with people dumping certs wherever the hell they want, and we need a system that doesn't mysteriously fail as much as possible.

If you believe the curl approach works for your software, then by all means, you may implement it in your own software. You could even create a third-party module like certifi that implements a similar API and also implements curl's logic. However, it's my belief that curl's logic is somewhat fragile and only works because it allows configuration overrides in the build scripts, which most distros take advantage of. Those same distros make equivalent changes in requests when shipped on their platform, so I see no reason to want to emulate curl in this case.

[luv]

sounds good

[michael-o]

@Lukasa /etc/ssl/cert.pem is now by default linked by the ca_root_nss port.

[ssbarnea]

I have a more practical question: I am a Mac OS user which is forced to use custom CA certificates. Importing them in KeyChain works well for ALL browsers and even for curl. Even if I am unfortunate enough to have to install the alternative curl via brew I do still have the option to use SSL_CERT_FILE environment variable in order to workaround the fact that that one fails to load certs from the keychain.

Now, the big problem is how to convince requests to work? And when I say convince I mean that I need a solution that does NOT require me to patch any library that may be using requests.

[luv]

Can't you use REQUESTS_CA_BUNDLE environment variable?

http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

[Lukasa]

As @luv suggested, that will work.

[ssbarnea]

@luv Thanks, I can confirm that if I do export REQUESTS_CA_BUNDLE=$SSL_CERT_FILE i would be able to use requests the same way as other libraries. I dont really know why requests needs another environment variable instead of re-using the same one but I guess there were some reasons for that.

While I did this I observed something that looks like a bug: requests.certs.where() returns the same location even when you define REQUESTS_CA_BUNDLE, even if the library is using the ones from REQUESTS_CA_BUNDLE. Should I open a bug for this or is not really a bug?

[Lukasa]

I dont really know why requests needs another environment variable instead of re-using the same one but I guess there were some reasons for that.

Strictly, SSL_CERT_FILE is an OpenSSL environment variable. Requests doesn't use it because it does not ever check for it being set, and never trusts that OpenSSL will locate certs. In principle it could watch for this variable, though: the patch would be small.

Should I open a bug for this or is not really a bug?

It's not a bug. requests.certs.where is an implementation detail of Requests and doesn't claim to tell you where the cert bundle Requests will actually use is. This is partly because it can't: the location of the bundle can be changed by properties of the Session and arguments to the various request methods. All where() ever does is point to the _on-disk_ location of the Requests bundled certificates.

[WhyNotHugo]

Can we close this? #2903 has already been merged into the proposed/3.0.0 branch.

[kennethreitz42]

@hobarrera thanks!