Httpx: Drop certifi, use system trust store by default

[sethmlarson]

This is an issue that many people have been trying to tackle for Python but hasn't been done yet. I think we're in a pretty good position to try to tackle this in a way that can be available to everyone.

• Linux/OpenSSL should use the default system CA bundle path compiled into OpenSSL.
• Windows should use SChannel which is capable of fetching updated CA certs if they're not in the trust store.
• macOS should use SecureTransport

This should probably be implemented as a separate library, maybe use a stripped-down oscrypto project as a starting point.

Reasons to do this:

• HTTPX will use the same CA certs as the system
• Easier configuration and deployment to corporate settings
• If a system is shipped with outdated OpenSSL (Windows, macOS) we don't care because the system is more likely to be up to date than whatever was shipped with Python.
• Windows and macOS automatically update their certs and CRLs.

[yeraydiazdiaz]

It's been a while since this was raised has been brought up recently a couple of times, could we go into detail of what's needed here?

For reference oscrypto lists other related libraries, would any of those fit our needs? (I'd look myself but I just don't know enough about the subject 🙂 )

[florimondmanca]

Hey!

So considering the amount of :+1:'s on this issue I thought it would make sense to take some kind of decision on this as part of 1.0 - https://github.com/encode/httpx/issues/947#issuecomment-629698226

I'm personally on the side of dropping certifi as well, mainly because 1/ it's an extra dependency and 2/ it's probably better for certifi to be _opt-in_ (e.g. we can hint it in the Requests migration guide) rather than _opt-out_. Other reasons mentioned by Seth in the issue description also apply.

I'm pretty much a n00b in terms of TLS implementations, let alone system compatibility, so let me ask what I feel might be a n00b question but maybe not after all…

What if we just used ssl.create_default_context()?

From the docs (emphasis mine):

Return a new SSLContext object with default settings for the given purpose. The settings are chosen by the ssl module, and usually represent a higher security level than when calling the SSLContext constructor directly. (So it's supposed to be *secure enough choice, right?)*

cafile, capath, cadata represent optional CA certificates to trust for certificate verification, as in SSLContext.load_verify_locations(). If all three are None, this function can choose to trust the system’s default CA certificates instead.

TL;DR: it looks to me ssl.create_default_context() should* create an ssl_context that uses the system defaults. Is there anything else we'd need? Or would we not be considering this because it's not _the always most secure choice everywhere_?

(The usage of "can*" in the docs looks suspicious to me. Are there cases when Python _wouldn't_ trust the default system CA? And what happens if it doesn't? Would the default ssl_context then be a "no verify" context?)

I'm wondering if this wouldn't be an okay default, mainly because two notable projects use and/or recommend the use of ssl.create_default_context() for configuring TLS:

• aiohttp
• websockets

[tomchristie]

Not my core expertise, but pretty sure it's not a solved issue.

I sat down with @lukasa last year briefly, and one of the technical issues we talked about was dropping certifi, and the complexities of accessing the system trust store on different O/S's.

I'm sure @sethmlarson has a much better handle on the current state of affairs here. I think either @tiran's or @glyph's name might also have come up in my conversation with Cory, wrt. folks having done some work in this area, but I might be getting that wrong. Kinda a PyCon type thing that would've benefited from being able to chat it out with relevant folks, but...

In any case, assuming I have got the landscape correct here, I think "use system trust store by default" sits firmly in "make this work in an independent, tightly-scoped third party package", rather than specifically "solve this in httpx".

If and when such a package exists, then yup we can take a look at using it, otherwise it's in the realm of "this isn't a resolved issue in the Python ecosystem, certifi is the best we can do until then".

[florimondmanca]

Just found out about this discussion starting back from 2016 in the Requests repo, loads of good background there: https://github.com/psf/requests/issues/2966

Edit, after reading it:

it's in the realm of "this isn't a resolved issue in the Python ecosystem, certifi is the best we can do until then".

Agreed.

(Also I think I found an answer there to "Wouldn't create_default_context() be enough?", which is "yes, but only on *nix, and only if OpenSSL is linked correctly". We already support passing a preconfigured ssl_context as verify, so we probably cover enough ground there already.)

[tiran]

I need to finish my prototype...

[florimondmanca]

Another data point in favor of us keeping certifi at this time: aiohttp not bundling certs by default seems to have been causing users a lot of pain… https://github.com/aio-libs/aiohttp/issues?q=is%3Aissue+verify+failed+

In a lot of these issues the solutions are often a mix of "disable cert validation" or "use certifi".

Eg https://github.com/aio-libs/aiohttp/issues/955

[tiran]

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960869 should help to solve a bunch of problems. Fedora / RHEL should have root CA certificates installed. I'll verify UBI mini tomorrow.

[ofek]

@tiran How far along is that prototype? 😄