Requests: Allow verify parameter to take a file-like object such as StringIO

[ronanpaixao]

Sometimes it is useful and more efficient to have SSL certificate data on memory, like when grabbing certificates from the Windows Certificate Store or some other non-file source.

As it is implemented now, requests neither grabs the certificates from the OS nor allows simple usage from a file-like object. Thus, the user is forced to store the certificates in a temporary file to allow usage, or the request fails when trying os.path.isdir (line 224 of adapters.py on requests version 2.12.4).

The unfortunate consequence is also that the dev has to keep the file as long as sessions, which is awkward specially for sessions which only request data occasionally. Otherwise, one has to grab all certificates again for each request. Both options also have the problem of leaving temp files if the process is terminated abruptly.

Example use case (on Windows 7, Python 3.6):

import ssl
import requests
from io import StringIO  # For Python 3.x
from tempfile import NamedTemporaryFile

# A site for which we have the CA in Windows Certificate Store (case of intranet on AD)
url = "https://some.secure.intranet.site"  
requests.get(url)  # Raises SSLError

# Grab certificates from Windows Certificate Store
# delete=False is required for some reason
tempcertfile = NamedTemporaryFile('w', encoding='utf8',delete=False)
memcertfile = StringIO()
context = ssl.create_default_context()
der_certs = context.get_ca_certs(binary_form=True)
pem_certs = [ssl.DER_cert_to_PEM_cert(der) for der in der_certs]
for pem in pem_certs:
    tempcertfile.write(pem + '\n')
    memcertfile.write(pem + '\n')
tempcertfile.seek(0)
memcertfile.seek(0)

requests.get(url, verify=tempcertfile.name)  # Works
requests.get(url, verify=memcertfile)  # Errors with a TypeError on adapters.py line 224

[Lukasa]

Requests is unlikely to support this on the verify kwarg because it interacts very poorly with the API for pointing to a directory or file. However, you can use a Transport Adapter to pass an SSLContext object to do the validation, and this object can almost certainly load the data from a string. See this explanation, which indicates how to provide an SSLContext.

[Sonique]

Also came here to see how can we provide string alike object. Very useful if you store certificates in database and do not want to use temp files.

[silverwind]

I think accepting a file-like object for verify should be re-considered. Adding a trusted certificate in-memory to requests is way harder than it should be.

I took a look at @Lukasa's SSLContext suggestion but it seems far too low-level. Certainly not something I would even consider using as it's bound to eventually break in future versions.

[nimish]

Having the ability to use an in-memory key and cert is very useful in oddball environments like cloud functions/lambda where otherwise one must do the named temp file dance.

Distinguishing between a str and file is fairly straightforward: isinstance(something, io.IOBase) in Python 3.

Using SSLContext is very un-Requests. It requires a deep understanding of the nature of TLS connections to do right, which is what Requests shouldn't require.

[danihodovic]

Having the ability to use an in-memory key and cert is very useful in oddball environments like cloud functions/lambda where otherwise one must do the named temp file dance.

Using SSLContext is very un-Requests. It requires a deep understanding of the nature of TLS connections to do right, which is what Requests shouldn't require.

:+1:

We are progressively moving towards cloud environments where programs are packaged as 12 factor apps. Many deployment environments promote passing config as environment variables (or reading from remote secret stores) over traditional file-based config.

Requests is unlikely to support this on the verify kwarg because it interacts very poorly with the API for pointing to a directory or file.

Would you be willing to reconsider this in 2020 if someone provided a patch?

Possible workaround by using tempfiles in https://stackoverflow.com/questions/30598950/python-requests-send-certificate-as-string/46570264

[danihodovic]

@Lukasa can you provide an example of using an adapter. I've tried the example in your blog, but it doesn't work.

import ssl
import requests
from requests.adapters import HTTPAdapter

class Adapter(HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        ctx = ssl.create_default_context(cadata=<data>)
        ctx.check_hostname = False
        ctx.verify_mode = ssl.CERT_NONE
        kwargs['ssl_context'] = ctx
        return super().init_poolmanager(*args, **kwargs)

s = requests.Session()
s.mount('https://foo', Adapter())

[nimish]

As far as I can tell this _requires_ changes to the stdlib ssl module or the use of pyOpenSSL's implementation of SSL contexts since that exposes the required functionality.

Unfortunately it appears that BPO-16487 has been outstanding for _8 years_ and counting...