Urllib3: Accessing HTTPS sites through proxy

[glaukon-ariston]

Why is it that urllib3 doesn't allow proxy access to HTTPS sites? See poolmanager.py#L416. This limitation seems to be also reflected in the requests library as well.

poolmanager.py#L416

    def connection_from_host(self, host, port=None, scheme='http', pool_kwargs=None):
        if scheme == "https":
            return super(ProxyManager, self).connection_from_host(
                host, port, scheme, pool_kwargs=pool_kwargs)

        return super(ProxyManager, self).connection_from_host(
            self.proxy.host, self.proxy.port, self.proxy.scheme, pool_kwargs=pool_kwargs)

requests/adapters.py#L346

    def request_url(self, request, proxies):
        """Obtain the url to use when making the final request.
        If the message is being sent through a HTTP proxy, the full URL has to
        be used. Otherwise, we should only use the path portion of the URL.
        This should not be called from user code, and is only exposed for use
        when subclassing the
        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
        :param proxies: A dictionary of schemes or schemes and hosts to proxy URLs.
        :rtype: str
        """
        proxy = select_proxy(request.url, proxies)
        scheme = urlparse(request.url).scheme

        is_proxied_http_request = (proxy and scheme != 'https')
        using_socks_proxy = False
        if proxy:
            proxy_scheme = urlparse(proxy).scheme.lower()
            using_socks_proxy = proxy_scheme.startswith('socks')

        url = request.path_url
        if is_proxied_http_request and not using_socks_proxy:
            url = urldefragauth(request.url)

        return url

[sigmavirus24]

It's a limitation of the fact that urllib3 uses httplib/http.request under the covers and the standard library is terrible at supporting this particular use-case. Luckily urllib3 is being slowly rewritten.

[glaukon-ariston]

Is there a roadmap of what needs to be done to support HTTPS over proxy in urllib3? How much work is it? What has already been done so far? Is there a single point in the httplib/http.request that is the root of the problem? Is there a reference implementation you can recommend. I would appreciate any specific details you can provide.

[OmriBaso]

Any updates on this issue? any work arounds?

[daqieq]

@J3wker - we used this repo below for our NTLM auth based proxy and it works for HTTPS sites:

https://github.com/dopstar/requests-ntlm2

I think requests supported Basic auth for HTTPS, but I'm not sure if there's a Kerberos auth solution yet. Let us know if you find one!

[pquentin]

@J3wker @glaukon-ariston Thanks to the work of @jalopezsilva, the next release will support HTTPS proxies fully, even offering TLS-in-TLS when both the proxy and the endpoint support HTTPS. https://urllib3.readthedocs.io/en/latest/advanced-usage.html#proxies

Is this what you asked for or is this issue about something else?