Urllib3: MaxRetryError when redirecting from http to https through PoolManager()

[jayvdb]

As just an example http://en.wikipedia.org/wiki/Main_Page redirects to https://en.wikipedia.org/wiki/Main_Page , but requesting http://en.wikipedia.org/wiki/Main_Page via PoolManager fails.

Using urllib3.connection_from_url:

python -c "import ssl, urllib3; r = urllib3.connection_from_url('https://en.wikipedia.org/wiki/Main_Page').request('GET', 'http://en.wikipedia.org/wiki/Main_Page', assert_same_host=False); print(r.status);"
urllib3/connectionpool.py:791: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
200

Using urllib3.PoolManager().connection_from_url

python -c "import ssl, urllib3; r = urllib3.PoolManager().connection_from_url('http://en.wikipedia.org/wiki/Main_Page').request('GET', 'http://en.wikipedia.org/wiki/Main_Page', assert_same_host=False); print(r.status);"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "urllib3/request.py", line 69, in request
    **urlopen_kw)
  File "urllib3/request.py", line 90, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "urllib3/connectionpool.py", line 653, in urlopen
    release_conn=release_conn, **response_kw)
  File "urllib3/connectionpool.py", line 653, in urlopen
    release_conn=release_conn, **response_kw)
  File "urllib3/connectionpool.py", line 653, in urlopen
    release_conn=release_conn, **response_kw)
  File "urllib3/connectionpool.py", line 638, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "urllib3/util/retry.py", line 273, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='en.wikipedia.org', port=80): Max retries exceeded with url: https://en.wikipedia.org/wiki/Main_Page (Caused by ResponseError('too many redirects',))

Is that correct? If it is correct, it feels a little confusing seeing MaxRetryError here.

[haikuginger]

Okay, first a couple things to clear up: Your two commands are acting differently, not because of the difference between getting a ConnectionPool from urllib3.connection_from_url() versus SessionManager.connection_from_url(), but because you're doing different things:

urllib3.connection_from_url('https://en.wikipedia.org/wiki/Main_Page')

and

PoolManager().connection_from_url('http://en.wikipedia.org/wiki/Main_Page')

Note that in one, you specify HTTPS; in the other, you specify HTTP; this is the big difference here. One returns an HTTPSConnectionPool; the other returns an HTTPConnectionPool.

Now, as to why this is raising a MaxRetryError _when specifying HTTP for the connection_. The big reason is that you've set the assert_same_host=False argument, which means that urllib3 won't check the hostname, scheme, or port to verify that the URL you're requesting is actually something that can be accessed by the ConnectionPool. This makes the behavior really naive. For a demo, check this out:

>>> z = urllib3.connection_from_url('https://en.wikipedia.org/wiki/Main_Page')
>>> z.request('GET', 'NOTAREALSCHEME://en.wikipedia.org/wiki/Main_Page', assert_same_host=False)

It works! This is because we skip validation and pass the URL into HTTP.client, which actually sends the HTTP request over the existing connection. As the host, port, and connection scheme are handled on the connection level, HTTP.client ALSO doesn't validate that the scheme matches the connection - it just builds the request based on the method, hostname, and path:

GET /wiki/Main_Page HTTP/1.1
Host: en.wikipedia.org

This works fine when initializing on an HTTPSConnectionPool, because there's no redirect; our request to the http site silently goes to the https site, and no one's the wiser. On an HTTPConnectionPool, however, there is a redirect - to https://en.wikipedia.org/wiki/Main_Page. Because the assert_same_host=False flag is set, this is naively treated as a redirect to that page _on the plain HTTP site_, which, of course, results in a redirect loop.

As a result, this issue has two real answers.

First, in general, you should be using PoolManager.request() unless you have a strong reason otherwise. It'll handle cases like this properly, because it can draw from multiple connection pools and select between them based on both scheme and host, meaning that a redirect from http to https on the same host will work fine. Using PoolManager.connection_from_url gives you a copy of the relatively primitive ConnectionPool class, which only works for a single host/port/protocol combination.

Second, this is especially confusing in your case because you've included the assert_same_host=False flag. When using that flag, you need to be especially aware of the fact that it's not just skipping verification of the hostname; it's also leaving out things like HTTP vs HTTPS or port number. If you had left that out, you would have gotten an error like this instead of the confusing redirect loop:

urllib3.exceptions.HostChangedError: HTTPConnectionPool(host='en.wikipedia.org', port=80): Tried to open a foreign host with url: https://en.wikipedia.org/wiki/Main_Page

[jayvdb]

Thanks you.
My apologies for not noticing I was wrote https in the call to urllib3.connection_from_url, and it worked.

Using urllib3.PoolManager().request did the trick for me. :+1:

[shazow]

@haikuginger Yay thanks for helping out here!

[rjnshch]

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='us-south.monitoring.cloud.im.com', port=443): Max retries exceeded with url: /api/v3/dashboards?light=True (Caused by ResponseError('too many 500 error responses'))

Hi I am getting similar error what to do ?