Httpx: NetworkError with https proxies

Created on 8 Apr 2020 · 7Comments · Source: encode/httpx

Checklist

[x] The bug is reproducible against the latest release and/or master.
[x] There are no similar issues or pull requests to fix it yet.

Describe the bug

getting NetworkError when using a proxy while with requests works

To reproduce

import requests
import httpx
import trio

proxies = {
    "https": "https://128.199.241.229:44344" ### tested many other proxies
    }

req= requests.get('https://ifconfig.me/ip', proxies=proxies)
print('requests', req.text)

async def main():
    async with httpx.AsyncClient(proxies=proxies) as client:
        response = await client.get('https://ifconfig.me/ip')
        print('httpx', response.text)

trio.run(main)

Expected behavior

requests 128.199.241.229
httpx 128.199.241.229

Actual behavior

requests 128.199.241.229
raise NetworkError(exc) from exc
httpx._exceptions.NetworkError

full actual behavior

requests 128.199.241.229
Traceback (most recent call last):
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\trio\_ssl.py", line 466, in _retry
    ret = fn(*args)
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_utils.py", line 364, in as_network_error
    yield
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_backends\trio.py", line 106, in open_tcp_stream
    await stream.do_handshake()
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\trio\_ssl.py", line 636, in do_handshake
    await self._handshook.ensure(checkpoint=True)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\trio\_ssl.py", line 212, in ensure
    await self._afn(*self._args)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\trio\_ssl.py", line 606, in _do_handshake
    await self._retry(self._ssl_object.do_handshake, is_handshake=True)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\trio\_ssl.py", line 471, in _retry
    raise trio.BrokenResourceError from exc
trio.BrokenResourceError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 23, in <module>
    trio.run(main)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\trio\_core\_run.py", line 1804, in run
    raise runner.main_task_outcome.error
  File "test.py", line 20, in main
    response = await client.get('https://ifconfig.me/ip')
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1224, in get
    return await self.request(
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1085, in request
    response = await self.send(
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1106, in send
    response = await self.send_handling_redirects(
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1133, in send_handling_r
edirects
    response = await self.send_handling_auth(
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1170, in send_handling_a
uth
    response = await self.send_single_request(request, timeout)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1196, in send_single_req
uest
    response = await dispatcher.send(request, timeout=timeout)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_dispatch\proxy_http.py", line 201, in sen
d
    return await super().send(request=request, timeout=timeout)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_dispatch\connection_pool.py", line 149, i
n send
    connection = await self.acquire_connection(
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_dispatch\proxy_http.py", line 110, in acq
uire_connection
    return await self.tunnel_connection(origin, timeout)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_dispatch\proxy_http.py", line 121, in tun
nel_connection
    connection = await self.request_tunnel_proxy_connection(origin)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_dispatch\proxy_http.py", line 158, in req
uest_tunnel_proxy_connection
    proxy_response = await connection.send(proxy_request)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_dispatch\connection.py", line 42, in send
    self.connection = await self.connect(timeout=timeout)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_dispatch\connection.py", line 62, in conn
ect
    socket = await self.backend.open_tcp_stream(
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_backends\auto.py", line 33, in open_tcp_s
tream
    return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_backends\trio.py", line 106, in open_tcp_
stream
    await stream.do_handshake()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\user\AppData\Roaming\Python\Python38\site-packages\httpx\_utils.py", line 368, in as_network_error
    raise NetworkError(exc) from exc
httpx._exceptions.NetworkError

Environment

OS: Windows
Python version: 3.8.2
HTTPX version: 0.12.1
Async environment: trio
HTTP proxy: yes and the problem is about it :)

proxies question tls+pki

Source

mIcHyAmRaNe

All 7 comments

Hi!

Can you try using http:// in the proxy URL?

This might be a limitation of our current implementation — I had already noted this in https://github.com/encode/httpx/pull/259#issuecomment-531372673. The workaround seems to use an HTTP URL, so that the proxy is established over TCP and _then_ the connection is upgraded to TLS. See https://github.com/encode/httpx/pull/259#issuecomment-531449348.

florimondmanca on 8 Apr 2020

Ahhh, i did other tests and yeah using http:// inside "https": is the solution :

Failed (logic) : `proxy:http url:https`

proxies = {
    "http": "http://128.199.241.229:44344"
    }
### url: https://ifconfig.me/ip
### result: logic
requests: own ip
httpx: own ip

Success: `proxy:http url:http`

proxies = {
    "http": "http://128.199.241.229:44344"
    }
### url: http://ifconfig.me/ip
### result: 
requests: 128.199.241.229
httpx:  128.199.241.229

Success: `proxy:https with http scheme; url:https`

proxies = {
    "https": "http://128.199.241.229:44344"
    }
### url: https://ifconfig.me/ip
### result: worked: making the scheme of the proxy as `http` even in `"https":` is the solution
requests: 128.199.241.229
httpx: 128.199.241.229

logic result: `proxy:https with http scheme; url:http`

proxies = {
    "https": "http://128.199.241.229:44344"
    }
### url: http://ifconfig.me/ip
### result: logic
requests: own ip
httpx: own ip

Success: `proxy:http & https with http scheme; url:http`

proxies = {
    "http": "http://128.199.241.229:44344",
    "https": "http://128.199.241.229:44344"
    }
### url: http://ifconfig.me/ip
### result:
requests: 128.199.241.229
httpx:  128.199.241.229

Success: `proxy:http & https with http scheme; url:https`

proxies = {
    "http": "http://128.199.241.229:44344",
    "https": "http://128.199.241.229:44344"
    }
### url: https://ifconfig.me/ip
### result: 
requests: 128.199.241.229
httpx:  128.199.241.229

mIcHyAmRaNe on 8 Apr 2020

@mIcHyAmRaNe Thanks for trying it out :)

I actually find the HTTP Proxying docs a bit unclear on whether it's possible or not to use https:// in the proxy URL.

The intro snippet uses http:// for both the http and the https schemes.
…But then there's this part (emphasis mine):

By default httpx.Proxy will operate as a forwarding proxy for http:// requests and will establish a CONNECT TCP tunnel for https:// requests. This doesn't change regardless of the proxy url being http or https.

So it seems like you should be able to use https:// in the proxy URL if you stray away from the default behavior (i.e. HTTP CONNECT), and you specify TUNNEL_ONLY, like this:

proxy_location = "128.199.241.229:44344"

proxies = {
    "http": f"http://{proxy_location}",
    "https": httpx.Proxy(f"https://{proxy_location}", mode="TUNNEL_ONLY"),
}

with httpx.Client(proxies=proxies) as client:
    ...

Can you try that out, see if my understanding is correct? If that's what we're doing then I can try drafting a docs update, to make them a bit clearer in a more "problem solving/what config should you use if your proxy supports/doesn't support HTTPS" kind of style.

florimondmanca on 8 Apr 2020

even with mode="TUNNEL_ONLY" we need to use http:// inside "https": otherwise it doesn't work

proxy_location = "128.199.241.229:44344"  ### protocol: HTTPS

### http://{proxy_location}  ::: works
proxies = {
    "http": f"http://{proxy_location}",
    "https": httpx.Proxy(f"http://{proxy_location}", mode="TUNNEL_ONLY"),
}
### with http url: 128.199.241.229
### with https url: 128.199.241.229

### https://{proxy_location}  ::: failed
proxies = {
    "http": f"http://{proxy_location}",
    "https": httpx.Proxy(f"https://{proxy_location}", mode="TUNNEL_ONLY"),
}
### with http url: 128.199.241.229
### with https url: httpx._exceptions.NetworkError

mIcHyAmRaNe on 8 Apr 2020

Hmm.

So, the way tunneling works is a) we open a connection using the CONNECT method, then b) we upgrade the connection to TLS. Step a) is done using the proxy_url defined in proxies. So if it contains https://, then we'll issue a CONNECT https://....

One case when this would fail, I assume, is when the proxy doesn't support CONNECT via HTTPS (although it's able to make HTTPS requests to websites on our behalf, once we're connected).

I'm very bad at decrypting the meaning of OpenSSL error messages, but maybe this is what it means…

ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)

florimondmanca on 8 Apr 2020

Is it possible the proxy does not expect a TLS handshake on that port?

I've seen that WRONG_VERSION_NUMBER error when trying to make an HTTPS request to a host on port 80.

yeraydiazdiaz on 9 Apr 2020

👍1

I'm going to temptatively close this as we've already merged some changes regarding proxies in httpcore and here.

If this issue persists in master please open a new ticket.

yeraydiazdiaz on 6 May 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

0.13: some `httpcore` exceptions are missing from top level package

iwoloschin · 4Comments

Keepalive connections aren't released when closing the ConnectionPool

edocod1 · 3Comments

Make debugging TLS errors easier for users

sethmlarson · 5Comments

Test case `test_write_timeout` fails

kde713 · 3Comments

Passing AsyncClient from an 'async with' context behaves unexpectedly

StevenBorg · 3Comments

Httpx: NetworkError with https proxies

Checklist

Describe the bug

To reproduce

Expected behavior

Actual behavior

Environment

All 7 comments

Failed (logic) : proxy:http url:https

Success: proxy:http url:http

Success: proxy:https with http scheme; url:https

logic result: proxy:https with http scheme; url:http

Success: proxy:http & https with http scheme; url:http

Success: proxy:http & https with http scheme; url:https

Related issues

Failed (logic) : `proxy:http url:https`

Success: `proxy:http url:http`

Success: `proxy:https with http scheme; url:https`

logic result: `proxy:https with http scheme; url:http`

Success: `proxy:http & https with http scheme; url:http`

Success: `proxy:http & https with http scheme; url:https`