Httpx: Connecting to HTTPS websites with auth proxy

[adam-68]

I have been trying to make this simple code work:

        proxies = httpx.Proxy(
            url="https://username:password@hostname:post",
            mode="TUNNEL_ONLY"
        )

        async with httpx.AsyncClient(proxies=proxies, verify="client.pem") as client:
            response = await client.get("https://httpbin.org/ip")
            print(response.text)

Firstly, i tried to connect via auth proxy without the verify paramter, but it failed and i was getting this error:

Process Process-1:
Traceback (most recent call last):
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Adam\Desktop\sizeer\test2.py", line 67, in what
    asyncio.get_event_loop().run_until_complete(new.main(proxy))
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "C:\Users\Adam\Desktop\sizeer\test2.py", line 59, in main
    response = await client.get("https://httpbin.org/ip")
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1408, in get
    return await self.request(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1241, in request
    response = await self.send(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1272, in send
    response = await self._send_handling_redirects(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1301, in _send_handling_redirects
    response = await self._send_handling_auth(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1338, in _send_handling_auth
    response = await self._send_single_request(request, timeout)
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1369, in _send_single_request
    ) = await transport.request(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_async\http_proxy.py", line 113, in request
    return await self._tunnel_request(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_async\http_proxy.py", line 230, in _tunnel_request
    await proxy_connection.start_tls(host, timeout)
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_async\connection.py", line 146, in start_tls
    self.socket = await self.connection.start_tls(hostname, timeout)
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_async\http11.py", line 84, in start_tls
    self.socket = await self.socket.start_tls(hostname, self.ssl_context, timeout)
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_backends\asyncio.py", line 108, in start_tls
    transport = await asyncio.wait_for(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\asyncio\tasks.py", line 483, in wait_for
    return fut.result()
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\asyncio\base_events.py", line 1200, in start_tls
    await waiter
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\asyncio\sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\asyncio\sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)

Process finished with exit code 0

Afterwards, i tried generating Ca certs with trustme-cli, but it also failed and i get this error every time i try to connect to website via proxy:

Traceback (most recent call last):
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Adam\Desktop\sizeer\test2.py", line 67, in what
    asyncio.get_event_loop().run_until_complete(new.main(proxy))
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "C:\Users\Adam\Desktop\sizeer\test2.py", line 59, in main
    response = await client.get("http://httpbin.org/ip")
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1408, in get
    return await self.request(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1241, in request
    response = await self.send(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1272, in send
    response = await self._send_handling_redirects(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1301, in _send_handling_redirects
    response = await self._send_handling_auth(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1338, in _send_handling_auth
    response = await self._send_single_request(request, timeout)
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1363, in _send_single_request
    (
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_exceptions.py", line 359, in map_exceptions
    raise mapped_exc(message, **kwargs) from None  # type: ignore
httpx._exceptions.ConnectTimeout

Process finished with exit code 0

I do not own the proxy server and i am paying to use the ip, so the only thing i have is this scheme hostname:port:username:password. As you mentioned in your docs, i can change ca_certs, but only on local server (Server of my proxy is not local, so i believe that i cannot change the certs manually. Correct me, if i am wrong). The project i am writing needs to make numerous requests at one time and each with different proxy ip. So can you tell me is there any option that will enable me to use your library in order to send requests with auth proxy to HTTPS and HTTP websites without timeout error or wrong version number error from ssl? By far i checked the proxy and connected via curl and Requests library. Both ways of changing ip while sending request worked, so i wonder if there's something wrong with httpx or my code. Thanks for help.

[florimondmanca]

Hi,

Can you confirm whether using FORWARD_ONLY instead of TUNNEL_ONLY works?

I remember that for some reason TUNNEL_ONLY doesn't seem to work for HTTPS proxies, which is definitely worth looking into.

[adam-68]

After changing to FORWARD_ONLY and url="http://username:passw@host:port" i got new error:

Process Process-1:
Traceback (most recent call last):
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Adam\Desktop\sizeer\test2.py", line 67, in what
    asyncio.get_event_loop().run_until_complete(new.main(proxy))
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "C:\Users\Adam\Desktop\sizeer\test2.py", line 59, in main
    response = await client.get("https://httpbin.org/ip")
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1408, in get
    return await self.request(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1241, in request
    response = await self.send(
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_client.py", line 1278, in send
    await response.aread()
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_models.py", line 964, in aread
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_models.py", line 964, in <listcomp>
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_models.py", line 975, in aiter_bytes
    async for chunk in self.aiter_raw():
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpx\_models.py", line 1008, in aiter_raw
    async for part in self._raw_stream:
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_async\connection_pool.py", line 50, in __aiter__
    async for chunk in self.stream:
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_bytestreams.py", line 72, in __aiter__
    async for chunk in self._aiterator:
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_async\http11.py", line 143, in _receive_response_data
    event = await self._receive_event(timeout)
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_async\http11.py", line 160, in _receive_event
    data = await self.socket.read(self.READ_NUM_BYTES, timeout)
  File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\httpcore\_backends\asyncio.py", line 138, in read
    raise ReadError("Server disconnected while attempting read")
httpcore._exceptions.ReadError: Server disconnected while attempting read

Process finished with exit code 0

While using
url="https://username:passw@host:port"
i got the same errors as before.

[adam-68]

What's more while using this command to setup HTTPS proxy scp server.key server.pem <USER>@<SERVER_IP>:/tmp/ i cannot connect to the server and i believe that's because it is not local server, so i cannot send ca_certs generated before.

[alissonlauffer]

Looks like this issue is only affecting 0.14.x releases. When I tried 0.13.x releases, proxy auth worked just fine.

[tomchristie]

@alissonlauffer I really want to dig into this - we can help if we're able to replicate the issue. What proxy are you using? Are you able to replicate with proxy.py or Charles Proxy? What's the absolutely simplest possible example you're able to replicate this with?

[alissonlauffer]

@tomchristie Using proxy.py worked without any issue. But when using the HTTP proxy I have here, no. I can't share the proxy publicly here, so how can I send that proxy to you for further investigation?

Code I'm using:

from httpx import Client

with Client(proxies="http://<my_proxy_here>") as s:
  r = s.get("https://httpbin.org/ip")
  print(r.text)

[tomchristie]

Okay, thanks. So two things...

1. What's the exact traceback you're seeing from your example above?
2. If try exactly the same thing but instead making the request to http://httpbin.org/ip what happens?...

from httpx import Client

with Client(proxies="http://<my_proxy_here>") as s:
  r = s.get("http://httpbin.org/ip")
  print(r.text)

[tomchristie]

It might also be worth trying installing httpcore master...

$ pip install git+git://github.com/encode/httpcore.git

I'm wondering if the issue could be related to https://github.com/encode/httpcore/pull/154 which is fixed in master, but not yet released.

Thanks so much for your time. ✨

[alissonlauffer]

Traceback:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 804, in get
    return self.request(
  File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 640, in request
    return self.send(
  File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 670, in send
    response = self._send_handling_redirects(
  File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 699, in _send_handling_redirects
    response = self._send_handling_auth(
  File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 736, in _send_handling_auth
    response = self._send_single_request(request, timeout)
  File "/usr/lib/python3.8/site-packages/httpx/_client.py", line 759, in _send_single_request
    (
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3.8/site-packages/httpx/_exceptions.py", line 359, in map_exceptions
    raise mapped_exc(message, **kwargs) from None  # type: ignore
httpx._exceptions.ConnectError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1123)

On http sites, the request is successful and no error occurs.

I also tried both httpcore and httpx from master, but the same error occurs.

[tomchristie]

Okay, if you're able to help me with the following, that'd be amazing...

# Note: Make sure to change `b"<my_proxy_host>"`, and make sure to keep it as a byte string.
# Also make sure to change `my_proxy_port`, and make sure to keep it as an int.
transport = httpcore.SyncHTTPProxy(proxy_url=(b"http", b"<my_proxy_host>", my_proxy_port, b"/"))
r = transport.request(b"GET", (b"https", b"httpbin.org", None, b"/ip"), headers=[(b"Host", b"httpbin.org")])
print(r)

If you're able to test that against httpcore 0.10.1 and 0.9.1, then we'll really be narrowing things down.

Thank you so much for your time.
Seems like a really big bug we've fallen into, so it's really appreciated.

[alissonlauffer]

The above code does not work with proxy auth.

[adam-68]

I confirm that this code works after downgrading httpx to 0.13.x

from httpx import Client

with Client(proxies="http://<my_proxy_here>") as s:
  r = s.get("https://httpbin.org/ip")
  print(r.text)

However, this code:

# Note: Make sure to change `b"<my_proxy_host>"`, and make sure to keep it as a byte string.
# Also make sure to change `my_proxy_port`, and make sure to keep it as an int.
transport = httpcore.SyncHTTPProxy(proxy_url=(b"http", b"<my_proxy_host>", my_proxy_port, b"/"))
r = transport.request(b"GET", (b"https", b"httpbin.org", None, b"/ip"), headers=[(b"Host", b"httpbin.org")])
print(r)

lacks authentication for auth proxy.

[tomchristie]

Okay, with proxy auth, I think it ought to be this...

import httpcore
from base64 import b64encode


def build_proxy_headers(username, password):
    userpass = (username.encode("utf-8"), password.encode("utf-8"))
    token = b64encode(b":".join(userpass))
    return [(b"Proxy-Authorization", b"Basic " + token)]

proxy_url=(b"http", b"<my_proxy_host>", <my_proxy_port>, b"/")     # <my_proxy_host> as `bytes`, <my_proxy_port> as `int`
proxy_headers = build_proxy_headers(<my username>, <my_password>)  # <my_username>, <my_password> as `str`
transport = httpcore.SyncHTTPProxy(proxy_url=proxy_url, proxy_headers=proxy_headers)
r = transport.request(b"GET", (b"https", b"httpbin.org", None, b"/ip"), headers=[(b"Host", b"httpbin.org")])
print(r)

Being able to test that out against 0.10.1 / 0.9.1 would be wonderful.
I'll try to confirm this side, that I've got this all exactly correct.

[tomchristie]

If either of you are able to hop onto https://gitter.im/encode/community that might help us work this through quickly?

[tomchristie]

Righty, I've adding the required imports there, fixed a trivial syntax error, and confirmed against proxy.py that's the correct snippet for making the request directly at the transport API layer, so the above should be good for helping us find out exactly which httpcore commit introduced the regression.

[tomchristie]

Appears to have been due to https://github.com/encode/httpcore/pull/154.
We've got an upcoming httpcore release in https://github.com/encode/httpcore/pull/165 that will resolve this.

[tomchristie]

Now resolved in httpcore 0.10.2.
pip install -U httpcore should do the job.