Httpx: Implement HTTP Forwarding Proxy

[sethmlarson]

Related: #36

[tomchristie]

What's the difference between this and #201?

[sethmlarson]

Forwarding is sending a request to a proxy and that proxy makes the request on your behalf and then gives you back the response.

Tunnel/CONNECT is asking the proxy for a TCP tunnel via CONNECT and then proceeding with HTTP(S) as normal through that tunnel.

The second one makes TLS possible as the proxy can't prove it is the website you're trying to connect to without some MITM stuff happening there but forwarding is a valid proxying method for HTTP.

[tomchristie]

So in the requests API, where we have proxy={http=..., https=...} then the http value should point at a simple forwarding proxy address, and the https value should point at a Tunnel/Connect proxy address? (Or have I got that wrong?)

[GnubiBORED]

@sethmlarson in my case i need to be able to add extra headers to those CONNECT/tunnel when proxy is used will be possible?

[sethmlarson]

@GnubiBORED Yes that will be possible via configuring your proxy like so (or something similar, API isn't nailed down yet):

import httpx

proxy = httpx.HTTPProxy(
    "http://127.0.0.1:8080",
    proxy_headers={"Proxy-Authorization": "..."}
)
client = httpx.Client(
    proxies={"http": proxy, "https": proxy}
)

For HTTP (forwarding) the headers are sent with the request.
For HTTPS (tunnel) the headers are sent with the CONNECT request but not the requests made after tunneling. Those are controlled by headers=....

Does that handle your use-case? :)

[GnubiBORED]

yep it handle my case totally if those headers are JUST SENT IN THE CONNECT and not in post/get, i mean if just are used when CONNECT requests happens, only question more is thath if i can order headers passing and collections.OrderectDict like i do in requests. but i think its possible and works, correct me if im wrong

[sethmlarson]

@GnubiBORED: To answer your questions:

For HTTP Forwarding proxies (http://...) the proxy headers are joined to the request headers that are sent to the proxy. It's up to the proxy to strip them (Proxy-Authorization, etc)

For HTTP CONNECT proxies (https://...) the proxy headers are only sent for the CONNECT request. The actual request won't include any of the proxy headers.

And yes if you pass an iterable of tuples order is preserved.

[GnubiBORED]

any update in proxy implementation in httpx?

[sethmlarson]

I'm working on combining the proxy implementations. I have them working I believe the issue now is writing tests for them. Will push what I have today and continue writing tests.

[GnubiBORED]

you will comment it on HTTP(S) Proxy Support (TODO) in readme once its done?

[sethmlarson]

Yeah I'll be removing that and in the docs. :)

[adrianmeraz]

Awesome work @sethmlarson , this has been a long-standing reason why I couldn't use HttpX just yet.

[sethmlarson]

@adrianmeraz I'm so glad! :) Have you tested the PR to see if it works properly for your user-case? Would love more eyes on it.

[adrianmeraz]

@sethmlarson I'll be sure to test it out ASAP!

[GnubiBORED]

im new to python/coding but its already testeable? if yes i would like to test these feature even if dosnt work good or whatever

[sethmlarson]

Yeah @GnubiBORED you would have to pull this exact branch though. Once it lands in master I'll be tagging a new release so everyone waiting for proxies can try the new feature. :)

[GnubiBORED]

@sethmlarson i downloaded these
https://github.com/sethmlarson/http3/tree/proxy-http-forwarding
once installed i try to run the same code you give me and seems i did something bad?

Traceback (most recent call last):
  File "C:/Users/localhost/PycharmProjects/TribalW_Test_Env/Basic/network/test4.py", line 3, in <module>
    proxy = httpx.HTTPProxy(
AttributeError: module 'httpx' has no attribute 'HTTPProxy'

Process finished with exit code 1

[GnubiBORED]

also with
client = httpx.Client( proxies={"http": proxy, "https": proxy} )
i get
```
proxies={"http": proxy, "https": proxy}
TypeError: __init__() got an unexpected keyword argument 'proxies'

Process finished with exit code 1
```

[sethmlarson]

@GnubiBORED @adrianmeraz Proxies are in the latest release! :tada: Could you give them a try?