Httpx: (improvement) Provide an easy way to change proxy in an instantiated client

[098799]

Currently, to change proxy in an instantiated client, one needs to substitute two objects:

• client.proxies[...].pool.proxy needs to be substituted with a new instance of urllib3.util.Url,
• client.proxies[...].pool.connection_pool_kw['_proxy'] needs to be changed as well.

This seems very far from ideal. This process is much easier in requests, and could be made much easier here as well. e.g. with a bit of property magic on setting client.proxies.

[tomchristie]

Yup probably valid.

I'm not totally clear what use cases there are for mutating a Client instance rather than configuring it once on setup(?), but I think for user happiness it's probably worth doing anyway.

[098799]

Thank you so much for the quick response.

Session (or Client) is a container for much more than just the proxy configuration. You may want to use the same cookies and headers, but rotate proxies. I know that modern proxy providers let you configure rotation on their side, but there may be use cases where you want to switch proxy providers as well.

Let me use this opportunity to thank you for all the awesome work! I'm just discovering encode stack but I'm already in love.

[florimondmanca]

@098799 A quick note that the current solution you pointed at relies on urllib3 under the hood, so that's likely _not_ going to be how we provide proxy rotation support eventually. :-) But I agree it's an acceptable feature to have.

[yeraydiazdiaz]

the current solution you pointed at relies on urllib3 under the hood

Do we have a link for that solution? I can't find any references on the proxy sections of urllib3 or requests.

[098799]

the current solution you pointed at relies on urllib3 under the hood

Do we have a link for that solution? I can't find any references on the proxy sections of urllib3 or requests.

I think he meant the solution I've sketched in the OP. Which works, but it's not pretty.

[yeraydiazdiaz]

So presumably you have a solution that is less complex using requests, could you share it?

[098799]

requests lets you change proxies on session after it's created:

import requests

session = requests.Session()

session.get('https://api6.ipify.org?format=json').text  # local IP
session.proxies = {'https': 'https://foo.com:8000'}
session.get('https://api6.ipify.org?format=json').text  # different IP

as well as pass it while making a request, which overrides the one from the session:

session.get('https://api6.ipify.org?format=json', proxies={'https': 'https://foo.com:8000'}).text

[tomchristie]

I'm going to suggest we close this off.

Replacing one transport with another is an operation that ought to have clearly defined semantics wrt. closing the connection pool. Allowing users to change the proxies by mutating a client instance makes the expected behaviour there really unclear.

Sometimes there are good reasons to enforce more explicit patterns in particular areas - this looks like one of them to me.

[098799]

That's a pity. Remember that by this design choice you're unnecessarily coupling proxy setup with everything else related to a session (cookies, headers etc.) and making it almost impossible to use the client context manager for tasks that require switching proxies.

[tomchristie]

I feel I need to clarify here. It's not an option for us. Supporting client.proxies = ... would necessarily introduce a resource leak.

In sync-land you can ensure that the property setter also closes any long lived proxy connections. In async-land closing the connections is an async operation, and needs a corresponding await <...> call, which can't happen with a property setter.

We could feasibly allow something like allowing .proxies = ... to be set only if the client is closed, but that's a bit awkward. Perhaps(?) there's something worth discussing there, if someone's up for putting in the work.

Keep in mind that you can also mount different proxies to different host prefixes, which means that you can use a client with multiple different proxies installed.