Flurl: Proxy per request

[ivanovevgeny]

It would be nice to add the ability to specify a proxy server.

[tmenier]

You can do this with a custom factory:

```c#
using Flurl.Http.Configuration;

public class ProxyHttpClientFactory : DefaultHttpClientFactory
{
private string _address;

public ProxyHttpClientFactory(string address) {
    _address = address;
}

public override HttpMessageHandler CreateMessageHandler() {
    return new HttpClientHandler {
        Proxy = new WebProxy(_address),
        UseProxy = true
    };
}

}

To register it globally on startup:

```c#
FlurlHttp.Configure(settings => {
    settings.HttpClientFactory = new ProxyHttpClientFactory("http://myproxyserver");
});

[ivanovevgeny]

Thanks!

[matteocontrini]

Hi, what if I wanted to set a proxy for a specific HTTP request?

[CreatedAutomated]

The custom proxy factory is nice, however, it would be awesome if you added proxy support that was much easier to use rather than creating a custom factory. I know httpclient does not allow proxy changing once the client has been created, however! There is actually a way of doing this. I've seen it done before and lost the code for it. Maybe add a "WithProxy(proxy)" feature that automatically sets the proxy with each request. It would be pretty awesome

[tmenier]

@matteocontrini @CreatedAutomated Can you describe the use case for why you want to proxy some calls and not others?

[CreatedAutomated]

@matteocontrini @CreatedAutomated Can you describe the use case for why you want to proxy some calls and not others?

If a proxy is dead and does not respond for a client instance, and we want to change it to a working proxy, it means the client needs disposed and a new client needs created just to use another proxy. It's more efficient just to keep one client instance, and be able to dynamically change the proxy.

[matteocontrini]

My case is similar. I have a number of proxies, and I want each request to use one proxy, randomly chosen from that pool of proxies. With the current way HttpClient and Flurl work, this seems impossible, and it's very limiting. In many other languages setting the proxy for a request is a matter of passing a parameter to a method!

[tmenier]

@CreatedAutomated

I know httpclient does not allow proxy changing once the client has been created

HttpClientHandler.Proxy is a read/write property, so I don't see why not, unless there's some logic that throws if you attempt to do it. Can you provide a reference on that?

[CreatedAutomated]

HttpClientHandler

The only issue is with this, once you set it, you cannot change it. You need to create a new httpclient instance. But there is a way to change it on the fly, I forgot how. If I find out, I will share the solution

[tmenier]

@matteocontrini @CreatedAutomated Yeah, I think this is a limitation of the HttpClient stack. I can't think of a way Flurl could work around the fact that you can't set a different proxy per request on a shard instance in a thread-safe way. All I can think of maintaining a pool of _clients_ (one per proxy) and switching between those as needed. You should actually be able to implement random or round-robin proxies with a custom FlurlClientFactory so it's completely abstracted away from requests. The details are a little beyond the scope of this issue but if you need help with an implementation feel free to ask on Stack Overflow, I typically answer everything tagged flurl.

[matteocontrini]

For anyone looking for some code beyond the discussion, here's a sample implementation of a FlurlClientFactory that creates and caches a client per proxy, for requests that are supposed to be proxied, and a client per host for all the other requests.

I haven't tested it thoroughly, but it should work. The disadvantage to this approach is that you need to know in advance a strategy for deciding whether the request should go through a proxy, based on its URL. This means that you can't actually choose whether to enable the proxy or not on a per-request basis.

EDIT: this code has a side effect that could create unexpected results in some cases. See #374 for details and an alternative solution

public class FlurlClientFactory : IFlurlClientFactory
{
    private readonly ConcurrentDictionary<string, IFlurlClient> clients =
        new ConcurrentDictionary<string, IFlurlClient>();

    public IFlurlClient Get(Url url)
    {
        if (url == null)
        {
            throw new ArgumentNullException(nameof(url));
        }

        if (ShouldGoThroughProxy(url))
        {
            string randomProxyUrl = ChooseRandomProxy();

            return ProxiedClientFromCache(randomProxyUrl);
        }
        else
        {
            return PerHostClientFromCache(url);
        }
    }

    private bool ShouldGoThroughProxy(Url url)
    {
        throw new NotImplementedException();
    }

    private string ChooseRandomProxy()
    {
        throw new NotImplementedException();
    }

    private IFlurlClient PerHostClientFromCache(Url url)
    {
        return clients.AddOrUpdate(
            key: url.ToUri().Host,
            addValueFactory: u => {
                return new FlurlClient();
            },
            updateValueFactory: (u, client) => {
                return client.IsDisposed ? new FlurlClient() : client;
            }
        );
    }

    private IFlurlClient ProxiedClientFromCache(string proxyUrl)
    {
        return clients.AddOrUpdate(
            key: proxyUrl,
            addValueFactory: u => {
                return CreateProxiedClient(proxyUrl);
            },
            updateValueFactory: (u, client) => {
                return client.IsDisposed ? CreateProxiedClient(proxyUrl) : client;
            }
        );
    }

    private IFlurlClient CreateProxiedClient(string proxyUrl)
    {
        HttpMessageHandler handler = new HttpClientHandler()
        {
            Proxy = new WebProxy(proxyUrl),
            UseProxy = true
        };

        HttpClient client = new HttpClient(handler);

        return new FlurlClient(client);
    }

    /// <summary>
    /// Disposes all cached IFlurlClient instances and clears the cache.
    /// </summary>
    public void Dispose()
    {
        foreach (var kv in clients)
        {
            if (!kv.Value.IsDisposed)
                kv.Value.Dispose();
        }

        clients.Clear();
    }

}

[tmenier]

I'm going to give some fresh consideration to WithProxy. Based on some planned changes for 3.0, it'll be possible to define a strategy for choosing an HttpClient instance based on other request criteria besides just the URL, which should make the implementation a lot easier.

@CreatedAutomated

I know httpclient does not allow proxy changing once the client has been created, however! There is actually a way of doing this. I've seen it done before and lost the code for it. If I find out, I will share the solution
...
once you set it, you cannot change it. You need to create a new httpclient instance. But there is a way to change it on the fly, I forgot how.

I don't want to hold you to something you said over a year ago, and you're sort of saying "you can't but you can" so I might be misunderstanding anyway, but if you're suggesting you can somehow change the proxy on an existing instance of HttpClient, I believe (based on some extensive googling) that this is false. Earlier in this thread I was uncertain based on the fact that it's a read/write property, but I confirmed it will throw an exception if you try to change it after the first call is made.

So I think we'll be stuck with an HttpClient instance per proxy under the hood, but Flurl can abstract that. If you're worried about the famous socket exhaustion problem caused by all those instances, here's a crucial question: Isn't a socket by definition specific to one address? If so, wouldn't 1000 proxies require 1000 new sockets to be opened anyway, regardless of whether you could somehow do it with a single HttpClient instance? I'm not an expert at the low-level sockets stuff, so I'm honestly not 100% certain. Hoping someone here might be able to confirm that.

[CreatedAutomated]

This is not false. I had code to do this in the past. I sourced it from
another developer on skype. I will need to try dig the code up somehow. I
will post it for you when I find it. Its 100% possible

On Friday, January 17, 2020, Todd Menier [e-mail] wrote:

Reopened #228 https://github.com/tmenier/Flurl/issues/228.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmenier/Flurl/issues/228?email_source=notifications&email_token=AKIZM2KNH5BGJXYICDKETATQ6ILQTA5CNFSM4D5S4KWKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOWBS7H4Y#event-2959471603,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKIZM2LEV52STOQ2XRUM7UTQ6ILQTANCNFSM4D5S4KWA
.

[developervariety]

Only true way of changing the proxy while using the same HttpClient is something like Titanium-Web-Proxy. Although if you want a proxy per specific request, it's practically impossible with HttpClient honestly (kinda sucks).

[CreatedAutomated]

I forgot about this. When I'm at my desk I will search my desktop. The code
I had was pretty low level. Httpclients is just a wrapper for the
HttpWebRequest class. There is a way you can actually modify the
httpwebrequest proxy property. Once I find the code I'm gonna share it here.

On Monday, March 23, 2020, Justin Lopez [e-mail] wrote:

Only true way of changing the proxy while using the same HttpClient is
something like Titanium-Web-Proxy
https://github.com/justcoding121/Titanium-Web-Proxy. Although if you
want a proxy per specific request, it's practically impossible with
HttpClient honestly (kinda sucks).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmenier/Flurl/issues/228#issuecomment-602377350, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AKIZM2IU4NTH5E3DQA3SWCLRI3P33ANCNFSM4D5S4KWA
.

[shravan2x]

@tmenier

Can you describe the use case for why you want to proxy some calls and not others?

Wanted to add another use-case here - I have an application that runs a number of separate tasks/operations/services in the same program i.e. a "monolith". Here, different proxies are used by separate tasks for their own use-cases eg. some for IP load balancing, some to circumvent geo-restrictions, some for corporate authentication, etc.

However, this doesn't require setting the proxy per request, setting it per client works fine (and I believe that's already possible via new FlurlClient { Settings = new ClientFlurlHttpSettings { HttpClientFactory = ... } }. Creating a custom HttpClientFactory is inconvenient though, as explained in @CreatedAutomated's comment.

[NogginBox]

My usecase is I have created an ApiWrapper and would like to offer users of the wrapper a way to disable the proxy, but I don't want my library to change the global behaviour of Flurl if the consumer of my wrapper is using it for something else.

[tmenier]

@NogginBox I'll keep that use case in mind. Due to contraints of the HttpClient stack, there would still need to be 2 clients at play here, but if you're using global configuration (i.e. FlurlHttp.Configure) it'll be easy to keep all your other settings the same for both.

[lenniebriscoe]

@tmenier do you have a timeline when this feature will be implemented please?