Runtime: HTTPS Proxy support feature requests

[matteo-prosperi]

Hello,
As the support of HTTPS proxies is becoming more common, we need to execute HTTP and HTTPS requests through an HTTPS proxy.

To be clear, this is not about executing an HTTPS request through an HTTP proxy (which works well), it is about connecting to the proxy itself using HTTPS.

As an additional requirement, we are also supposed to authenticate the client using client certificates.

When I try to use a proxy created from an https URI
c# HttpClientHandler handler = new HttpClientHandler(); handler.Proxy = new WebProxy("https://127.0.0.1"); HttpClient client = new HttpClient(handler);
I receive an Only the 'http' scheme is allowed for proxies. NotSupportedException.

I found some indication at https://github.com/dotnet/corefx/issues/20435 that this is not supported at all, so I am reopening this as a feature request.

Thanks

[richlander]

This looks like a good feature for .NET 5. I haven't heard enough requests about this scenario to warrant adding it earlier.

[scalablecory]

Can you give some use cases for HTTPS proxies?

[richlander]

Wouldn't it be the same as an HTTP proxy but with TLS? An HTTPS -> HTTP transition through an HTTP proxy would seem to undermine the value proposition of HTTPS. I cannot imagine that it is any more complicated than that.

[scalablecory]

An HTTPS -> HTTP transition through an HTTP proxy would seem to undermine the value proposition of HTTPS.

HTTPS requests have end-to-end encryption, regardless of if they go through a proxy, or if that proxy has encryption -- it is protected from the proxy as well.

A HTTPS proxy does not really add security, it merely shifts it: it makes it so that instead of your ISP being able to track what IPs you connect to, the proxy can.

[scalablecory]

Maybe @GrabYourPitchforks has some thoughts.

[matteo-prosperi]

We have been using https://github.com/fancy-owl/docker-mtls-https-proxy to test it.

CURL supports it by doing

curl --proxy-cert ./client.pem --proxy-cacert ./ca.crt --proxy https://some.uri.com https://i.want.to.see.this.com

We have customers who are setting up HTTPS proxies in their networks and requiring clients to authenticate against the proxy with client certificates.

[karelz]

Related to dotnet/runtime#17740 (Socks4/5 proxies)

[richlander]

@scalablecory -- that doesn't make sense to me. If you used an HTTP proxy, I assume any traffic after that proxy was "in the clear". That was my point, and avoiding that was my intended use case for an HTTPS proxy. If the proxy was one you trusted, then this model would seem to be fine. Right?

My mental model is this:

• There are scenarios for proxies.
• There are scenarios for HTTPS.
• This request targets the points where those scenarios overlap.

Again, is this any more complicated that that? I assume if you don't have an HTTPS proxy, then you violate the HTTPS-oriented requirements? I understand your point about proxy now being a trust point. It seems similar to using a third-party VPN. If the proxy isn't third party, then maybe no trust issue. Fair?

This seems to align with what @matteo-prosperi is saying.

[davidsh]

A HTTPS proxy does not really add security, it merely shifts it: it makes it so that instead of your ISP being able to track what IPs you connect to, the proxy can.

Besides consumer related scenarios (i.e. ISP's), we have heard from enterprise customers over the years wanting "TLS everywhere" which means that communicating with a proxy on the LAN should also use HTTPS. Currently even with communicating with a destination server over HTTPS, the initial 'CONNECT' verb to the proxy is unencrypted (HTTP right now). And that means any headers that get sent along (include 'Proxy-Authorization' headers) would be visible.

So, the scenario itself is reasonable. We would just have to prioritize this work and get a design/cost for it.

[richlander]

There has been talk about DNS with TLS for the same reason.

[davidsh]

There has been talk about DNS with TLS for the same reason.

There are significant disagreements about the usefulness of some of these related protocols (like DNS over HTTPS):
https://www.zdnet.com/article/dns-over-https-causes-more-problems-than-it-solves-experts-say/

But pure DNS over TLS (DoT) or using IPSEC would be the best for DNS solutions.

[matteo-prosperi]

I suppose in this case using client certificates also allows limiting which users or maybe even applications can interact with servers outside of a certain network boundary.

[scalablecory]

So lets draw out what a proxy connection looks like.

Client -> Client's ISP -> Proxy -> Proxy's ISP -> a series of tubes -> Server's ISP -> Server

For an HTTP proxy, the Client's ISP and the Proxy can see that you are talking to Server. Everyone else sees Proxy talking to Server.

For an HTTPS proxy, the Client's ISP can only see that you are talking to Proxy. This is the sole benefit of an HTTPS proxy -- if you can trust the proxy itself, or at least distrust your ISP, then you gain some small bit of anonymity from your ISP. Everything else remains the same.

Now, regardless of if your proxy is over TLS or not, an HTTPS request is encrypted all the way to Server: even if the Client's ISP can see that you're talking to Server, they still don't know what you're saying. You don't lose any of HTTPS benefits by going over an unencrypted proxy.

[scalablecory]

Note: not saying this is necessarily a bad idea, just that the real benefit is deceptively slim. I'd welcome some examples of when hiding from your ISP will be important under HttpClient usage scenarios.

If there are businesses that want to take a "feel good" TLS-everything approach, I'd hope they would use IPsec rather than relying on HTTP-level encryption inside their network, but I trust @davidsh knows the wants of that space better than me. That is a valid viewpoint to consider.

[richlander]

This is probably all true, but imagine that all the calls and all the end-points are running within a cloud environment. That's what this comment is making reference to: https://github.com/dotnet/corefx/issues/41658#issuecomment-539792003. There is no "ISP".

[richlander]

Let's also assume that the network traffic never leaves the cloud region and you are using an HTTPS version of Envoy.

[scalablecory]

The one significant benefit I'm seeing here is that HTTPS allows cert-based authentication. I'm not sure if we'd need a new API for this or if we could use the existing cert collections in HttpClientHandler.

Would love to know how common cert-based auth is for HTTPS proxies.

[richlander]

I assume there are two things going on:

• Use a proxy as part of a micro-services architecture where the network architecture is transparent to a given app.
• Use HTTPS as a defense-in-depth mechanism to prevent a class of attacks. It's also possible that this is required for regulated businesses.

[richlander]

This answers some of your questions: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl

[davidsh]

If there are businesses that want to take a "feel good" TLS-everything approach, I'd hope they would use IPsec rather than relying on HTTP-level encryption inside their network, but I trust @davidsh knows the wants of that space better than me. That is a valid viewpoint to consider.

There are various initiatives in the regulatory space (government etc.) that are requiring encryption everywhere.

And despite its strong engineering appeal, IPsec hasn't really caught on as much as we would have liked with many enterprises because of the higher cost of setup, configuration, diagnostics, etc.

[opinionmachine]

So for instance, let's say you are using a service for ID&V which is well known and well funded but still unable to deploy normal web service auth. This world famous service provider instead asks its clients to authenticate to their proxy on their network and then proxy the web service calls through there. Obviously, talking HTTP over the entire internet whilst submitting PII in for validation purposes is not only undesireable but probably illegal, so this is not an option.

Today it isn't possible to call the proxy over HTTPS, while it was possible in .NET Core 2.x, presumably by mistake, and Microsoft is shouting that it's "Insane" to want to connect to a proxy over HTTPS, while I would argue it's "Insane" to make it impossible on purpose as there are no valid downsides.

[opinionmachine]

So lets draw out what a proxy connection looks like.

Client -> Client's ISP -> Proxy -> Proxy's ISP -> a series of tubes -> Server's ISP -> Server

For an HTTP proxy, the Client's ISP and the Proxy can see that you are talking to Server. Everyone else sees Proxy talking to Server.

For an HTTPS proxy, the Client's ISP can only see that you are talking to Proxy. This is the sole benefit of an HTTPS proxy -- if you can trust the proxy itself, or at least distrust your ISP, then you gain some small bit of anonymity from your ISP. Everything else remains the same.

Now, regardless of if your proxy is over TLS or not, an HTTPS request is encrypted all the way to Server: even if the Client's ISP can see that you're talking to Server, they still don't know what you're saying. You don't lose any of HTTPS benefits by going over an unencrypted proxy.

So, as I stated in my comment above, a scenario that is realistic for millions of people is the following:

1. Client network -> 2. Internet -> 3. Service Provider Proxy (because they don't know how to do normal web service auth, despite the billions of dollars of revenue) -> 4. Service

I e we need to connect to 3 without talking pure http over 2 as we're sending PII (which granted might be fake - that's what we're trying to validate, but still).

There is _no valid reason not to have it_, you had to _break something_ for it not to be there by default. That's why people like this Identity Validation provider has no second thought requiring people to do this dodgy form of authentication, because they don't think it's totally mindblowing.

[karelz]

Triage: Original request -- support HTTPS proxies sounds reasonable, we should try to implement it in 5.0.
Some of the code already exists in the code base under if-statements.
With @davidsh's enterprise scenario testing, we should be able to test it easier.