Aiohttp: Assert HTTPS through redirects on client request (GET, POST, ...)

[Raphael-C-Almeida]

I need to send some critical data on a post request, but the URL is given by a client, so I need to ensure that the data will be sent under SSL. I check the URL for https schema, but if the host send me 301 and redirect to an HTTP website my data will be sent through an insecure channel.

I think it would be nice if I could set "ensure_https" or "ensure_ssl" on a post request for example. If the host redirected me from https to http and "ensure_https" was set, it should raise an exception.

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are https://github.com/aio-libs/aiohttp/issues/3390 (redirects?), https://github.com/aio-libs/aiohttp/issues/1557 (POST request always follow redirects), https://github.com/aio-libs/aiohttp/issues/141 (body for HTTP GET request), https://github.com/aio-libs/aiohttp/issues/3082 ([client] 301 in POST gets redirected to GET), and https://github.com/aio-libs/aiohttp/issues/852 (Http client HEAD request failed to release).

[asvetlov]

You can achieve the same result right now without waiting for a new feature.
Just disable automatic redirection (allow_redirects=False) and process 301 responses on your own.

[yjqiang]

@asvetlov I think you can delete the labels since it is not a bug or enhancement issue.

[asvetlov]

Well, it is not a bug but a possible request for an enhancement.

[webknjaz]

I think it's impossible to automatically ensure such case. Web server can be doing such redirects conditionally and even if we'll send pre-flight HEAD request it doesn't guarantee that the server's behavior will be redirecting to https. There's no way to check this without sending an actual request with actual data.
OTOH if the client changes http to https before sending a request and disables redirect it could give some level of projection.

I'm not sure whether this feature should be in the framework but if yes, I'd add a flag as forbid_insecure or forbid_http probably. Insecure version is better because it has a potential to be extended to non-http protocols. Like ws vs. wss.

[yjqiang]

I think it's impossible to automatically ensure such case. Web server can be doing such redirects conditionally and even if we'll send pre-flight HEAD request it doesn't guarantee that the server's behavior will be redirecting to https. There's no way to check this without sending an actual request with actual data.
OTOH if the client changes http to https before sending a request and disables redirect it could give some level of projection.

I'm not sure whether this feature should be in the framework but if yes, I'd add a flag as forbid_insecure or forbid_http probably. Insecure version is better because it has a potential to be extended to non-http protocols. Like ws vs. wss.

Yeah. Just like https://www.chromium.org/hsts, and it's different from 301.

[asvetlov]

In other words, aiohttp client should raise an exception if secure mode is requested and the connection is not protected.

Any champion for the feature?

[HeavenVolkoff]

Hello. I also have an interest in this feature and can attempt to implement it.

I think it's impossible to automatically ensure such case. Web server can be doing such redirects conditionally and even if we'll send pre-flight HEAD request it doesn't guarantee that the server's behavior will be redirecting to https. There's no way to check this without sending an actual request with actual data.
OTOH if the client changes http to https before sending a request and disables redirect it could give some level of projection.

I'm not sure whether this feature should be in the framework but if yes, I'd add a flag as forbid_insecure or forbid_http probably. Insecure version is better because it has a potential to be extended to non-http protocols. Like ws vs. wss.

I may be oversimplifying the issue, but restricting the schemes accepted by the r_url should solve this, no?

https://github.com/aio-libs/aiohttp/blob/66d1f9c33da5405e4167d17605050f57d507c533/aiohttp/client.py#L479-L481

What would be the necessity for a pre-flight HEAD request?

[asvetlov]

I suggest checking conn.transtort.get_extra_info('sslcontext') in ClientSession._request() / ClientSession._ws_connect() instead, it is stricter and cleaner flag for secured TSL connection.

[HeavenVolkoff]

I suggest checking conn.transtort.get_extra_info('sslcontext') in ClientSession._request() / ClientSession._ws_connect() instead, it is stricter and cleaner flag for secured TSL connection.

Something like this?
https://github.com/aio-libs/aiohttp/blob/66d1f9c33da5405e4167d17605050f57d507c533/aiohttp/client.py#L402-L404

if redirects > 0 and forbid_insecure and conn.transport.get_extra_info("sslcontext") is None:
  raise InsecureRedirect()

[asvetlov]

Yes, but without redirects > 0.
If a user wants to enforce TLS -- it should be checked regardless HTTP redirection.

Minor neat: forbid_insecure is a little long name. I have no strong opinion but enforce_tls is shorter. Does everybody know what TLS means? Maybe we can find something better?

[yjqiang]

Yes, but without redirects > 0.
If a user wants to enforce TLS -- it should be checked regardless HTTP redirection.

Minor neat: forbid_insecure is a little long name. I have no strong opinion but enforce_tls is shorter. Does everybody know what TLS means? Maybe we can find something better?

https://tools.ietf.org/html/rfc6797
How about hsts(HTTP Strict Transport Security)? But with hsts, aiohttp should refuse the http requests while users enforce this. Just like hsts on chrome.

[asvetlov]

@yjqiang please elaborate

[yjqiang]

@yjqiang please elaborate

Ok. Edited.

[HeavenVolkoff]

Yes, but without redirects > 0.
If a user wants to enforce TLS -- it should be checked regardless HTTP redirection.

Fair point

https://tools.ietf.org/html/rfc6797
How about hsts(HTTP Strict Transport Security)? But with hsts, aiohttp should refuse the http requests while users enforce this. Just like hsts on chrome.

I support the idea of implementing hsts...

Would you guys mind if I open a pull request with the mentioned changes?

[yjqiang]

@HeavenVolkoff "The policy is declared by websites via the Strict-Transport-Security HTTP response header field and/or by other means, such as user agent configuration, for example." So maybe we need to handle it when the server returns hsts response?

[HeavenVolkoff]

I think implementing HSTS spec would be an important security feature to aiohttp. However, after understanding it a bit better, I do not see it as being a solution for this feature request. If what I understood from @Raphael-C-Almeida request is correct. What he (and I, for that matter) want is some form of enforcement to prevent cases when the webserver, being contacted through tls, sends a redirect status code to another insecure URL (eg. http://foo.bar). HSTS would not prevent this as it protects only against insecure access of secure-only domains and sub-domains that could lead to data leak or become a phishing entrypoint.

@Raphael-C-Almeida is my understanding of your request correct?

[yjqiang]

I think implementing HSTS spec would be an important security feature to aiohttp. However, after understanding it a bit better, I do not see it as being a solution for this feature request. If what I understood from @Raphael-C-Almeida request is correct. What he (and me, for that matter) want is some form of enforcement to prevent cases when the webserver, being contacted throught tls, sends a redirect status code to another insecure URL (eg. http://foo.bar). HSTS would not prevent this as it protects only against insecure access of secure-only domains and sub-domains that could lead to data leak or become a phishing entrypoint.

@Raphael-C-Almeida is my understanding of your request correct?

You are right. HSTS only supports when we are at the same domain. And the feature request from @Raphael-C-Almeida is that we should keep tls while redirecting.(eg: url_from_google_result -> actual_url)

[yjqiang]

Yes, but without redirects > 0.
If a user wants to enforce TLS -- it should be checked regardless HTTP redirection.

Minor neat: forbid_insecure is a little long name. I have no strong opinion but enforce_tls is shorter. Does everybody know what TLS means? Maybe we can find something better?

Maybe "tls_redirects" is more readable? We ensure that during redirections, every point is url with https.

[asvetlov]

1. HSTS is nice to have feature but it is a different beast: TLS enforcement is negotiated by a server side. If you want to add it -- please create a new issue.
2. In my vision, the discussed option should restrict non-TLS usage for all client connections, not only redirects from a server.

[Raphael-C-Almeida]

I think implementing HSTS spec would be an important security feature to aiohttp. However, after understanding it a bit better, I do not see it as being a solution for this feature request. If what I understood from @Raphael-C-Almeida request is correct. What he (and me, for that matter) want is some form of enforcement to prevent cases when the webserver, being contacted throught tls, sends a redirect status code to another insecure URL (eg. http://foo.bar). HSTS would not prevent this as it protects only against insecure access of secure-only domains and sub-domains that could lead to data leak or become a phishing entrypoint.

@Raphael-C-Almeida is my understanding of your request correct?

Yes @HeavenVolkoff this is what I meant. It's not exactly HTST.

About the name, @yjqiang, I think the aiohttp user should be able to restrict the connection to a secure one even for in case where no redirects happen, so I think that something like forbid_insecure is better than anything that contains redirect in the name.

@asvetlov also commented about that a few posts above: https://github.com/aio-libs/aiohttp/issues/3416#issuecomment-443469977

[yjqiang]

@Raphael-C-Almeida But the way we are going to use is just forbidding http redirection and http requests. Maybe there will be other ways to hack it? So it is not good for "forbid_insecure".

[Raphael-C-Almeida]

Makes sense.

[HeavenVolkoff]

IMHO enforce_ssl is better than enforce_tls due to the fact that aiohttp uses the argument ssl for passing in a SSLContext and Python, in general, still uses the ssl name...

[asvetlov]

The proposal is forbidding not SSL/TSL transport. The intention is clear, the implementation is obvious.
I don't see a way to hack it

[yjqiang]

The proposal is forbidding not SSL/TSL transport. The intention is clear, the implementation is obvious.
I don't see a way to hack it

forbid_insecure means aiohttp will ban all ways that people can hack connection. So it means aiohttp will be a 100% safe module even in the future. Even if it might be 100% safe right now. But who knows about the future?

[webknjaz]

IMHO enforce_ssl is better than enforce_tls

The modern thing is called TLS, SSL is a deprecated: https://tools.ietf.org/html/rfc7568

[HeavenVolkoff]

The modern thing is called TLS, SSL is a deprecated: https://tools.ietf.org/html/rfc7568

Yes, I know. Just saying that aiohttp and Python still use the term ssl in several places. But I'm ok with either one...

[webknjaz]

@HeavenVolkoff actually, there's a PEP with is seeking to fix that https://www.python.org/dev/peps/pep-0543/