Aiohttp: Add support for http_proxy environment variable

[wrobell]

A lot of software is checking for "http_proxy" env variable and if it exists, then it uses the proxy specified in the variable.

Would it be possible to add appropriate code in aiohttp.request or should each software using aiohttp do it on its own?

[asvetlov]

I think handling http_proxy environment variable is out of aiohttp duties.
requests library doesn't handle it also for example.

[aeroevan]

IIRC, requests does handle http{,s}_proxy environment variables:
http://docs.python-requests.org/en/latest/user/advanced/#proxies

[asvetlov]

@aeroevan that sounds strange because I cannot find http_proxy string neither in request nor in urllib3 source code.

[wrobell]

Also, urllib.request does

In addition, if proxy settings are detected (for example, when a *_proxy environment variable like http_proxy is set), ProxyHandler is default installed and makes sure the requests are handled through the proxy.

https://docs.python.org/3/library/urllib.request.html#module-urllib.request

[asvetlov]

@wrobell urllib is not the best example.
Anyway, I've made decision: proxy environment variables will not be processed by aiohttp

[aeroevan]

@asvetlov I think they build the environment variable lookup programmatically based on protocol to get HTTPS_PROXY, FTP_PROXY, etc.

[qsz13]

@asvetlov Sorry, but I still don't get it.
Actually, requests use urllib.request.getproxies() to check the env:
https://github.com/kennethreitz/requests/blob/master/requests/compat.py
So, why not add support for the http_proxy in env?

[popravich]

Hi @qsz13, using proxy settings fetched from env will work in a one single case -- when you use all defaults (I mean with ClientSession() as sess: sess.get/post/etc).
Implicitly changing default connector to other one is bad idea.
Consider following example:
You have a project which fetches web pages parses them and store results in elasticsearch.
Lets say that crawler part depends on http_proxy setting and its ok, but elasticsearch
uses http protocol as well, so elasticsearch client will end up trying to connect to your
local elasticsearch server using the same proxy as fetcher (one from http_proxy env var).
Such cases are really hard to debug. Obvious decision in this case to use another env var no_proxy
with elasticsearch server address and so you end up "programming" your environment instead of
correctly programming your system.
Now replace _elasticsearch_ with any other external service using http as a transport and you'll
stuck with the same problem.
BTW, _elasticsearch-py_ driver uses requests lib and (I'm almost sure) will suffer from this issue.

Anyway, import this: explicit is better than implicit

[asvetlov]

I don't want to relay on environment variables by default but adding a function like session_from_env() which reads HTTP_PROXY, creates appropriate connector and return a session with this connector does make sense.
Is any volunteer for the task?

[ashald]

It's a shame that aiohttp is not compatible with a def-facto standard environment variablehttp_proxy that is being used by many standard libraries, languages and platforms to seamlessly introduce a proxy without any code changes. Is there is a chance this can be revisited?

[jettify]

@Ashald could you please describe your use case? why suggested session_from_env will not work for you?

[ashald]

@jettify well, yeah, indeed, session_from_env better than nothing but the problem here is it has to be used at all to achieve such a behavior. Here is a simple example that I faced last week: I have to use some apps to deal with resources that are available to me at this moment only via a proxy. Those apps are written in different languages such as Golang, C, Python etc. And none of their developers though about proxies when they were developing those apps so they didn't do anything special to support them. But thanks to standard libraries in those languages supporting this "de-facto standard" I was able to make those apps work just by executing while having http_proxy set. That's it, voila. This is a total win-win as on one hand developers didn't have to worry about proxies at all and I was able to use the tools without having to patch them.

[asvetlov]

-10 for implicit environment variables usage. Period.
If you want to open potential security manhole by inserting untrusted proxy in chain -- please do it explicitly by your code. Don't ask aiohttp for it.

[ashald]

That's why I said it's a shame such an OSS project is being driven by subjective opinions rather than wide-spread practices and standards. Anyway, thanks for spending your time to explain that this feature is not going to be implemented in aiohttp.

[asvetlov]

Python itself is driven by subjective opinions of Python Core Developers and it's leader Guido van Rossum especially.
The language design is significantly different from wide-spread practices and standards like Java, C, JavaScript, Go (which are in turn very different languages itself).
Is it a shame?

[wrobell]

Java supports HTTP proxy properties (that would be equivalent of the environment variables) -
https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html.

HTTP client in Go language seems to support the http_proxy env variables, look for DefaultTransport description at https://golang.org/pkg/net/http/.

[wrobell]

It seems we can simply use now

async with aiohttp.ClientSession(trust_env=True) as session:

to pull proxy configuration from environment variables?

[asvetlov]

Yes, exactly.

[wrobell]

This is perfect, thank you.

[kstaniek]

This works fine unless you don't use no_proxy env which is is ignored. I would suggest to reopen this issue and provide the full fix.

[asvetlov]

Please make a pull request

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].