Pip: Add option to bypass http proxy

[lordyavin]

• Pip version: 10.0.1
• Python version: 2.7.8 & 3.6.3
• Operating system: Windows 7

Description:

I'm trying to install packages from a local (SVN) repository. Since I'm behind a proxy I have configured the proxy in the pip.ini. For local resources pip should bypass the proxy.

Please add an option to specify a list of IP address ranges, domains and domain wildcards to connect directly.

What I've run:

prompt> pip install https://svnsrv.domain.net/svn/MyRepo/trunk/some-package-1.0.0.zip

Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', timeout('timed out',))': /svn/MyRepo/trunk/some-package-1.0.0.zip

[pradyunsg]

Hey @lordyavin! Thanks for filing this issue.

As a workaround, if you pass --proxy="", pip will not use the proxy for that run.

I'm not sure how simple/difficult it would be to provide this support. Do you happen to know how other package managers handle this? That might be a good reference to gain insight from.

[lordyavin]

Why looking at other package managers? Just look at your frontend to the internet. Every browser is capable to bypass a proxy as I requested.

[pfmoore]

Because pip is a package manager, not a browser?

Regardless, as far as I know, pip respects the usual http_proxy, https_proxy and no_proxy environment variables, so why not just use those?

[lordyavin]

This information is new to me. Didn't find that in the manual. I'll give it a try, but I would prefer to configure the no_proxy side by side with the proxy server settings in the ini file.

[pradyunsg]

You're referring to the user guide? We could improve the documentation for this.

[pfmoore]

Agreed we probably should have docs for this. It's a "standard" feature, probably implemented by requests or one of our dependencies rather than by pip itself, so it's not entirely clear where we'd put it in the pip docs, but it would make sense to do so. (When I say "standard" feature, it's standard on Unix, but not on Windows, so that makes documenting it as a cross-platform feature is even mode relevant).

[kishankarun]

@pradyunsg , the workaround option didn't work for me to disable proxy

`
$ pip install flask-socketio --proxy=""
Collecting flask-socketio

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to timed out. (connect timeout=15)')':
`

Can we list the currently used proxy in pip?

[omniproc]

Because pip is a package manager, not a browser?

Regardless, as far as I know, pip respects the usual http_proxy, https_proxy and no_proxy environment variables, so why not just use those?

Because real life configurations are much more complex then this. Proxy/No-Proxy can change depending on application context. Also those proxy ENV vars are probably the most heavily used and yet most uncouth defined pseudo-standard in IT. The syntax is not defined in a standard leading to multiple competing / supported syntaxes. One app may use "*.domain" while another one will use ".domain". One App expects domains to be seperated by , another one expects ; as seperator. One allows to ignore subnets by using CIDR style adressing another doesn't support IP ranges at all.
Great fun if you try to actually run more then one application on a system.

GitHub issues for hundrets of projects are full of issues around this not defined pseudo standard syntax. Most refer to this documentation when asked for the formal definition of the expected syntax. Note how loosely defined it is. Note also how this is not a standard or something worth to be called _definition_ at all since it leaves so much stuff open to interpretation. It's simply how wget decided to handle it.
And still people write software around this crap and expect it to work. Sorry: this is designed to break.

In addition command line args should always have higher precedence then ENV var imho since they are more explicit. However: when those ENV vars are set the suggested --proxy="" option did not work in my tests (Windows platform). Also it seems to be possible to set that param in the pip.ini. It's ignored just like the commandline arg regardless.

IMHO using the http_proxy ENV vars is a nice to have default. But because of the lack of definition and app-level control really every app should at least define a app-specific way to control proxy settings that always should have precedence over the http_proxy system config. At least until we someday will maybe finally have an OS independed, strictly defined proxy setting that can provide those often needed level of control.

[ssbarnea]

In the absence of a --no-proxy option I can confirm that neither --no-proxy="" is available. Like any other CLI tools, the is a rule that a CLI option always take precedence over ENV or config-file options.

This does not work with pip, as pip refuses to disable proxy when that is defined using http_proxy env vars.

Some woud wonder why is such a big deal? Try to remember that the HTTP proxy may do SSL rewriting and pip does not respect the system installed certificates.

So that chain of bugs contribute to providing a poor experience.

[stefanoborini]

Because pip is a package manager, not a browser?

Anything that uses HTTP is, in the end, a browser.

Regardless, as far as I know, pip respects the usual http_proxy, https_proxy and no_proxy environment variables, so why not just use those?

Because your proxy may depend on the index url. For example, I have to use different proxies depending on which URLs I am connecting to. The browser generally handles this through a proxy.pac, that computes dynamically the proxy depending on the URL, but neither pip nor using http_proxy does that.

[pradyunsg]

Is there prior art for other package managers to have a --no-proxy option? That would be a fairly strong indicator. :)

[stefanoborini]

Is there prior art for other package managers to have a --no-proxy option? That would be a fairly strong indicator. :)

Package managers generally have fairly flexible configuration files. Pip is not that smart on that respect.

$ wget --help | grep proxy
--no-proxy explicitly turn off proxy
--proxy-user=USER set USER as proxy username
--proxy-password=PASS set PASS as proxy password

npm has it (can be passed as --option)

https://docs.npmjs.com/misc/config#noproxy

[maths-qureshi]

On windows you can also use the following commands to set the proxy to nothing and then use pip without issues:
set https_proxy=
set http_proxy=

Execute the above two commands. It works for me that way.

[pradyunsg]

Okie; I think it's perfectly reasonable to add a --no-proxy flag for this case. If someone wants to file a PR adding a --no-proxy flag, they're welcome to.

[deveshks]

Hi @pradyunsg

I wanted to work on this PR. From what I could understand from looking at the code on how to approach it is as follows.

We check if the --proxy option is set at https://github.com/pypa/pip/blob/master/src/pip/_internal/cli/req_command.py#L118 and set the session.proxies with the values provided.

I assume that we can do a similar thing with checking if the --noproxy option is provided, and if it is, set the session.proxies['no_proxy] to True which will be checked at https://github.com/pypa/pip/blob/master/src/pip/_vendor/requests/sessions.py#L293 and the proxy won't be used in further requests.

I am assuming that's all there is to it, but. feel free to add more to what is needed to be done.