Requests: Issues with HTTP proxy and accessing localhost - does requests ignore no_proxy?

[victorhooi]

Hi,

I have a Django app, exposing a REST API using TastyPie.

I have it running locally using Django's runserver, and I'm trying to access it using Requests.

For some reason though, accessing it with Curl or Firefox works fine:

curl "http://localhost:8000/api/v1/host/?name__regex=&format=json"

And on the Django runserver console, I see:

[02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=&format=json HTTP/1.1" 200 2845

However, when I try to use the Python requests module (http://docs.python-requests.org/en/latest/), I get a 404:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>

or:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>

or:

>>> payload = { 'format': 'json'}
>>> r = requests.get('http://localhost:8000/api/v1', params=payload)
>>> r
<Response [404]>
>>> r.url
u'http://localhost:8000/api/v1?format=json'

Also, on the Django runserver console, I see:

[02/Oct/2012 17:25:01] "GET http://localhost:8000/api/v1/host/?name__regex=&format=json HTTP/1.1" 404 161072

Notice that runserver prints the whole path in this case, rather than just the part after localhost:8000 - not sure why that is.

We have a HTTP proxy setup in this environment (Microsoft ISA, I believe), which uses authentication (I've put the username/password in the variable itself).

I've set the no_proxy environment variable and put localhost and 127.0.0.1 in there.

However, requests seems to still be doing something funky with the proxy.

If I actually unset http_proxy - requests works fine:

$ unset http_proxy
$ python
Python 2.7.3 (default, Sep  7 2012, 12:28:31) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> hostname = ''
>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex={}&format=json'.format(hostname))
>>> r
<Response [200]>

So firstly, I'm curious why requests doesn't work with the proxy variable set? And secondly, is there any any way to get it to respect no_proxy?

Cheers,
Victor

[timsavage]

I have a similar problem where I just want to disable the proxy for certain requests (connecting to internal WebServices that are not available outside our DMZ). After digging through the requests code to find out how proxies are handled and how proxy configuration is loaded I have come up with a work-around to disable the proxy when needed:

NO_PROXY = {
    'no': 'pass',
}

r = requests.get(uri, proxies=NO_PROXY)

How this works, if the _proxies_ parameter is passed to the get (or other http method) it is has any empty entries removed before checking if there are any values still in the dictionary. If the dictionary is empty then the variables are loaded from the environment using the method _requests.utils.get_environ_proxies_. By including a value in the 'no' key then the proxies parameter is used which includes empty _http_ and _https_ entries thereby disabling the proxy. The method is a bit hacky but does solve the issue.

[Lukasa]

By default, requests trusts any environment variables the system has in place, and assumes you want to use them (which is usually the default behaviour of UNIXy programs). You can disable this behaviour either by using @timsavage's method (which is clever, nice workaround), or by setting the trust_env config value to False, like so:

r = requests.get('http://www.google.com', config={'trust_env': False})
s = requests.Session(config={'trust_env': False})

You can see everything you can change with the config dictionary here.

Hope that helps!

[victorhooi]

heya,

Hmm, yes, but by the same logic, wouldn't requests also trust the NO_PROXY environment variable as well?

This is meant to be a list of specific hosts for which we don't use the proxy - and it's supported by most *nix HTTP clients, as far as I'm aware

• wget (http://www.editcorp.com/Personal/Lars_Appel/wget/wget_8.html)
• lynx (http://lynx.isc.org/release/lynx2-8-6/lynx_help/keystrokes/environments.html)
• chef (http://wiki.opscode.com/display/chef/Chef+Configuration+Settings)

just to pick a few random ones.

I don't think it's a case of not trusting the http_proxy variable, but more just having support for the no_proxy variable as well?

Cheers,
Victor

[Lukasa]

Ah, good point well made. I have genuinely never seen the NO_PROXY variable used, so didn't even know about it. In that case, we should probably endeavour to respect it. =)

[Lukasa]

Oh, this is interesting! We do actually check for NO_PROXY, but we treat it like a standard proxy prefix, like HTTP_PROXY and FTP_PROXY. Put another way: we do totally the wrong thing with it. =D

[piotr-dobrogost]

just to pick a few random ones.

In every list of command line http clients there must be curl even if it's a random list :)

[Lukasa]

Resolved by #945 (hopefully). Next minor release of Requests should have the fix in it. Please let me know if this turns out not to have fixed it for you.

[tanmay9]

Hi @Lukasa , I was having this issue with Requests 1.2. After reading this thread, I thought it will go away on upgrading to 2.0.1 (which contains your fix). However, it did not fix it for me.
The only way I got around it was to add the following code:

import os
os.environ['no_proxy'] = '127.0.0.1,localhost'

[Lukasa]

@tanmay9 How are you setting the environment variable in question?

[bhechinger]

Neither of these appear to work for me.

@timsavage's method does nothing at all and the config= trick results in:

TypeError: request() got an unexpected keyword argument 'config'

This is python 3.4.0 and requests 2.2.1.

I had to resort to @tanmay9's method of setting the no_http environmental.

Just FYI.

[natejgardner]

Has this been resolved yet? Requests does not appear to respect the NO_PROXY environment variable currently (at least on Windows).

[nateprewitt]

Hi @natejgardner, this issue is over 3 years old and we’ve added support for portions of NO_PROXY since then. There are still a few quirks that won’t be addressed due to backwards incompatibility until the next major version of Requests, but the basic functionality works.

If you’ve found a specific defect, please open a new issue. Thanks!