Requests: Comma encoding in request params

[sweenzor]

I'm making requests against an API which takes comma separated parameter values.

A working request looks like this:

curl http://localhost:4444/endpoint?val=555,666

When I pass these parameters in with requests.get() however, the resulting URL looks like this:

>>> r = requests.get('http://localhost:4444/endpoint/', params={'val': '555,666'})
>>> r.url
u'http://localhost:4444/endpoint/?val=555%2C666'

I can build the URL myself, but I'm wondering if this is the intended behavior?

[Lukasa]

Ehh...standards are hard.

The comma (along with some other characters) is defined in RFC 3986 as a reserved character. This means the comma has defined meaning at various parts in a URL, and if it is not being used in that context it needs to be percent-encoded.

That said, the query parameter doesn't give the comma any special syntax, so in query parameters we probably shouldn't be encoding it. That said, it's not entirely Requests' fault: the parameters are encoded using urllib.urlencode(), which is what is percent-encoding the query parameters.

This isn't easy to fix though, because some web services use , and some use %2C, and neither is wrong. You might just have to handle this encoding yourself.

@kennethreitz, can you think of anyone who would have a better insight into this problem?

[kennethreitz42]

If you are passing it in via params, we should be doing a full encoding. If you're adding it to the URL yourself, it should not be escaped.

[sweenzor]

Thanks for the quick response and clarification!

[freetstar]

thanks.it cleared the mist in my brain

[dkrieger]

It seems a little inconsistent to encode the comma despite the standard allowing commas in that context, while a param set to a list is encoded like ?myparam=foo&myparam=bar, which has different behavior in different languages/frameworks (some parse this as an array, some take the last value, etc) due to the standard not defining correct behavior there.

To prevent comma encoding, it's as simple as calling urlencode with kwarg safe set to "," if using python3. I'm sure a suitable polyfill to support python2 would be trivial to implement.

Obviously the default behavior of passing a list as a params dict entry value shouldn't change, but perhaps a configuration parameter could be passed to override this behavior?

If either/both of those endeavors get a maintainer's blessing, I'd be happy to whip up a pull request or two.