Requests: TypeError: CaseInsensitiveDict is not JSON serializable

[woozyking]

The recent change that makes header case insensitive dict makes it impossible to dump request headers into a JSON object.

To reproduce this issue:

>>> import requests
>>> r = requests.get("http://google.com")
>>> r.headers
CaseInsensitiveDict({'x-xss-protection': '1; mode=block', 'transfer-encoding': 'chunked', 'set-cookie': 'PREF=ID=12d7ece5ef3bf530:FF=0:TM=1369254983:LM=1369254983:S=c6aSx8zzedxojmb5; expires=Fri, 22-May-2015 20:36:23 GMT; path=/; domain=.google.ca, NID=67=pu4Hj0vWxZzqQ1L48tpF50mv0AI3igxPOMqq4AkfeLSORhKgdxMC8Eyv2iSm84UflzVXSTSaaddKx6KPM4NAWGd2JYIm95YPC3BWiJMZsSkj652Mvs4i5hk94weAz6Xz; expires=Thu, 21-Nov-2013 20:36:23 GMT; path=/; domain=.google.ca; HttpOnly', 'expires': '-1', 'server': 'gws', 'cache-control': 'private, max-age=0', 'date': 'Wed, 22 May 2013 20:36:23 GMT', 'p3p': 'CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."', 'content-type': 'text/html; charset=ISO-8859-1', 'x-frame-options': 'SAMEORIGIN'})
>>> import json
>>> json.dumps(r.headers)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: CaseInsensitiveDict({'x-xss-protection': '1; mode=block', 'transfer-encoding': 'chunked', 'set-cookie': 'PREF=ID=12d7ece5ef3bf530:FF=0:TM=1369254983:LM=1369254983:S=c6aSx8zzedxojmb5; expires=Fri, 22-May-2015 20:36:23 GMT; path=/; domain=.google.ca, NID=67=pu4Hj0vWxZzqQ1L48tpF50mv0AI3igxPOMqq4AkfeLSORhKgdxMC8Eyv2iSm84UflzVXSTSaaddKx6KPM4NAWGd2JYIm95YPC3BWiJMZsSkj652Mvs4i5hk94weAz6Xz; expires=Thu, 21-Nov-2013 20:36:23 GMT; path=/; domain=.google.ca; HttpOnly', 'expires': '-1', 'server': 'gws', 'cache-control': 'private, max-age=0', 'date': 'Wed, 22 May 2013 20:36:23 GMT', 'p3p': 'CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."', 'content-type': 'text/html; charset=ISO-8859-1', 'x-frame-options': 'SAMEORIGIN'}) is not JSON serializable

[woozyking]

While this is not actually requests bug, it does break some use cases prior to the introduction of CaseInsensitiveDict usage.

[sigmavirus24]

This is easily worked around by doing:

import requests
import json

r = requests.get(url)
headers_json = json.dumps(dict(r.headers))

Technically we could return any object that behaves like a dict without it being a dict, so I don't see this as much of a problem. What would be fantastic is if there were a __serialize__ method we could declare on objects so that modules like json could just reference that as the author of the object intends.

I'll defer to @Lukasa on this one though. I personally don't find this so offensive frankly but I'm not against changing it either.

[woozyking]

Thank you @sigmavirus24. I'm aware of this workaround, I brought this up only because it broke existing code of mine (so possibly others' who haven't updated to 1.2.x).

I'm :+1: for a __serialize__. Magic methods are sometimes nice.

[sigmavirus24]

Magic methods are so often nice. That's more of a pipe-dream than anything else. The stdlib json module is pretty much frozen except for bug fixes I think. I have to wonder if simplejson or itsdangerous would consider (or even already have) that kind of support.

[Lukasa]

I'm with @sigmavirus24 here. This is a pain, but the Case-Insensitive Dict is awesome.

As for magic methods, I think JSON will never add one. JSON should be round-trippable, e.g: json.loads(json.dumps(obj)) == obj. Allowing arbitrary serialisation breaks that functionality.

@woozyking, for what it's worth, if you find yourself doing this a lot, you can use a [custom JSON encoder with the default method defined](http://docs.python.org/2/library/json.html#json.JSONEncoder.default). Define thedefault` method such that it turns the Case-Insensitive Dict into a dict. Slightly less code repetition if you want it.

Sorry that we caused you this inconvenience, but I think things are going to stay as they are. Thanks so much for raising the issue though! :cake:

[woozyking]

@sigmavirus24 a quick test with the latest simplejson indicates that they don't have built-in resolution for this case.

@Lukasa fully understand.

[sigmavirus24]

@Lukasa that's true. But it would still be convenient. :-P