Flask: Expose JSONDecodeError in flask.json

[bloer]

This is a request for a simple addition to expose JSONDecodeError in flask.json

Expected Behavior

The flask documentation states:

So for starters instead of doing:

try:
   import simplejson as json
except ImportError:
   import json

You can instead just do this:

from flask import json

But if I follow that advice, there is no way to catch decoding exceptions:

try:
    json.loads("abcd")
except json.JSONDecodeError:
    abort(400, "Received invalid json data")

Since I don't know whether flask.json is using json or simplejson, I have to bring back that try/except statement, or else catch generic Exceptions.

Actual Behavior

As written the silly example raises an AttributeError because JSONDecodeError isn't a member of flask.json.

JSONDecodeError                           Traceback (most recent call last)
<ipython-input-43-51448d03474f> in <module>()
      1 try:
----> 2     json.loads("abcd")
      3 except json.JSONDecodeError:

/usr/local/lib/python3.6/site-packages/flask/json/__init__.py in loads(s, app, **kwargs)
    235         s = s.decode(encoding)
--> 236     return _json.loads(s, **kwargs)
    237 

/usr/local/lib64/python3.6/site-packages/simplejson/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, **kw)
    534         kw['parse_float'] = Decimal
--> 535     return cls(encoding=encoding, **kw).decode(s)
    536 

/usr/local/lib64/python3.6/site-packages/simplejson/decoder.py in decode(self, s, _w, _PY3)
    369             s = str(s, self.encoding)
--> 370         obj, end = self.raw_decode(s)
    371         end = _w(s, end).end()

/usr/local/lib64/python3.6/site-packages/simplejson/decoder.py in raw_decode(self, s, idx, _w, _PY3)
    399                 idx += 3
--> 400         return self.scan_once(s, idx=_w(s, idx).end())

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
<ipython-input-43-51448d03474f> in <module>()
      1 try:
      2     json.loads("abcd")
----> 3 except json.JSONDecodeError:
      4     print("caught")
      5 

AttributeError: module 'flask.json' has no attribute 'JSONDecodeError'

Environment

• Python version: 3.6.8
• Flask version: 1.0.3
• Werkzeug version: 0.14.1

[davidism]

I don't want to turn flask.json into a proxy namespace for all of json. Right now, the things it exports are the things that it actually changes. pallets/itsdangerous#146 is suggesting to remove simplejson altogether since it doesn't really add anything once we drop Python < 3.6.

Related to that, there have also been requests in the past for allowing other JSON implementations. While I'm not sure if it will go anywhere, I have some vague ideas about how that might be doable if we remove simplejson.

Basically, I don't want to add this just to have to turn around and deprecate it. As a project, you probably know whether you're using json or have a dependency on simplejson, so you can import the relevant exception directly. If you're a library and need to support both, there's a private flask.json._json attribute that's whatever module is being used, although you might want to do what's shown in that first example so as not to rely on a private attribute.

[phrfpeixoto]

@bloer +1 this issue SO BADLY.
I Lost a good chuck of time trying to understand why my web hook consumers were getting an HTTP 400 and all my attempts to log the error failed. All I could get was "The browser (or proxy) sent a request that this server could not understand."

The obvious step was try to catch a JSONDecodeError (or at least a ValueError) and log it, but since Flask's request (and so does Werkzeug's for that matter) simply eats away that without any apparent reason, I can't catch the error. The exception is being silently replaces by Werkzeug's BadRequest. I can't even use an error handler to understand what is happening. This is completely hidden.

I can list a few reasons why this should be exposed:

• Zen of python, line 2: Explicit is better than implicit. Let developer see the root cause.
• Flask has an error handling system. If the framework wants to convert JSONDecodeError exceptions into BadRequest to ease out our work, then use its own Error Handler to do that.
• IMO, frameworks should only touch exceptions that the developer him(her)self didn't. The final code being executed is supposed to be under the developer's control, and this replacement is taking that control away
• Exposing the exception would not turn flask.json into a proxy of json. In fact it has nothing to do with that package. The issue happens in flask.wrappers.json and its JSONMixin.on_json_loading_failed

To top all of that, why don't the dreaded BadRequest can't even carry a reference to the original exception causing it? That would be so simple.

As it is right now, we can't catch the original exception in any way, so it seems very wrong to me.

[phrfpeixoto]

@jab @kebin-brown Downvoting without much comment is not very useful.

[phrfpeixoto]

I asked for comments from a few of my friends in development regarding my previous input here, and most of them pointed out that I may have misexpressed myself. This may be related to the fact that I'm not native to English, so I'm here to try to clarify some points.

By any means, I intended to sate that the 50K+ people using Flask are doing non-obvious things and "getting along" with the treatment the framework currently users. This obviously work for the majority of your users, otherwise, there wouldn'd be so many. I merely stated what felt obvious to me at the time of debugging my particular issue.

I really didn't want to sound condescending, or someone that came here just to complain at someone's (free) work for the community. I came here expecting to submit the issue myself and get a discussion going, and I'm really sorry If I expressed myself poorly.

I really hope I can get a feedback on this, and I'm glad to submit a contribution within a pull request if the community deems it plausible.

[bloer]

@phrfpeixoto for what it's worth I found the suggestion to use flask.json._json appropriate. It addressed my need, but is explicitly not a stable interface. Saves the developers having to write a new true public interface only to deprecate it.

I'm far from an expert, but your issue doesn't look like it's directly related to mine. I was seeing JSONDecodeErrors hit the debugger, I just couldn't catch them without an annoying amount of fiddling. Exceptions being caught and transformed into others without the original message being preserved is a different and much more significant problem. I suggest making either a post on SO or a separate issue.

[phrfpeixoto]

@bloer Thanks for taking the time to reply. Now that you mention it, I indeed realize the issue you had is different from mine.
My issues were with request.json (hence my mention to flask.wrappers.json
It seems I've made a mess.