Falcon: request.stream.read() returns the POST body wrapped in another set of quotes, thus cannot pass directly to json.loads (string is escaped)

[et304383]

I honestly have no idea how the example given here:
http://falcon.readthedocs.org/en/latest/user/quickstart.html
works:

        body = req.stream.read()
        if not body:
            raise falcon.HTTPBadRequest('Empty request body',
                                        'A valid JSON document is required.')

        try:
            req.context['doc'] = json.loads(body.decode('utf-8'))

When I do request.stream.read() I end up with a string that is escaped:

    request_body = request.stream.read()
    print(request_body)

This gives me something like:

"{\"resource-path\": \"/status\"}"

I tried flask and I do not see this behaviour. The JSON object returned by flask (calling request.json) is a perfectly acceptable JSON string that passes into json.loads fine.

I am using Python 2.7 and falcon 0.3.0. I am making my request calls using the requests module version 2.8.1.

This is definitely confined to falcon as things run fine under other WSGI frameworks. Perhaps a more intelligent method should be added to unescape the string? Unless the intent here is to do very little for the application using this framework to reduce overhead of unnecessary calls.

[bcthomas]

Hi @eric-tucker

I am not sure I understand your problem and without seeing the exact code you are running, it's hard to tell. However, I modified the quickstart falcon script you reference to add in some print statements around the code you mention. I cannot recreate your symptoms (using either py2.7 or py3.5)...

# run server
$ python testfalcon.py 

# prepare json payload file
$ cat test.json
{
  "id": 43
}

# test with httpie
http POST 127.0.0.1:8000/43/things X-Auth-Token:124 X-Project-ID:abc < test.json

# output on server terminal...

JSONTranslator:process_request() -- body is
{
  "id": 43
}

done with body
JSONTranslator:process_request() -- json.loads(body.decode('utf-8')) is
{u'id': 43}
done with json.loads(body.decode('utf-8'))
StorageEngine:add_thing() -- adding thing from post
{u'id': 43}
127.0.0.1 - - [17/Dec/2015 09:25:51] "POST /43/things HTTP/1.1" 201 0

modified script attached.

testfalcon.py.txt
test.json.txt

[michaelgugino]

@eric-tucker request.stream is a buffer object passed in by the WSGI server. It is not necessarily json data that is waiting in that buffer, thus you should not expect it to print as clean json without first using json.loads().

Most likely, flask or other frameworks have implemented a method similar to
body = request.stream.read()
request.json = json.loads(body)

falcon does not provide this convenience by default. This is most likely due to performance reasons; not every application or request method is going to need JSON from the body of a request, thus it's not practical to implement such a feature.

request.stream = env['wsgi.input']

See https://www.python.org/dev/peps/pep-0333/ for more information regarding wsgi.input

[et304383]

It was a long time ago when i was looking at this (and I've since abandoned trying to get a python REST API running under WSGI due to issues with nearly every framework I tried) but from what I recall, I was simply constructing a dictionary object, passing to to the requests module as the json payload, and then calling my API endpoint.

What I would see on the other end was that the request stream would come in as a double escaped string rather than a string that could easily be loaded using json.loads.

I've seen this behaviour elsewhere with another framework and I believe the given solution was to call json.loads twice against the incoming body, but to me this seems like a hack to fix another underlying issue.

Since I've moved on to using another way to deploy my Python REST API (using Amazon API Gateway and Lambda) there's little chance I'll have time to revisit this item in an attempt to reproduce. If you folks cannot reproduce, then I'm sorry for wasting your time and you can close this item.

Thanks for your help either way.

[kgriffs]

Hi @eric-tucker, based on what I know of the source code, the Falcon framework itself doesn't perform any extra escaping on the incoming request body. Just to be sure, I ran the following app under Python 2.7 with Falcon 0.3:

import falcon


class Test(object):
    def on_post(self, req, resp):
        resp.body = req.stream.read()


app = falcon.API()
app.add_route('/', Test())


if __name__ == '__main__':
    from wsgiref.simple_server import make_server
    server = make_server('localhost', 8000, app)
    server.serve_forever()

Then I used requests to do a simple POST:

In [28]: resp = requests.post('http://localhost:8000', json={'some':'data'})

In [29]: resp.content
Out[29]: '{"some": "data"}'

In [30]: resp.json()
Out[30]: {u'some': u'data'}

Meanwhile, the terminal running the app printed out the following:

{"some": "data"}

I'm going to go ahead and close this for now. We can reopen if we get a repro.

[ghost]

Problem in the client. Request library automatically making json.dumps for json argument.
A problematic piece of client code is looks like:
request.post(host, json=json.dumps(data))
Need:
request.post(host, json=data)