Sanic: How to rewrite `response.json()` with middleware?

[ndaidong]

Hello,

I have a router as below:

@app.get('/api/feed/entries')
async def load_feed_entries(request):
    return list_entries(request)

And the handler:

from bson.json_util import dumps
from json import loads

from sanic.response import json

from helpers.MongoStore import mFeed
from helpers.Utils import simplify


def list_entries(req):
    # extract and process args
    # before loading data from mongodb
    cursor = mFeed.find() # with more options

    for item in cursor:
        entries.append(loads(str((dumps(item, ensure_ascii=False)))))

    if len(entries) > 0:
        entries = list(map(simplify, entries))

    result = dict(
        query=dict(
            skip=skip,
            limit=limit,
            language=lang,
            category=category
        ),
        entries=entries
    )

    return json(result)

Then I got the output with encoded non-ascii characters:

In order to get these unicode display as they are, I specify dumps option of json() method as below:

    def utf8_dumps(data):
        return dumps(data, ensure_ascii=False)

    return json(result, dumps=utf8_dumps)

Result now looks like below:

This approach resolves the problem, however I have many endpoints like that. How to avoid repeating the same code? I think of middleware and try:

from json import dumps

from sanic import Sanic
from sanic.response import json

@app.middleware('response')
async def modify_json_response(request, response):
    def utf8_dumps(data):
        return dumps(data, ensure_ascii=False)

    def json2(data):
        json(data, dumps=utf8_dumps)

    response.json = json2

This doesn't work. What's wrong? How to fix it? Thanks.

[Tronic]

Response middleware runs after your handler, when the JSON is already written to response.body. JSON is converted to bytes as soon as you call sanic.response.json() with your data. Thus, the only way this can work is if you can read and write back the body in correct format.

Another approach would be to write your own json function and use it instead of sanic.response.json:

def json(
    body,
    status=200,
    headers=None,
    content_type="application/json",
    dumps=json_dumps,
    **kwargs
):
    return HTTPResponse(
        dumps(body, **{"ensure_ascii": False, **kwargs}),
        headers=headers,
        status=status,
        content_type=content_type,
    )

The above function is copied from sanic/response.py, with only that one line changed. I've written it in a funky way with a dict literal to allow overriding ensure_ascii by kwargs without causing an error about duplicate keyword argument to dumps.

EDIT: A few iterations of corrections (disregard the email notifications you got about this message)

[Tronic]

Ping other developers: would it make sense to default to ensure_ascii=False? I suppose that JSON is always in UTF-8, not ISO-8859-1 or others?

[ndaidong]

@Tronic thank you. It looks like I should create my own json response method based on your suggestion.
Totally agree with you about default support for UTF-8.

[harshanarayana]

@ndaidong Like @Tronic mentioned, it is easy to customize the json dump mechanism and use. The easiest is to patch the global entity in the sanic.respose to use your custom serialization mechanism.

xref: #1609

[Tronic]

There's also an interesting alternative approach: make your handlers return bare dict objects, and add a middleware that converts this into a HTTPResponse.

@app.get("/api/<param>")
def apifunc(request, param):
    return {"parameter": param}

@app.middleware("response")
def rest_response_middleware(request, response):
    if isinstance(response, dict):
        return sanic.response.HTTPResponse(
            ujson.dumps(response, ensure_ascii=False),
            content_type="application/json",
        )

[ndaidong]

@Tronic thank you so much, I like this approach better. However it does not work for me:

Traceback (most recent call last):
  File "/workspace/py/news-crawler/venv/lib/python3.7/site-packages/sanic/server.py", line 442, in write_response
    response.output(
AttributeError: 'dict' object has no attribute 'output'

What I'm missing?

[Tronic]

That error comes from after response middleware have run, if the response is not HTTPResponse by then.

Probably the middleware crashes, and thus doesn't convert the object. Did you import ujson and sanic? If that doesn't help, try printing/logging something within the middleware to see if it runs at all. I only tested this with current git master and it might not work with earlier Sanic versions.

I would presume that such tricks by middleware were never intended in the original design, but by the way middleware are implemented, it happens to work.

[ndaidong]

@Tronic I import ujson and sanic both. Everything work well if I jsonify the result instead of keep it as dict. Look at the exception, I guess that while calling write_response, sanic still treats response object as instance of HTTPResponse class, but it's actually a dict.

I have logs file as below:

MainThread      2019.09.05 21:48:26   INFO      Goin' Fast @ http://0.0.0.0:7535
MainThread      2019.09.05 21:48:26   INFO      Starting worker [20323]
MainThread      2019.09.05 21:48:30   DEBUG     List entries: skip: 0, limit: 15, category: any, lang: any
MainThread      2019.09.05 21:48:30   ERROR     Exception occurred in one of response middleware handlers
Traceback (most recent call last):
  File "/workspace/py/news-crawler/venv/lib/python3.7/site-packages/sanic/app.py", line 985, in handle_request
    request, response
  File "/workspace/py/news-crawler/venv/lib/python3.7/site-packages/sanic/app.py", line 1271, in _run_response_middleware
    _response = middleware(request, response)
  File "server.py", line 27, in rest_response_middleware
    return response.HTTPResponse(
AttributeError: 'dict' object has no attribute 'HTTPResponse'
MainThread      2019.09.05 21:48:30   ERROR     Invalid response object for url b'/api/feed/entries?limit=15', Expected Type: HTTPResponse, Actual Type: <class 'dict'>
MainThread      2019.09.05 21:48:30   ERROR     Exception occurred while handling uri: 'http://0.0.0.0:7535/api/feed/entries?limit=15'
Traceback (most recent call last):
  File "/workspace/py/news-crawler/venv/lib/python3.7/site-packages/sanic/server.py", line 442, in write_response
    response.output(
AttributeError: 'dict' object has no attribute 'output'

My main script:

#!/usr/bin/env python3

import ujson

from sanic import Sanic, response

from engine.crawler import start
from api.feed import list_entries

app = Sanic()

@app.middleware('response')
def rest_response_middleware(request, response):
    if isinstance(response, dict):
        return response.HTTPResponse(
            ujson.dumps(response, ensure_ascii=False),
            content_type='application/json',
        )

@app.get('/api/feed/entries')
async def load_feed_entries(request):
    return list_entries(request)

[ndaidong]

@Tronic ah, my fault. Sorry, there is an error with the variables response. I rename response argument in the middleware to res then it works now!

[Tronic]

Conflict with variable name response and module name response. import sanic or from sanic.response import HTTPResponse. You easily get such conflicts when you use from sanic import response.

[ndaidong]

@Tronic yes, thank you so much. This approach worked for me. Since now I can just return data in the handlers, similar to in the past when I used bottle.py

[kporika]

is it possible to use faster json serializers like orjson in the custom middleware ?

[ndaidong]

is it possible to use faster json serializers like orjson in the custom middleware ?

@kporika I think it's possible, if that lib provides the same json module's apis, with the common methods dumps and loads.

[ahopkins]

Something like this:

from orjson import dumps

response.json(data, dumps=dumps)