Fastapi: [QUESTION] Validation in the FastAPI response handler is a lot heavier than expected

[grigi]

First check

• [x] I used the GitHub search to find a similar issue and didn't find it.
• [x] I searched the FastAPI documentation, with the integrated search.
• [x] I already searched in Google "How to X in FastAPI" and didn't find any information.

Description

So I have built a Tortoise ORM to Pydantic adaptor, and it's about stable, so I started profiling and found some interesting.

Pydantic will validate the data I fetch from the DB, which seems redundant as the DB content is already validated. So we are doing double validation.
Further profiling I found that the majority of time is spent by FastAPI preparing the data for serialisation, and then validating it, and then actually serialising (This specific step is what https://github.com/tiangolo/fastapi/issues/1224#issuecomment-617243856 refers to)

So I am doing essentially triple validation...

I then saw that there is the orjson integration, I tried that... and it made no difference that I could tell. (I'll get to this later)

I did a few experiments (none of them properly tested, but just to get an idea) with a simple benchmark:
(The database was populated with 200 junk user profiles generated by hypothesis, response is 45694 bytes)

Key:
R1 → Using FastAPI to serialise a List[User] model automatically (where User is a Pydantic model)
R2 → Using FastAPI to serialise a List[User] model automatically, but disabled the validation step in serialize_response
R3 → Manually serialised the data using an ORJSONResponse
R4 → Using FastAPI to serialise a List[User] model automatically, bypassed the jsonable_encoder as I'm serialising with orjson
R5 → Using FastAPI to serialise a List[User] model automatically, bypassed both validation and jsonable_encoder
C1 → Use provided pydantic from_orm
C2 → Custom constructor that doesn't validate

My results are:
R1 + C1 → 42req/s
R1 + C2 → 43req/s (Seems the 3 FastAPI steps overpower the validation overhead of from_orm)
R2 + C1 → 56req/s (Disabling the validation IN FastAPI has a much bigger impact?)
R2 + C2 → 63req/s (So, no extra validation)
R3 + C1 → 75req/s
R3 + C2 → 160req/s
R4 + C1 → 53req/s (This orjson-specific optimization gave us a 26% speedup here!)
R4 + C2 → 64req/s (This orjson-specific optimization gave us a 48% speedup here!)
R5 + C1 → 74req/s (So, almost as fast as bypassing the FastAPI response handler)
R5 + C2 → 147req/s

Was somewhat surprised by these results. Seems that Disabling all validation AND skipping the FastAPI response handler gave me a nearly 4x improvement!!

Outcomes:

1. Doing an optimal build from ORM to Pydantic doesn't help much by itself, but with an optimal response handler, it can fly!
2. We should really consider as https://github.com/tiangolo/fastapi/issues/1224#issuecomment-617243856 proposed if using orjson, as it gives a big improvement by itself!
3. Validation in the FastAPI response handler is a lot heavier than expected

Questions:
How advisable/possible is it to have a way to disable validation in the FastAPI response handler? Is it dangerous to do so? Should it be conditionally bypassed? e.g. we specify a way to mark it as safe?

I'm just trying to get rid of some bottlenecks, and to understand the system.

[sm-Fifteen]

Nice, thanks for confirming my suspicions and getting some actual performance measurements on that issue. Out of curiosity, do you benchmark requests per second sequentially or in parallel? Given how encoding is part of the critical path on the event loop, jsonable_encoder and the json renderer are expected to have the worst performance impact when attempting to handle multiple requests in parallel, since they both block the event loop for whatever amount of time they end up running (since they are CPU-bound).

[grigi]

I tested with a concurrency of 10, but making it single concurrency hardly affected the results, as it was using a sqlite db as a backend (which is also single concurrency).

The benchmarks was just to find bottlenecks, and not to be representative of "real-world" use cases.

I think we should see if the default responder is orjson, and then skip jsonable_encoder. And do a full regression test with that config to find out if this is a valid use-case?

[chris-allnutt]

@grigi

• Do you have numbers when using multiple workers?
• Can you try it with postgres (may not help if you don't use something like asynvpg)

I don' think there's anything fastapi can do about the speed of pydantic validation, by design that's how pydantic is setup to work so you can't populate invalid properties on its objects, there's no guarantee your database.

I do agree however there are a good number of dependencies that could probably be improved with either faster or async aware alternatives, the less we block the better this will be, but also realizing there will always be blocking somewhere.

[grigi]

I expect it will scale nearly linearly, the DB is only 1.5% of the execution time...

Also, i'm very aware of how the event loop is susceptible to long-running sync code. This is an exercise in identifying bottlenecks, and then discussing the outcomes (as I'm not that familiar with fastapi internals at this stage)

Here is a annotated flamegraph of the unmodified stack:

1) When one builds a Pydantic object from an existing class-instance/dict, Pydantic will always validate, unless you completely bypass validation using construct. But when doing that, recursive models are not built properly, so ideally one would want to use the validation when building. Also bypassing validation in Pydantic comes with BIG warnings.

In the flamegraph (used vmprof) the exact same C function that does the Pydantic validation was run again part of serialise_response. This is not ideal as when building a Pydantic model off existing data, it already validates in the same way.

Possibly it is needed for the case where one returns a dict, but then the standard validation will happen in any case?

2) Using a faster JSON serialiser is going to speed up the 2.8% portion of the runtine (hence my comment that using orjson does very little in this case), but orjson & Rapidjson could do much of what the jsonable_encoder is doing, and can possibly be skipped. That is now a 39.4% (36.6% + 2.8%) eligible runtime to speed up. Which is significant.

My comments was that ALWAYS forcing validation seems redundant to me, but I don't know all the use-cases. So am seeking advice on how to do this right in the future? Should the serialise_response be automatically selective when it does validation? Skip it altogether? Require manual notice that it's not required?

[grigi]

To Victor (your message disappeared here?)

I manually removed the validation code in FastAPI for this test purposes...
It does seem that we might want to consider forcing validation off on a per-route option?

I can do some PR's for this, time is limited though. I am thinking of 2 different PR's:

1. Skip jsonable_encoder if the Response has a response_model_skip_jsonable_encoder set. (Possibly we want to allow this to be manually set by the route?)
2. Skip validation if when you specify a response_model_skip_validation option when setting up the route.

[acnebs]

I like this last suggestion @grigi.

Though I'm fairly certain jsonable_encoder is called whether you have a response_model set or not, so it would probably make sense to just call it skip_jsonable_encoder.

Clarification: I mean that jsonable_encoder is called for any endpoint where you are not explicitly returning a response. It is not called if you explicitly return a Reponse.

[victoraugustolls]

@grigi thanks for the responde! Soon after I posted I answered my own question in my head and deleted my comments 😬.

About the flamegraph, did you use vprof / py-spy to generate it? Just out of curiosity!

[grigi]

@victoraugustolls vmprof + a local vmprof-server. I find vmprof is one of the easiest profilers to use.
py-spy has slightly higher resolution, but requires more setup and the result isn't as easily browsable as vmprof.
So I tend to use vmprof for high-level profiling, and py-spy for more focused profiling (if the perf criteria has not been met)

@acnebs Those two suggested PR's are for different things. Ideally both will have to happen.
1→jsonable_encoder bypass if your JSON encoder is capable (like orjson)
2 → Skip validation when not needed? (I would prefer an automatic way, but don't know how to make it automatic)

Right now if you use explicit response model, it works nice and fast, but the implicit response model is what is used in all the examples (and it reads nicer). The difference in performance wasn't explained properly, nor the case why we are forcing validation on a strict data structure?

I suppose This should have been 2 different tickets describing two different bottlenecks.

[sm-Fifteen]

Skip validation when not needed? (I would prefer an automatic way, but don't know how to make it automatic)

I do find output validation to be useful in production to ensure that the API format contracts are always respected (which can be difficult to prove otherwise, your data source may end up throwing a null value in a place you didn't expect), so I don't know about disabling it automatically, but I would agree that being able to disable it for routes that return very large payloads would be useful performance-wise.

As for skipping jsonable_encoder, I'm 100% with you that it should be skippable if the framework user knows the json response renderer can handle any configuration of datatypes returned by the given route.

[grigi]

I had an attempt at #1434 But it seems that by skipping jsonable_encoder I need to know if I use the prepared, or validated response object.
What I mean by that is that the prepared object is in serialised form, and validated is in deserialised Pydantic object.

The original response object can be either, and if it is already a Pydantic object it's already fixed up, but a raw datastructure is not. So we can only safely skip if the original response object is already a Pydantic object, else we run the risk of emitting the non-fixed data.

So I think it can only be skipped if we turn validation off for that response.

[acnebs]

For my use-case I'm not using pydantic at all where I want this. I want to be able to return a normal python dict and for that to be dumped by orjson without using jsonable_encoder.

[grigi]

Oh, you mean you didn't specify a response_model?
Yes, that case doesn't even have validation.
Good point, I'll ensure that happens as well :+1:

[grigi]

@acnebs Oh dear. My example tried to serialise Decimal and orjson doesn't support that? So it's missing one of my more common datatypes.

So we can't make this automatic.

So regardless, I should only allow skipping jsonable_encoder if the dev manually specified that one knows what one is doing. e.g. skip_validation=True is specified.

[grigi]

Seems we can't specify a JSON override for unsupported types like in stdlib's. Would love to be able to do something like this: https://github.com/django/django/blob/master/django/core/serializers/json.py#L77

[grigi]

Possibly python-rapidjson would be a good compromise: https://python-rapidjson.readthedocs.io/en/latest/dumps.html
We can use a default() hook to handle unsupported cases, and it supports a fair amount of extra stuff natively (uuid, decimal, datetime)

[acnebs]

We can do that with orjson.

import decimal
import orjson

def default(obj):
    if isinstance(obj, decimal.Decimal):
        return str(obj)
    raise TypeError

orjson.dumps(decimal.Decimal("0.0842389659712649442845"), default=default)

> b'"0.0842389659712649442845"'

Could (should?) then expand this default function to achieve parsing parity with jsonable_encoder when using this automatic orjson behaviour.

[grigi]

How did I miss that in the docs? It works, and will be part of #1434
Although it loses accuracy on Decimal (I had to let it return as a float to make it work the same as the jsonable_encoder)

[sm-Fifteen]

Now that FastAPI has its own ORJSONResponse class, it might be interesting to define the default() callback in that response directly in order to avoid surprises like Decimal not being supported.

[grigi]

Yup, I did this in the linked PR: https://github.com/tiangolo/fastapi/pull/1434/files#diff-f5a428b2dd894bf2a31f0e8020faa3e8R19-R31