Django-rest-framework: Calling reverse is very expensive

[JockeTF]

After profiling our system I noticed that calling reverse appears to be very expensive. To see how it affects performance I disabled all our URL fields. Below I show the average request times Before and After disabling the URL fields. Type shows how expensive the request is, and Count the number of requests that were made. All requests are list requests using the JSON format.

| Type | Count | Before | After | Speedup |
| --- | --- | --- | --- | --- |
| Heavy | 100 x | 401 ms | 259 ms | 52 % |
| Middle | 250 x | 170 ms | 107 ms | 59 % |
| Simple | 750 x | 034 ms | 022 ms | 54 % |

The system I'm working on has a URL field for every relation, and uses namespace versioning.

[k4nar]

Interesting numbers. It would be great to have a small demo app for that.

I saw this gist some time ago: https://gist.github.com/un33k/6694257a1d24f3dee1d5
It has interesting ideas to speed-up HyperlinkedRelatedField, but IMO it's not clean enough to be in DRF.

[xordoquy]

Interesting. Thanks to both for the informations.
I think the next step would be to setup a test case that highlight this issue.
I'd assume this also vary on how many urls you do define.

[tomchristie]

Yeah, not super surprised by there - there's a lot of machinery called into when reversing URLs.
There should be lots of potential for using caching the matched pattern, but might be fiddly.

[thedrow]

@xordoquy Feel free to issue a PR to drf-benchmakrs.

[mjtamlyn]

Is this DRF specific or is most of the time in Django (I'm guessing the latter, although there are a few more function calls using DRF's full reversing machinery).

It is however not remotely surprising that calling multiple functions to work out a complete url is more expensive than reading an attribute's value.

[tomchristie]

I believe the later, (tho without hard evidence) based on:

• Memory
• Fairly light footprint of the reverse call through REST framework.

[gcbirzan]

Relatively related, I've changed our get_url to look like this:
https://gist.github.com/gcbirzan/a6395358b854c1e36ce3fc43637a8bbf

It won't help for simple lists, but you get, for free, deduplication of calls to reverse, for nested lists.

[debnet]

We done something similar than @gcbirzan in order to keep cache of reversed URLs but with the possibility to retrieve an URL from another app and/or namespace, it's not perfect but it does the job for us.

https://gist.github.com/debnet/168c423098e88b31e6d99dba972a9380

[akx]

I've noticed a lot of the time is also spent in Django's request.build_absolute_uri().

I have a synthetic benchmark that does 10 requests via Django's Client to a DRF list endpoint that has link serializers and manual URL construction with request.build_absolute_uri(reverse(...)). For this project, I can replace request.build_absolute_uri() with a version that uses a settings-set "URL base", and the speedup is fairly noticeable. This is running under cProfile.

| Version | Total request time | Spent in build_absolute_uri | Spent in reverse |
| -------- | ----:| ----:| ----:|
| request.build_absolute_uri() | 4.500 | 1.344 | 1.524 |
| custom build_absolute_uri() | 2.590 | 0.020 | 1.249 |

[gcbirzan]

This is part of something that I've meaning to get into django, and it's what we use, it should be sensibly faster and should preserve all the current functionality: https://gist.github.com/gcbirzan/f8ce7c3ba8f9b4467010320cdba3b449

[carltongibson]

I'm going to de-milestone this. It's exactly clear what the _TODO_ is.

Happy to see Super-Hero input!

[carltongibson]

I've opened #5614 to investigate Benchmarking and Performance Improvements. I'm going to close this as blocked pending that. As and when we get a decent benchmarking solution in place we will revisit this and the other related performance issues.

My initial take on this one, reviewing, is that unless Django itself makes speed improvements here we're not likely to be able to (or want to) do much. @gcbirzan's couple of suggestions look sensible enough in outline. Maybe we could add hooks to allow caching versions of key functions to be used... (#5614 first.)

[gcbirzan]

I've submitted https://code.djangoproject.com/ticket/28828 that should help.