Django-filter: Filtering is reaalllllllly slow

[JeromeDesseaux]

Hi and thanks for this awesome library !

I am currently using it in a project and facing a huge problem : adding django-filter multiply render time by 6.

I went from 700 ms to 5 to 6 seconds. This is really huge.

I computed the server side render time which is only 0.23s. Measuring requests using django-toolbar makes an honnest 42ms request time.

This only concerns templates as soon as I remove the filter form, it's working fine again. This is something related to form rendering. I tried with and without crispy : it is the same.

I don't know what to do to improve this load time that is not acceptable.

Thanks for your help :)

[carltongibson]

You have a model choice field or similar trying to render many thousands of rows.

[jonathan-golorry]

Could we have a global setting that just renders those as integer fields?

[carltongibson]

No. Use prefetch_related and co. (As per usual.) (Or customise the widget if that's what you prefer.)

[jonathan-golorry]

The issue isn't the query speed, it's serving and rendering the dropdown lists, especially when combined with the debug toolbar. At a certain point, the REST api browsable interface just times out test servers due ram overusage.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/debug_toolbar/middleware.py", line 100, in __call__
    response.content = insert_before.join(bits)
  File "/usr/local/lib/python3.8/site-packages/django/template/response.py", line 134, in content
    HttpResponse.content.fset(self, value)
  File "/usr/local/lib/python3.8/site-packages/django/http/response.py", line 323, in content
    content = self.make_bytes(value)
  File "/usr/local/lib/python3.8/site-packages/django/http/response.py", line 235, in make_bytes
    return bytes(value.encode(self.charset))
MemoryError

Changing the widget to a NumberInput works fine, but not many people know how to set it up. Here's the change for ForeignKeys:

from django.db.models import ForeignKey
from django.forms import NumberInput
from django_filters import rest_framework as filters
from django_filters.filterset import remote_queryset
...
        filter_overrides = {
            ForeignKey: {
                "filter_class": filters.ModelChoiceFilter,
                "extra": lambda f: {
                    "widget": NumberInput,
                    "queryset": remote_queryset(f),
                },
            },
        }

I haven't tried a ModelMultipleChoiceFilter yet, but I imagine you can do something similar.

It might be good to add something about this to the ModelChoiceFilter documentation. For regular django projects, people will need to either filter the queryset or switch to a more complex widget, but django rest api projects really just need something to fix the browsable test interface.

[merit-mthompson]

@jonathan-golorry thanks for the example! For our internal stuff we landed on making this default behavior for all foreign key fields. This way we don't have to setup classes and inherit stuff all over the place.

We did this by creating a class based on your example and using it as the base class for the back end. This way you can use fieldset_fields in your DRF view sets.

Settings:

REST_FRAMEWORK = {
    # ...
    'DEFAULT_FILTER_BACKENDS': [
        'someproject.filters.SomeProjectFilterBackend',
        'rest_framework.filters.OrderingFilter',
        'rest_framework.filters.SearchFilter',
    ],
    # ...

someproject.filters.py:

from django import forms
from django.db import models
from django_filters.filterset import remote_queryset
from django_filters.rest_framework import ModelChoiceFilter
from django_filters.rest_framework.backends import DjangoFilterBackend
from django_filters.rest_framework.filterset import FilterSet


class ForeignKeyFilterSet(FilterSet):
    """
    Make sure ForeignKey fields show as text input when using the API browser.

    The default widget is a select and with large sets it will take a long time
    to render and probably crash your browser when you try to open the select
    with tens of thousands of items.

    Usage:

    ```
    from someproject.filters import ForeignKeyFilterSet
    from rest_framework import viewsets

    class SomeModelFilter(ForeignKeyFilterSet):
        class Meta(ForeignKeyFilterSet.Meta):
            fields = ['some_foreignkey']
            model = SomeModel

    class SomeModelViewSet(viewsets.ModelViewSet):
        filterset_class = SomeModelFilter
        queryset = SomeModel.objects.all()
        serializer_class = serializers.SomeModelSerializer
    ```

    If this is global via backend and you want default behavior, you need to create
    a filter and use filterset_class instead of filterset_fields on your viewset(s).

    ```
    from django_filters import rest_framework as filters
    class SomeModelFilter(filters.FilterSet):
        class Meta():
            fields = ['some_foreignkey']
            model = SomeModel

    class SomeModelViewSet(viewsets.ModelViewSet):
        filterset_class = SomeModelFilter
        queryset = SomeModel.objects.all()
        serializer_class = serializers.SomeModelSerializer
    ```

    """
    class Meta():
        filter_overrides = {
            models.ForeignKey: {
                'filter_class': ModelChoiceFilter,
                'extra': lambda f: {
                    'queryset': remote_queryset(f),
                    'widget': forms.NumberInput,
                },
            },
        }


class SomeProjectFilterBackend(DjangoFilterBackend):
    """
    REST backend for filtering.

    Use our custom ForeignKeyFilterSet so that you don't have to rename
    ForeignKey fields (e.g., "model__id") or do per-filter/viewset
    customizations to get the equivalent of "raw_id_fields" for
    filters in the browsable API.

    See ForeignKeyFilterSet to restore default behavior as needed.
    """
    filterset_base = ForeignKeyFilterSet

[merit-mthompson]

Also, FWIW, prefetch_related wouldn't help in our case. It would still yield tens of thousands of items which when put into a drop-down on a page can lead to browser stability issues.

It'd be great if there were some equivalent to Django's ModelAdmin.raw_id_fields so you can flip the input per-viewset or something.

[iragm]

I noticed slowness during template rendering like this and I was able to work around it.

In my case, I am filter users by pk and the field is hidden, but you could just as easily enter a name (CharFilter instead of NumberFilter) and show the field.

class MyFilter(django_filters.FilterSet):
    # this creates a list of every single user, even if it's set to display:none
    #user = django_filters.ModelChoiceFilter(label='', queryset=User.objects.all(), method='filter_by_user_slow', widget=Select(attrs={'style': 'display:none'}))
    # better: just use pk of user
    user = django_filters.NumberFilter(label='', method='filter_by_user', widget=NumberInput(attrs={'style': 'display:none'}))

    class Meta:
        model = Lot
        fields = {} # nothing here so no buttons show up

    def filter_by_user(self, queryset, name, value):
        return queryset.filter(user__pk=value)

    def filter_by_user_slow(self, queryset, name, value):
        return queryset.filter(user=value)