Django-filter: Automatic __isnull filter field does not work in django-rest-framework usage

[sarvi]

The documentation says that defining __isnull in the filter_fields for a django-rest-framework viewset should allow for filtering based on __isnull.
Didnt work for me.

I have the following code on the server side
from django_filters import rest_framework as filters
class SnapshotViewSet(views.ModelViewSet):
serializer_class = volserializers.SnapshotSerializer
filter_fields = ('snapshot_name', 'branch__branch_name',
'branch__id', 'branch__team__group__name', 'volume__id', 'mirrorof__isnull')

The following code on the client side
result = self.client_action('api/wit/snapshot/list'), {'page': 1, 'ordering': 'id', 'mirrorof__isnull': True})

always returned records with and without mirror=None

I root caused it to the autofield __isnull doesnt seem to be using
django_filters.BooleanField
instead of
django_filters.rest_framework.filters.BooleanField

I fixed by not using the auto filter field, instead defining
class SnapshotFilter(filters.FilterSet):
mirrorof__isnull = filters.BooleanFilter(name='mirrorof', lookup_expr='isnull')
class Meta:
model = Snapshot
fields = ('snapshot_name', 'branch__branch_name',
'branch__id', 'branch__team__group__name', 'volume__id', 'mirrorof__isnull',
'mirrorof__id', 'mirrorof__mirrorof__id'
)

The key being the filters is from django_filters.rest_framework instead of django_filters.

Not sure if this is an expected behaviour, thought i'd report it considering it wasnt clear from the documentation and took a while to figure out.

[rpkilby]

Hi @sarvi. My guess is that your settings are using the old filter backend that is provided by DRF. This has been deprecated in favor of the backend shipped with django-filter. The correct path is django_filters.rest_framework.DjangoFilterBackend.

[sarvi]

This is from my settings file

REST FRAMEWORK Configuration Settings

REST_FRAMEWORK = {
# Use Django's standard django.contrib.auth permissions,
# or allow read-only access for unauthenticated users.
'DEFAULT_AUTHENTICATION_CLASSES': (
'rest_framework.authentication.TokenAuthentication',
'rest_framework.authentication.SessionAuthentication',
),
'DEFAULT_PERMISSION_CLASSES': ('rest_framework.permissions.IsAuthenticated',),
'PAGE_SIZE': 10,
'TEST_REQUEST_DEFAULT_FORMAT': 'json',
'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend', 'rest_framework.filters.OrderingFilter')
}

It looks the same as you suggest

[rpkilby]

Oh, the problem is that the list syntax for Meta.fields only accepts model field names. mirrorof__isnull includes a lookup expression, which isn't valid. You would need to use the dict syntax. eg,

filter_fields = {
    'mirrorof': ['isnull'],
    ...
}

I wrote a brief test just to check, and I'm surprised that you don't get an exception (or maybe it's just hidden by the client code?) You should see something like...

Traceback (most recent call last):
  File "./django-filter/tests/rest_framework/test_backends.py", line 215, in test_fields_with_lookup_exprs
    filter_class = backend.get_filter_class(view, view.get_queryset())
  File "./django-filter/django_filters/rest_framework/backends.py", line 40, in get_filter_class
    class AutoFilterSet(self.default_filter_set):
  File "./django-filter/django_filters/filterset.py", line 86, in __new__
    new_class.base_filters = new_class.get_filters()
  File "./django-filter/django_filters/filterset.py", line 313, in get_filters
    "%s" % ', '.join(undefined)
TypeError: 'Meta.fields' contains fields that are not defined on this FilterSet: mirrorof__isnull

[rpkilby]

Hi @sarvi - any followup here? Were your issues resolved?

[sarvi]

The way I resolve the issue was to do this

from django_filters import rest_framework as filters

class SnapshotFilter(filters.FilterSet):
    ''' filter on snapshot fields '''
    mirrorof__isnull = filters.BooleanFilter(name='mirrorof', lookup_expr='isnull')

    class Meta:
        ''' filter on snapshot fields'''
        model = Snapshot
        fields = ('snapshot_name', 'branch__branch_name',
                  'branch__id', 'branch__team__group__name', 'volume__id', 'mirrorof__isnull',
                  'mirrorof__id', 'mirrorof__mirrorof__id')

class SnapshotViewSet(viewsModelViewSet):
    permission_classes = (permissions.IsAuthenticated,)
    queryset = Snapshot.objects.all().order_by('created_at', 'id')
    serializer_class = volserializers.SnapshotSerializer
    filter_class = SnapshotFilter

I was expecting the following to work

from django_filters import rest_framework as filters

class SnapshotViewSet(viewsModelViewSet):
    permission_classes = (permissions.IsAuthenticated,)
    queryset = Snapshot.objects.all().order_by('created_at', 'id')
    serializer_class = volserializers.SnapshotSerializer
    fields = ('snapshot_name', 'branch__branch_name',
                  'branch__id', 'branch__team__group__name', 'volume__id', 'mirrorof__isnull',
                  'mirrorof__id', 'mirrorof__mirrorof__id')

The problem seems to be that mirrorof__isnull gets mapped to BooleanFilter from django_filter instead of django_filter.rest_framework

I have it working. But I think this is a bug

[rpkilby]

Hi @sarvi - what happens when you do the following?

from django_filters import rest_framework as filters

class SnapshotFilter(filters.FilterSet):
    class Meta:
        model = Snapshot
        fields = ('mirrorof__isnull', )

You should get a TypeError as I mentioned above. It doesn't make sense that you would get a valid FilterSet.

[sarvi]

Hi @rpkilby
Yes. I do get the Type Error like you say I should.

But I wasn't trying that.
I was trying to list the filter fields directly in the viewset as follows. My understanding from the documentation is that this is the simplest way to define the filter fields for a viewset.

class SnapshotViewSet(views.ModelViewSet):
    permission_classes = (permissions.IsAuthenticated,)
    queryset = Snapshot.objects.all().order_by('created_at', 'id')
    serializer_class = volserializers.SnapshotSerializer
    filter_fields = ('snapshot_name', 'branch__branch_name',
                     'branch__id', 'branch__team__group__name', 'volume__id', 'mirrorof__isnull',
                     'mirrorof__id', 'mirrorof__mirrorof__id')

I do see the following warning, but can't tell what that warning means.
I don't see any TypeError in this case though.

/Users/sarvi/virtenv/toothless/lib/python2.7/site-packages/django_filters/rest_framework/backends.py:80: UserWarning: <class 'wit.views.volume.SnapshotViewSet'> is not compatible with schema generation
  "{} is not compatible with schema generation".format(view.__class__)

[rpkilby]

But I wasn't trying that.

If you look at the underlying code, the example is essentially the same as setting filter_fields. You should also be getting a TypeError with the following:

class SnapshotViewSet(views.ModelViewSet):
    filter_fields = ('mirrorof__isnull', )
    ...

The schema generation compatibility warning is an artifact of the underlying TypeError. Are you using the coreapi client to fetch data?

[sarvi]

yes. I am using coreapi to fetch the data. Ok I sorta get it now. Looks like coreapi is some home masking the original error. Is there a better way for me to set the environment such that I am able to see the original TypeError?

I think this issue can be closed.

[rpkilby]

This is a breakdown of what's happening:

• The example usage of filter_fields is invalid, and a TypeError is raised when the backend attempts to generate the FilterSet class.
• The coreapi client relies on the backend to generate a coreschema description of the filterset's fields.
• Schema generation fails due to the original TypeError. A warning is raised, but this masks the relevant details from the underlying exception. Filtering is a no-op since no FilterSet was created.
• By creating a custom FilterSet class with a declaratively created mirrorof__isnull filter, the original TypeError is prevented. Even though mirrof__isnull is present in Meta.fields, automatic filter generation will no-op when a declarative filter already exists.

Is there a better way for me to set the environment such that I am able to see the original TypeError?

I don't think so - the warning just needs to be updated to include the original exception's text.

[yaplej]

I was having a very similar problem. Im using filtering, searching and sorting but trying to filter on __isnull would not work. I tried what @sarvi did and was able to get it working.

from assets.models import Asset
from rest_framework import viewsets
from assets.serializers import AssetSerializer
from rest_framework.filters import SearchFilter, OrderingFilter
from django_filters import rest_framework as filters
from django_filters.rest_framework import DjangoFilterBackend


class AssetFilter(filters.FilterSet):
    ''' filter on Asset fields '''
    parent_fk__isnull = filters.BooleanFilter(name='parent_fk', lookup_expr='isnull')

    class Meta:
        model = Asset
        fields = ('id', 'asset_type_fk', 'asset_type_fk__name', 'parent_fk', 'parent_fk__isnull', 'parent_fk__serial_num', 'serial_num', 'model_fk', 'asset_num', 'model_fk__name', 'location_fk')

class AssetViewSet(viewsets.ModelViewSet):
    queryset = Asset.objects.all()
    serializer_class = AssetSerializer
    filter_backends = (DjangoFilterBackend, SearchFilter, OrderingFilter)
'parent_fk__serial_num', 'serial_num', 'model_fk', 'asset_num', 'model_fk__name', 'location_fk')
    filter_class = AssetFilter
    search_fields = ('id', 'asset_type_fk__name', 'parent_fk__serial_num', 'serial_num', 'asset_num', 'model_fk__name')
    ordering_fields = ('__all__')

[rpkilby]

Hi @yaplej. Most likely, Meta.fields contains a lookup or an invalid model field. It's not possible to determine this without seeing the corresponding models. I'd try to instantiating the FilterSet in your python shell and see what errors are raised.

I think this issue can be closed.

@sarvi - yep. I've opened a separate issue to specifically address the warning's message.

[yaplej]

Hello @rpkilby honestly I'm just fumbling though this right now learning as I go. I entirely expected I was doing something wrong but excited someone else bumped into it and had a solution.

from .models import Asset
from rest_framework import serializers

class AssetSerializer(serializers.ModelSerializer):
    asset_type_fk = serializers.StringRelatedField()
    model_fk = serializers.StringRelatedField()
    location_fk = serializers.StringRelatedField()

    class Meta:
        model = Asset
        fields = ('id', 'asset_type_fk', 'parent_fk', 'serial_num', 'asset_num', 'model_fk', 'location_fk')

and my Model

class Asset(models.Model):
    objects = AssetManager()

    id = models.AutoField(primary_key=True)
    asset_type_fk = models.ForeignKey(AssetType)
    parent_fk = models.ForeignKey('Asset', null=True, blank=True)
    serial_num = CharNullField(max_length=64, null=True, blank=True, unique=False, default=None)  # Name should never be changed.
    asset_num = CharNullField(max_length=64, null=True, blank=True, unique=True, default=None)  # Name should never be changed.
    model_fk = models.ForeignKey(AssetModel, null=True, blank=True)
    location_fk = models.ForeignKey(AssetLocation, null=True, blank=True)

    class Meta:
        unique_together =(('asset_type_fk','serial_num'),('asset_type_fk','asset_num'),)

    def get_absolute_url(self):
        return reverse('asset-detail', kwargs={'pk': self.pk})

    def __str__(self):

        return str("Asset Type:" + self.asset_type_fk.name + " Serial #:" + self.serial_num + " Asset #: " + self.asset_num)

        line = ''
        if self.model_fk:

            if self.model_fk.manufacturer_fk:
                line += self.model_fk.manufacturer_fk.name.strip() + ' '

            line += self.model_fk.name.strip()

            if self.serial_num:
                line += ' (Serial Number: ' + self.serial_num + ')'
            elif self.asset_num:
                line += ' (Asset ID: ' + self.asset_num + ')'
            return line
        else:
            return str(self.id).strip()

    def clean(self, *args, **kwargs):
        if not(self.serial_num or self.asset_num):
            raise ValidationError('Serial # or Asset ID is required.')
        return super(Asset, self).clean()

    def natural_key(self):
        return (self.asset_type_fk.name, self.serial_num)

[rpkilby]

My only guess is that either your AssetType model or AssetModel model does not have a name field, so asset_type_fk__name or model_fk__name fails.

Otherwise, this is difficult to debug without a meaningful stack trace. If there is an issue with Meta.fields, then the class should raise an appropriate exception.