Django-filter: NullBooleanField and None value

[GuillaumeCisco]

Hello, I encountered something very interesting today regarding NullBooleanField and the empty ('', 'None') value for filtering null values in db.
For the record, I use django rest framework.

Here is my experience:

I create a django model:

class MyModel(models.Model):
    is_solved = models.NullBooleanField()

And in django rest framework, I declare my View:

class MyModelFilter(filters.FilterSet):
    class Meta:
        model = MyModel
        fields = ('is_solved',)


class MyModelViewSet(ModelViewSet):
    queryset = MyModel.objects.all()
    serializer_class = MyModelSerializer
    filter_backends = (filters.DjangoFilterBackend,)
    filter_class = MyModelFilter

With this, I have a filter on field is_solved with django rest framework.
The filter use a select widget with label/values:

choices = (('', _('Unknown')),
           ('true', _('Yes')),
           ('false', _('No')))

As described here : https://github.com/carltongibson/django-filter/blob/develop/django_filters/widgets.py#L103

When I use this filter, if I select Unknown, everything is returned... so nothing is filtered...
What I wanted is to have only objects with is_solved set to null in db.

Furthermore, I wanted to change the way is_solved is displayed, so I 've made a little modification on my Model.
New model:

CHOICES = (
    (None, "OFF"),
    (True, "YES"),
    (False, "NO")
)
class MyModel(models.Model):
    is_solved = models.NullBooleanField(choices=CHOICES)

With this modification, The filter in django rest framework gives me correct Select Widget, but it now uses the ChoiceField, with its own widget.
It's also make more sense, as now I have a selectBox when creating a Model instead of a text input.
But when I select OFF for filtering, the url set a query_parameter as ?is_solved= :/
It works well with True and False anyway.

So for making it works with django rest framework, I changed my choices to:

CHOICES = (
    ('None', "OFF"),
    (True, "YES"),
    (False, "NO")
)

Note 'None' instead of None. And it's ok, I now have a query parameter ?is_solved=None
but now, nothing is returned with this filter... The list is empty...

Made me put my hands in the django_filters code :)

The key is in this part of the code:
https://github.com/carltongibson/django-filter/blob/develop/django_filters/filters.py#L172

    def filter(self, qs, value):
        if isinstance(value, Lookup):
            lookup = six.text_type(value.lookup_type)
            value = value.value
        else:
            lookup = self.lookup_expr
        if value in EMPTY_VALUES:
            return qs
        if self.distinct:
            qs = qs.distinct()
        qs = self.get_method(qs)(**{'%s__%s' % (self.name, lookup): value})
        return qs

This little piece of code do the filtering.
And I've found something very very interesting with the dkango QuerySet behaviour.

One part of the important lines are :

if value in EMPTY_VALUES:
    return qs

The query is returned non filtered if we have an empty value, which is [], (), u'', {}, None
But our value is 'None' so it's okay, all the precedent code did not transform the 'None' value to None. But using the NullBooleanField without choices does something totally different as the field is not considered as a ChoiceField :
code from django core:

class NullBooleanField(BooleanField):
    """
    A field whose valid values are None, True and False. Invalid values are
    cleaned to None.
    """
    widget = NullBooleanSelect

    def to_python(self, value):
        """
        Explicitly checks for the string 'True' and 'False', which is what a
        hidden field will submit for True and False, for 'true' and 'false',
        which are likely to be returned by JavaScript serializations of forms,
        and for '1' and '0', which is what a RadioField will submit. Unlike
        the Booleanfield we need to explicitly check for True, because we are
        not using the bool() function
        """
        if value in (True, 'True', 'true', '1'):
            return True
        elif value in (False, 'False', 'false', '0'):
            return False
        else:
            return None

    def validate(self, value):
        pass

Now things get really really interesting...
Take a look at this line :

qs = self.get_method(qs)(**{'%s__%s' % (self.name, lookup): value})

It does the filtering to pass to django core db Queryset.
We can translate it to:

qs = qs.filter(is_solved__exact='None')

This operation will return an empty queryset...
It will produce a sql query with is_solved = None

But if we test :

qs = qs.filter(is_solved__exact=None)

It correctly returns only the instances with null values in db.
It will produce a sql query with is_solved IS NULL
This is the exact same thing to test:

qs = qs.filter(is_solved__isnull=True)

However, 'True' and 'False' values does not have this problem. The transcription is correct by Django.

We could think of overriding the lookup_epxr of our filter by isnull but our filter will not work with the True and False values...

At this point, I really don't know how to handle this behavior, django bug ? django filters missing something ?

Anyway, I've been able to make it works by rewriting the filter function of django filters:

    def filter(self, qs, value):
        from django.db.models import NullBooleanField

        if isinstance(value, Lookup):
            lookup = six.text_type(value.lookup_type)
            value = value.value
        else:
            lookup = self.lookup_expr
        if value in EMPTY_VALUES:
            return qs
        if self.distinct:
            qs = qs.distinct()

        # if original field is a NullBooleanField, we need to transform 'None' value as None
        if isinstance(self.model._meta.get_field(self.name), NullBooleanField) and value in ('None',):
            value = None

        qs = self.get_method(qs)(**{'%s__%s' % (self.name, lookup): value})
        return qs

This looks like a hack and it does not please me for now...
If someone is interested on thinking on this issue I will be glad to hear it! :D

For now, any thoughts on this modification?
Right now, I think I will use an IntegerField with a choices option for not having to use a forked version of django filters... as I think it's not a django filters issue a the beginning :)

Thanks,

[rpkilby]

Hi @GuillaumeCisco - it seems like there are three issues here:

1. BooleanWidget does not work w/ NullBooleanFields, as it doesn't validate null values.

There should probably be a NullBooleanWidget that accommodates the additional option.

1. You want to override the text labels for the choices (on/off instead of yes/no).

This is probably best handled by subclassing the widget class and setting the choices property manually. BooleanWidget can't really accept a choices argument, as it expects specific values ('true'/'false' instead of True/False). The same would apply to a potential NullBooleanWidget.

1. There are issues filtering by None, as it doesn't pass the empty value checking.

A magic value such as 'None' or 'null' is going to be necessary here. The filter() method will have to account for this.

[carltongibson]

I think the bulk of this was covered by #519.

@GuillaumeCisco please review that. If there is a specific test case that you think should be covered could you open a PR adding that and we can review.

Thanks.

[GuillaumeCisco]

Thank you @carltongibson and thank you too @rpkilby
Just pulled the develop branch.

The NullBooleanField is working now with django rest framework with this configuration:
Model:

CHOICES = (
    (None, "OFF"),
    (True, "YES"),
    (False, "NO")
)
class MyModel(models.Model):
    is_solved = models.NullBooleanField(choices=CHOICES)

View:

choices = (
    (True, "YES"),
    (False, "NO")
)


class MyModelFilter(filters.FilterSet):

    is_solved = ChoiceFilter(null_label='OFF', choices=choices)

    class Meta:
        model = MyModel
        fields = ('is_solved',)


class MyModelViewSet(ModelViewSet):
    queryset = MyModel.objects.all()
    serializer_class = MyModelSerializer
    filter_backends = (filters.DjangoFilterBackend,)
    filter_class = MyModelFilter

Note the difference between the choices tuples.