Django-filter: TypeError: can't pickle odict_items objects in python 3.5

[dangusev]

Hello, I caught an error during instantiation of FilterSet-derived class in python>=3.5. In 3.4 everything works ok.

File "/home/dan/venvs/notes3.5/lib/python3.5/site-packages/django_filters/filterset.py" in __init__
  291.         self.filters = copy.deepcopy(self.base_filters)

File "/home/dan/venvs/notes3.5/lib/python3.5/copy.py" in deepcopy
  182.                 y = _reconstruct(x, rv, 1, memo)

File "/home/dan/venvs/notes3.5/lib/python3.5/copy.py" in _reconstruct
  320.                 value = deepcopy(value, memo)

File "/home/dan/venvs/notes3.5/lib/python3.5/copy.py" in deepcopy
  182.                 y = _reconstruct(x, rv, 1, memo)

File "/home/dan/venvs/notes3.5/lib/python3.5/copy.py" in _reconstruct
  297.             state = deepcopy(state, memo)

File "/home/dan/venvs/notes3.5/lib/python3.5/copy.py" in deepcopy
  155.         y = copier(x, memo)

File "/home/dan/venvs/notes3.5/lib/python3.5/copy.py" in _deepcopy_dict
  243.         y[deepcopy(key, memo)] = deepcopy(value, memo)

File "/home/dan/venvs/notes3.5/lib/python3.5/copy.py" in deepcopy
  155.         y = copier(x, memo)

File "/home/dan/venvs/notes3.5/lib/python3.5/copy.py" in _deepcopy_dict
  243.         y[deepcopy(key, memo)] = deepcopy(value, memo)

File "/home/dan/venvs/notes3.5/lib/python3.5/copy.py" in deepcopy
  174.                         rv = reductor(4)

Exception Type: TypeError at /notes/
Exception Value: can't pickle odict_items objects

Example of my FilterSet class. Fields DateTimeFIlter and CommaSeparatedCharFilter are custom:

class NoteListFilterset(django_filters.FilterSet):
    strict = STRICTNESS.RAISE_VALIDATION_ERROR

    keywords = CommaSeparatedCharFilter(name='keyword', lookup_expr='in')
    datetime_from = DateTimeFilter(name='datetime', lookup_expr='gte')
    datetime_to = DateTimeFilter(name='datetime', lookup_expr='lte')
    get_hidden = django_filters.MethodFilter(method='filter_by_get_hidden')

[rpkilby]

Hi @dangusev. Initial reaction is that it doesn't make sense. Two thoughts:

• The test suite is ran against python 3.5, this kind of issue would have been picked up.
• base_filters should be an OrderedDict, not an odict_items. Is something overriding base_filters?

[dangusev]

Sorry, my bad). I have a lot of code like this:

category = ChoiceField(choices=CATEGORIES.items())

When copy.deepcopy() is being called on self.base_fields it tries to pickle choices of fields and fails, because choices have type dict_items.

[Ryanb58]

For those still running into this issue, I solved it like so:

list(CATEGORIES.items())

[paulschreiber]

[With Python 3.85 and Django 3.1.1] Similarly, I’m getting this error in one of my list views:

TypeError: cannot pickle 'dict_items' object

Nothing in the backtrace is in my code:
https://pastebin.com/S4xn8zpT

Turns out the problem was adding a choices list to a CharField in my model:

fails:

record_request_status = models.CharField(max_length=4, null=True, choices=RECORD_REQUEST_STATUS)

works ok:

record_request_status = models.CharField(max_length=4, null=True)

RECORD_REQUEST_STATUS is a dict:

with open('case-manager/src/json_fixtures/record_request_status.json') as json_file:
        RECORD_REQUEST_STATUS = json.load(json_file).items()

Assigning that same choices list to other CharFields in the model works fine.
Assigning other choices lists to record_request_status also fails.

this works:

record_request_status = models.CharField(max_length=4, null=True, choices=list(RECORD_REQUEST_STATUS))

Can someone explain what's going on?

[Ryanb58]

@paulschreiber in your situation RECORD_REQUEST_STATUS returns a dict_items type instead of just a primitive List or Tuple.. dict_items is not a known type for pickle to convert so it throws that error. dict_items does have the proper methods implemented to count as an iterator so the list method works in converting it and thus choices can finally accept what you are passing in.

Hopefully, that explains it in good form, it's been a few years since I last looked at this.

[paulschreiber]

@Ryanb58 I don't understand why it fails only for this field, but works for all other fields in my various models. In each case, I'm reading in a JSON file that's a dictionary.

[Ryanb58]

@paulschreiber I'm interested in the problem but also don't want to blow up this thread. Sounds like it might be something outside the scope of this specific project. I'll shoot you an email.