Kuma: AttributeError: 'Query' object has no attribute '_filtered_relations'

[Gregoor]

https://sentry.prod.mozaws.net/operations/mdn-prod/issues/7415690/

Resolver404: {'tried': [[<URLPattern '^media/(?:redesign/)?css/(?P<doc>.*)-min.css$'>], [<URLPattern '^media/(?:redesign/)?js/(?P<doc>.*)-min.js$'>], [<URLPattern '^media/(?:redesign/)?img(?P<suffix>.*)$'>], [<URLPattern '^media/(?:redesign/)?css(?P<suffix>.*)$'>], [<URLPattern '^media/(?:redesign/)?js(?P<suffix>.*)$'>], [<URLPattern '^media/(?:redesign/)?fonts(?P<suffix>.*)$'>], [<URLPattern '^media/uploads/demos/(?:.*)$'>], [<URLPattern '(?i)^(?P<one>.*)//(?P<two>.*)//(?P<three>.*)$'>], [<URLPattern '(?i)^(?P<one>.*)//(?P<two>.*)$'>], [<URLPattern '(?i)^samples/canvas-tutorial/2_1_canvas_rect.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/2_2_canvas_moveto.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/2_3_canvas_lineto.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/2_4_canvas_arc.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/2_5_canvas_quadraticcurveto.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/2_6_canvas_beziercurveto.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/3_1_canvas_drawimage.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/3_2_canvas_drawimage.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/3_3_canvas_drawimage.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/3_4_canvas_gallery.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_1_canvas_fillstyle.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_2_canvas_strokestyle.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_3_canvas_globalalpha.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_4_canvas_rgba.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_5_canvas_linewidth.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_6_canvas_linecap.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_7_canvas_linejoin.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_8_canvas_miterlimit.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_9_canvas_lineargradient.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_10_canvas_radialgradient.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/4_11_canvas_createpattern.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/5_1_canvas_savestate.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/5_2_canvas_translate.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/5_3_canvas_rotate.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/5_4_canvas_scale.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/6_1_canvas_composite.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/6_2_canvas_clipping.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/globalCompositeOperation.html$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/backdrop.png$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/bg_gallery.png$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/gallery_1.jpg$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/gallery_2.jpg$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/gallery_3.jpg$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/gallery_4.jpg$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/gallery_5.jpg$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/gallery_6.jpg$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/gallery_7.jpg$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/gallery_8.jpg$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/picture_frame.png$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/rhino.jpg$'>], [<URLPattern '(?i)^samples/canvas-tutorial/images/wallpaper.png$'>], [<URLPattern '(?i)^samples/domref/mozGetAsFile.html$'>], [<URLPattern '(?i)^samples/raycaster/input.js$'>], [<URLPattern '(?i)^samples/raycaster/Level.js$'>], [<URLPattern '(?i)^samples/raycaster/Player.js$'>], [<URLPattern '(?i)^samples/raycaster/RayCaster.html$'>], [<URLPattern '(?i)^samples/raycaster/RayCaster.js$'>], [<URLPattern '(?i)^samples/raycaster/trace.css$'>], [<URLPattern '(?i)^samples/raycaster/trace.js$'>], [<URLPattern '(?i)^samples/webgl/sample1$'>], [<URLPattern '(?i)^samples/webgl/sample1/index.html$'>], [<URLPattern '(?i)^samples/webgl/sample1/webgl-demo.js$'>], [<UR...
  File "redirect_urls/middleware.py", line 14, in __call__
    resolver_match = self.resolver.resolve(request.path_info)
  File "newrelic/hooks/framework_django.py", line 600, in wrapper
    return _wrapped(*args, **kwargs)
  File "newrelic/hooks/framework_django.py", line 588, in _wrapped
    result = wrapped(path)
  File "django/urls/resolvers.py", line 567, in resolve
    raise Resolver404({'tried': tried, 'path': new_path})

AttributeError: 'Query' object has no attribute '_filtered_relations'
(28 additional frame(s) were not displayed)
...
  File "rest_framework/serializers.py", line 678, in <listcomp>
    self.child.to_representation(item) for item in iterable
  File "rest_framework/serializers.py", line 529, in to_representation
    ret[field.field_name] = field.to_representation(attribute)
  File "rest_framework/fields.py", line 1905, in to_representation
    return method(value)
  File "taggit/utils.py", line 129, in inner
    return func(self, *args, **kwargs)
  File "taggit/managers.py", line 230, in names
    return self.get_queryset().values_list("name", flat=True)

AttributeError: 'Query' object has no attribute '_filtered_relations'

[escattone]

The error above occurred when @Gregoor and I deployed the mdnwebdocs/kuma:385c4b0 Docker image to prod, and was discovered by the integration tests as well as by users (Sentry reported 331 user events). We rolled back on prod soon after the discovery, but we did not roll back on stage, since oddly, we did not see the error on stage (the integration tests run fine on stage). This issue is currently blocking deployments to prod, and so seems like an urgent P1.

[peterbe]

I'm pretty sure I know what's going on. But it won't be easy to reproduce.
The filtering depends on kuma.search.jobs.AvailableFiltersJob which is a cacheback class that relies on a 24h cache. But the problem is that it's not caching plain objects. It's caching a pickled QuerySet instance object.

>>> from kuma.search.jobs import AvailableFiltersJob
>>> AvailableFiltersJob().get()
<QuerySet [<Filter: Open Web Apps>, <Filter: HTML>, <Filter: CSS>, <Filter: JavaScript>, <Filter: APIs and DOM>, <Filter: Canvas>, <Filter: SVG>, <Filter: MathML>, <Filter: WebGL>, <Filter: XUL>, <Filter: Marketplace>, <Filter: Firefox>, <Filter: Firefox OS>, <Filter: Mobile>, <Filter: Web Development>, <Filter: Add-ons & Extensions>, <Filter: Games>, <Filter: Writing Documentation>, <Filter: XPCOM>, <Filter: HTTP>, '...(remaining elements truncated)...']>

So, if you make at least 1 query before the Django 2.2 upgrade, it serializes a Python object into a Redis binary string. Then, you change to Django 2.2 and now unserialize binary string back to a Python object that suddenly appears back into the world where the underlying APIs for Django has changed.

I know I'm vague on details but that's because I actually don't understand how it's going wrong exactly, but it's most certainly indicative of trouble.

All you need to do is to drain the AvailableFiltersJob cacheback cache after you've upgraded to Django 2.2. You can either trigger any save of a kuma.search.models.Filter object or simply start a ./manage.py shell_plus and run:

>>> from kuma.search.jobs import AvailableFiltersJob
>>> AvailableFiltersJob().invalidate()
>>>

This can and will happen again. In some form.
Ideally, we shouldn't be storing pickle objects at all because it might unpickle back into a world where the Python APIs are different.
It would be better if it could cache the tag STRINGS or something. But this stuff goes pretty deep into how the rest_framework's serializer stuff works.

Another mid-quick-and-longterm solution would be to always clear some of these dangerous cacheback caches when you boot up new code. We could, in kuma/search/apps.py::SearchConfig.ready do something like this:

    def ready(self):
        """Configure kuma.search after models are loaded."""
        # We've started a new Python process! Let's not re-use anything from the cache
        # if it has been pickled in case any of our underlying APIs have changed. 
        from kuma.search.jobs import AvailableFiltersJob
        AvailableFiltersJob().invalidate()

        ...

A solution like that might be a good half-way between A) manually clearing the various caches between upgrades, and B) stop storing stuff that can't be JSON serialized in the caches.

[peterbe]

By the way, the reason this isn't happening on Stage would probably be because people aren't searching there often enough, so the very first search someone did (right after you discovered the production problem?) was maybe a cache miss so it didn't suffer from the pickle problem.

Also, the path to reproduce this would involve using persistent Redis and doing some searches BEFORE you upgrade to Django 2.2 and doing some searches AFTER.

[escattone]

For the record, I deployed to prod and ran @peterbe's suggestion. It worked like a charm! Here's the actual output from the Django Python shell that confirms @peterbe's intuition:

>>> from kuma.search.jobs import AvailableFiltersJob
>>> AvailableFiltersJob().invalidate()
/usr/local/lib/python3.8/site-packages/django_redis/serializers/pickle.py:24: RuntimeWarning: Pickled model instance's Django version 1.11.28 does not match the current version 2.2.10.
  return pickle.loads(value)
/usr/local/lib/python3.8/site-packages/django_redis/serializers/pickle.py:24: RuntimeWarning: Pickled queryset instance's Django version 1.11.28 does not match the current version 2.2.10.
  return pickle.loads(value)