Readthedocs.org: Search isn't indexing the index page of nested pages in htmldir builder

[stsewd]

You can test with this project

https://anymail.readthedocs.io/en/latest/search/?q=Sending+email&check_keywords=yes&area=default

There aren't results from this page https://anymail.readthedocs.io/en/latest/sending/

This is kind of related to the doctype that we want to remove https://github.com/rtfd/readthedocs.org/issues/4638 and kind of the same as https://github.com/rtfd/readthedocs.org/issues/5254

In a more minimal example this is more visible, our search returns 0 results and fallback to sphinx.

The explanation is that we try to index the file sending.fjson but that file doesn't exists, what we want is to index sending/index.fjson

[ericholscher]

Hrm, I thought we fixed this here: https://github.com/rtfd/readthedocs.org/blob/e4966e51a521a5cae44842aa87bb870f59b13b36/readthedocs/projects/models.py#L1149-L1159

[stsewd]

We shouldn't use doctype anymore :).

I think basename is sending (not sending/index or index), that's why.

[ericholscher]

How do you figure out if you should check for an HTMLDir path without doctype? I guess we could just try both, but that feels like it would lead to other bugs.

[stsewd]

That's what I want I figure out now. If there isn't a clean way, we have the doctype from the config module in the database (old projects don't have this, but we could create a migration).

[ericholscher]

Seems like this should work?

In [6]: for p in HTMLFile.objects.filter(project__slug='anymail', version__slug='latest', path='sending/index.html'):
   ...:     print(p.json_file_path)
   ...:
readthedocs.projects.models:1154[6212]: INFO Adjusted json file path: sending/index -> sending
/home/docs/checkouts/readthedocs.org/media/json/anymail/latest/sending.fjson

[ericholscher]

Oh, the filename is /home/docs/checkouts/readthedocs.org/media/json/anymail/latest/sending/index.fjson -- that's the issue.

[ericholscher]

Seems like the logic is broken, and without it it would work 😢

[ericholscher]

Seems like this logic works for everything except the index file... There's some weird edge case here:

In [1]:
   ...: import os
   ...: for p in HTMLFile.objects.filter(project__slug='anymail', version__slug='latest', path__startswith='sending'):
   ...:     print(p.json_file_path)
   ...:     print(os.path.exists(p.json_file_path))
   ...:
   ...:
readthedocs.projects.models:1154[9789]: INFO Adjusted json file path: sending/templates/index -> sending/templates
/home/docs/checkouts/readthedocs.org/media/json/anymail/latest/sending/templates.fjson
True
readthedocs.projects.models:1154[9789]: INFO Adjusted json file path: sending/tracking/index -> sending/tracking
/home/docs/checkouts/readthedocs.org/media/json/anymail/latest/sending/tracking.fjson
True
readthedocs.projects.models:1154[9789]: INFO Adjusted json file path: sending/django_email/index -> sending/django_email
/home/docs/checkouts/readthedocs.org/media/json/anymail/latest/sending/django_email.fjson
True
readthedocs.projects.models:1154[9789]: INFO Adjusted json file path: sending/signals/index -> sending/signals
/home/docs/checkouts/readthedocs.org/media/json/anymail/latest/sending/signals.fjson
True
readthedocs.projects.models:1154[9789]: INFO Adjusted json file path: sending/exceptions/index -> sending/exceptions
/home/docs/checkouts/readthedocs.org/media/json/anymail/latest/sending/exceptions.fjson
True
readthedocs.projects.models:1154[9789]: INFO Adjusted json file path: sending/anymail_additions/index -> sending/anymail_additions
/home/docs/checkouts/readthedocs.org/media/json/anymail/latest/sending/anymail_additions.fjson
True
readthedocs.projects.models:1154[9789]: INFO Adjusted json file path: sending/index -> sending
/home/docs/checkouts/readthedocs.org/media/json/anymail/latest/sending.fjson
False

[stsewd]

Yeah, index is the only one affected.

[ericholscher]

It's because sending.rst or sending/index.rst both lead to the same file :/

[stsewd]

There isn't a sending.rst file, just sending/index.rst https://github.com/anymail/django-anymail/blob/master/docs/sending/index.rst

[ericholscher]

Right, but both files could lead to the same output file, so there's no way to know which it came from :/