Readthedocs.org: ReadTheDocs search and MkDocs

[d0ugal]

(Sorry for adding this as an issue @ericholscher, but we keep missing each other on IRC. If there is a better place, let me know.)

We added the mkdocs json command to MkDocs for RTD's to use in serverside search. Since then I have been working on clientside search support. This means we generate a JSON index in a slightly different format to work with Tipue search and thus we will potentially have two different JSON output formats which is a bit of a pain.

The Tipue format looks like this: https://github.com/Tipue/Tipue-Search/blob/master/tipuesearch/tipuesearch_content.js . The key difference is that it is one file containing all pages and the key names are different but otherwise I think it contains pretty much the same data.

So, my question is, can RTD be adapted to use this format so we can drop the specific json command from MkDocs? If so, I'd be happy to make the change in the ReadTheDocs code.

[ericholscher]

Seems reasonable. Our code for that is here.

https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/search/utils.py#L15

https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/tasks.py#L408-L417

https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/tasks.py#L656-L657

Not super great logic/code, mostly left over from how we do Sphinx.
Definitely happy to have someone else write it :)

On Sat, Dec 13, 2014 at 8:05 AM, Dougal Matthews [email protected]
wrote:

(Sorry for adding this as an issue @ericholscher
https://github.com/ericholscher, but we keep missing each other on IRC.
If there is a better place, let me know.)

We added the mkdocs json command to MkDocs for RTD's to use in serverside
search. Since then I have been working on clientside search
https://github.com/tomchristie/mkdocs/pull/222 support. This means we
generate a JSON index in a slightly different format to work with Tipue
search and thus we will potentially have two different JSON output formats
which is a bit of a pain.

The Tipue format looks like this:
https://github.com/Tipue/Tipue-Search/blob/master/tipuesearch/tipuesearch_content.js
. The key difference is that it is one file containing all pages and the
key names are different but otherwise I think it contains pretty much the
same data.

So, my question is, can RTD be adapted to use this format so we can drop
the specific json command from MkDocs? If so, I'd be happy to make the
change in the ReadTheDocs code.

—
Reply to this email directly or view it on GitHub
https://github.com/rtfd/readthedocs.org/issues/1088.

Eric Holscher
Maker of the internet residing in Portland, Or
http://ericholscher.com

[marcelstoer]

Is this still relevant? mkdocs json has since become deprecated...

[d0ugal]

Yes, I believe this is relevant. We have deprecated it in MkDocs but we won't remove it while RTD still depends on it.

[marcelstoer]

Fair enough. Sorry, I was mislead but I'm grasping at straws trying to figure out why the search for _some_ MkDocs projects works on RTD but not for others (#2020).

[d0ugal]

No problem. I'm not sure how to help you with that issue, I'm not familiar with RTD's search.

[ergonlogic]

It appears that the form action for the searchbox is being injected with this javascript: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/core/static-src/core/js/doc-embed/mkdocs.js.

It looks like it's just building the wrong query, along the lines of: https://readthedocs.org/search/?q=bar&check_keywords=yes&area=default&project=foo&version=X.y&type=file. Either of these paths actually work though:

• https://readthedocs.org/projects/foo/search/?q=bar
• http://foo.readthedocs.io/en/X.y/search.html?q=bar

[marcelstoer]

Thanks for the hint. The first doesn't work for me but the second does:

• https://readthedocs.org/projects/nodemcu/search/?q=docker -> NOK, no results
• http://nodemcu.readthedocs.io/en/dev/search.html?q=docker -> OK

[ergonlogic]

The easiest work-around for this is overriding the searchbox template like so: https://github.com/aegir-project/documentation/compare/2884f80...3a44eec1

Note the change in the form's id (e.g. rtd-search-form-alt). This stops the RTD JS from breaking the default search functionality.

[marcelstoer]

The fact that your work-around has the desired affect means that RTD by default replaces the MkDocs RTD theme? The original theme's searchbox.html isn't so different from yours: https://github.com/mkdocs/mkdocs/blob/master/mkdocs/themes/readthedocs/searchbox.html. So, the custom theme kinda re-introduces the original theme?

[ergonlogic]

RTD isn't replacing the mkdocs theme, just altering it with some javascript. As per http://www.mkdocs.org/user-guide/custom-themes/#creating-a-custom-theme, I just copied the default mkdocs RTD theme's searchbox.html, and altered the form id, so that the JS would no longer have any effect.

[marcelstoer]

@ergonlogic the only "problem" with your nifty custom theme approach I found so far is that RTD changes the implementation of the footer fly-out menu it adds. While it normally sits in the lower left corner it is moved to the lower right and becomes detached once you declare a custom theme in mkdocs.yml.
An alternative is to add the following JavaScript function to your extra.js. It reverts the changes RTD applies to the search form:

  function fixSearch() {
    var target = document.getElementById('rtd-search-form');
    var config = {attributes: true, childList: true};

    var observer = new MutationObserver(function(mutations) {
      // if it isn't disconnected it'll loop infinitely because the observed element is modified
      observer.disconnect();
      var form = $('#rtd-search-form');
      form.empty();
      form.attr('action', 'https://' + window.location.hostname + '/en/' + determineSelectedBranch() + '/search.html');
      $('<input>').attr({
        type: "text",
        name: "q",
        placeholder: "Search docs"
      }).appendTo(form);
    });
    // don't run this outside RTD hosting
    if (window.location.origin.indexOf('readthedocs') > -1) {
      observer.observe(target, config);
    }
  }

[sahilTakiar]

Thanks you @marcelstoer the work-around you implemented in https://github.com/nodemcu/nodemcu-firmware/commit/7dd89dd15ef993c9740ba4d4361e6be8edd1284b is working for me!

[agjohnson]

The latest mkdocs (0.17) removes the json output feature RTD relies on to index, so builds are broken completely with the latest version. It would be great to get someone that knows more about mkdocs maintaining these pieces, we don't have the bandwidth.

Raised in #3174

[marcelstoer]

That is odd because earlier in this issue @d0ugal stated

We have deprecated it in MkDocs but we won't remove it while RTD still depends on it.

[d0ugal]

I don’t have the bandwidth for MkDocs now. So I guess that goal stopped with me. I’d recommend pinning at the previous release for now.

[virtualtam]

Hi,

This issue seems to have been fixed (and/or worked around) by #3361 and #3496, I can confirm the ReadTheDocs search module works properly with MkDocs for the Shaarli Documentation (ref. https://github.com/shaarli/Shaarli/issues/1033)

Thanks @d0ugal, @ericholscher and @humitos :)

[humitos]

@virtualtam I'm not very into mkdocs nor search, but, does your last comment mean that we need to close this issue or there is something else missing to track here?

(thanks for the update, though!)

[virtualtam]

@humitos from a user's perspective, the search feature works for documentation generated with MkDocs and hosted on RTD, so I would suggest closing this issue (unless further changes are needed).

[davidfischer]

It sounds like this is resolved.

[safwanrahman]

I believe this Issue should be open as we are not indexing MkDocs in Elasticsearch index.

[stsewd]

But it uses the default search from mkdocs (I think this issue was about the search in the docs, not in the rtd.org site?)

[safwanrahman]

@stsewd Yeah. currently it uses built in search of mkdocs. But it would be better if we could index the content to our elasticsearch index so we can provide better search result to our users.

[stsewd]

@safwanrahman maybe this interest you https://github.com/rtfd/readthedocs.org/issues/4205

[humitos]

But it would be better if we could index the content to our elasticsearch index so we can provide better search result to our users

IIRC, the problem we have with MkDocs is there is no way to generate a context to create the .json file as we do with Sphinx. The limitation was on MkDocs side, not on us.

I may be wrong, though.

[safwanrahman]

@humitos The json file is created with the HTML file by default. But the structure is different. We need to port our code to index the json file into our Elasticsearch.

[ericholscher]

We need a way to dump mkdocs HTML unthemed, which is the main problem. It needs to be consistent across themes and builds.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stsewd]

This is still valid, but blocked and/or needs a design decision

[humitos]

This issue is related to https://github.com/rtfd/readthedocs.org/issues/4588

[stsewd]

This isn't fully closed. We are going to be indexing search data for mkdocs since next week, but not yet overriding the mkdocs search for ours.

[stsewd]

Hopefully this is going live next week, if someone wants to try the server side search for mkdocs projects next week, let us know via email at [email protected].

[marcelstoer]

What are the changes MkDocs-based projects have to make to be compliant with what goes live next week? Or maybe, which work-arounds have to/can be removed next week?

[stsewd]

Let me explain this a little more. We used to index search data only for sphinx projects, but now we index search data for mkdocs too (search works here now https://readthedocs.org/projects/mkdocs-clone/search/?q=test&type=file&project=mkdocs-clone&version=latest for example).

With the other change (to be released next week), now you can get those results inside the doc pages. It should work with all mkdocs themes that have the #mkdocs-search-results and mkdocs-search-query elements. For now, you'll need to trigger a re-build for all versions you want search enabled after we enable the feature for your project. After the beta, you should be able to just use the server side search in all projects without needing to re-build

[stsewd]

If someone is curious, this is how it looks

• https://mkdocs-clone.readthedocs.io/en/latest/
• https://mkdocs-test-stsewd.readthedocs.io/en/latest/search.html?q=versions

If you want to test this let us know via email at [email protected].