Pkgdown: search

[jennybc]

It would be great to search a pkgdown site.

[jayhesselberth]

I've dug into this a bit and will need some feedback on how to proceed.

Search engine

AFAICT the most popular lightweight, client-side library is lunr.js. The strategy with lunr is to create a static JSON file during pkgdown::build_site that contains content to be indexed.

The JSON file needs to be indexed by lunr.js on the fly (i.e., when the first search is executed), which might create some lag time. This JSON can be pre-indexed by lunr.js (into another JSON file), but this requires node/npm so may be clunky to run as part of pkgdown::build_site.

Unfortunately it doesn't look like there is a spec for the JSON used directly by lunr.js, but it could probably be easily reverse engineered to avoid the pre-indexing step.
Here is a draft lunr JSON spec.

What to index

• Content from Rd files. Title and description for sure. Examples? Notes?
• Vignette content. This might be nice, but also increases the size of the search index.

Displaying search results

Easiest is to display results on a new search.html page that lists search results sorted by match. Each result would link back to the original page.

Where to put a search bar?

In the navbar, at the top? Or in the sidebar?

lunr.js resources

• tutorial
• blog post.

[michaellevy]

+1. I imagine folks would often use this over the reference page, so I'd put it right in the navbar.

[hadley]

Hosted option with free for open source option: https://www.algolia.com/

I think that's probably a better way than generating a massive js file that needs to be downloaded prior to initiating a search.

[jayhesselberth]

I tried algolia and it's a fair bit of overhead to set up per site.

I still think the mkdocs approach would be fine. Not sure what you mean by "massive": the lunr.min.js download (the search lib underlying mkdocs) is about 25 KB and a JSON search index for a small/medium site is about the same.

[hadley]

Ok, if the file sizes are that small, then it definitely seems worthwhile to try with lunr.

Rather than generating the spec "by hand", it might be worthwhile to see if we could use the V8 package to call lunr.js from R

[jayhesselberth]

426 includes code to generate the lunr index. Should probably move this discussion there.

[hadley]

Don't we want to build the index locally and then serialise it? That seems like it will make things faster (since we only need to build the index once, not once for every search), and I suspect will create a smaller file.

[hadley]

It would also be useful if you could run it on a bigger package (like ggplot2) and report back on how big the index is.

[jayhesselberth]

I agree that it would be nice to build the search index (which is itself another JSON file) and serialize it.

But the build / serialization step requires that you run lunr.js, which means you would have to call out to node during the pkgdown build process (and also requires you have node installed in the first place). Probably not ideal.

This is why I was exploring the lunr spec in #396: to be able to implement the serialization on the R side. This is possible as the schema is somewhat well-specified but I ended up going with the easy route. This is potential maintainability issue if the lunr.js index spec changes.

[jayhesselberth]

The ggplot2 site-index.json file (i.e., raw data, prior to serialization) is 103.1 KB.

[jayhesselberth]

Takes about 0.1s to build and serialize the index and you get about a 10-fold reduction in size upon serialization.

jayhesselberth@hurk ~/devel/ggplot2/docs
$ time cat site-index.json | node build-index.js > index.json

real    0m0.138s
user    0m0.098s
sys 0m0.031s

jayhesselberth@hurk ~/devel/ggplot2/docs [0 0]
$ ll *json
-rw-r--r-- 1 jayhesselberth staff  12K Feb 21 09:46 index.json
-rw-r--r-- 1 jayhesselberth staff 104K Feb 21 09:42 site-index.json

[hadley]

100 kb is a bit on the large side, but 10 kb seems perfect!

lunr.js can run in the browser, right? I think that implies it doesn't have a bunch of node dependencies, so it's at least worth exploring if we can use the V8 package to run it from R (esp. since we already need to embed lunr.js for the client side ops)

[jayhesselberth]

I figured out how to use V8 to serialize the lunr index, so we should be good there.

But this creates a new small issue: the V8 package has specific install requirements that will force users to install libv8 (most won't have it) before installing pkgdown.

[hadley]

That only affects linux users, who are a small subset of R users. I don't mind having it as a dependency - pkgdown has taken a kitchen sink approach to dependencies so far

[jayhesselberth]

Check out searching at https://competent-tesla-1443ac.netlify.com and let me know what you think. Search for "interval" etc. I can try to address ideas for aesthetic improvements.

It would be really nice to have a highlighted context in the search result, but this has proved difficult (though likely doable eventually).

[hadley]

I think ideally the search would occur entirely from the navbar - i.e. you'd get a drop-down of suggestions, which you could then click on.

You should be able to pass the term along as a query parameter and then add highlighting on the target page.

[pat-s]

Great to see that this topic is moving forward - we can't wait to have a working search feature for https://pat-s.github.io/mlr/ :tada:

[jayhesselberth]

Here's another go. https://pedantic-gates-c86f2f.netlify.com/index.html

Getting search results to appear in a dropdown is too fiddly and not directly supported in bootstrap 3. I think a modal is the most robust way to show the results.

FWIW moving to bootstrap 4 wouldn't be too bad (mostly updates to navbar and sidebar) and there is more support for custom elements.

I think text highlighting should be doable in a PR.

[hadley]

Ok, let's stick with a modal for now (and I'll try to get some help from an RStudio designer to make it work as dropdown). But if we're going to use a modal, I think it'll need an explicit trigger (i.e. pushing enter).

In this PR, let's add the search term as an attribute to the generate link, and then we can do text highlighting in a different PR.

Thanks for all your work on this!

[hadley]

@jayhesselberth where are we on this? Is your PR ready for another round of review?

[jayhesselberth]

Not ready yet. Will submit a revised PR but someone with more serious JS skills is going to need to fix the triggering submit event. I agree it should be pushing enter over keyup, but I couldn't get it to go.

[jayhesselberth]

I'm going to give this a spin: https://community.algolia.com/docsearch/

It may be the easiest way forward if you want functional search before a release

[hadley]

I thought you were getting pretty close with the other approach. What's the major remaining challenge?

[jayhesselberth]

I think docsearch is absolutely the way to go. Very robust, low likelihood of downstream maintainability issues, and free. I will put together a PR that auto-fills docsearch params from _pkgdown.yml.

Docsearch can automatically crawl a website to fill an index that they host. A small amount of JS is then added to the search webpage to enable searching. I'll document this in the PR.

[jayhesselberth]

@hadley Here's a site with functional docsearch. Let me know what you think. https://rnabioco.github.io/valr/

Search for: "end", "group", "interval"

To fix-up #591, need to:

• [x] document setting up an algolia account
• [x] generate a json docsearch config file
• [x] document submitting this config file to scrape a website
• [x] mention keeping the search up-to-date

[jayhesselberth]

Of all the sites listed at the bottom of the Docsearch page, I think the search results for Docusaurus are the cleanest.

Some of their CSS modifications could be ported into pkgdown (search for "docsearch" to find the right tags).

[hadley]

This looks great! I'd say we should review and merge this first pass and can then polish more.

[jayhesselberth]

Another item to prioritize for searching are function names themselves. Currently the only place function names are explicitly included are in the usage statement. Depending on how the documentation is written they might also be included in the section title or examples.

But Docsearch prefers not to include text from <code> sections (usage statements and examples are wrapped in <code>); I can see how these might contribute to noisy search results.

One idea would be to explicitly include function names in a <meta> field that isn't displayed on the page but can be easily scraped. I imagine other search-specific data could also be included via <meta> tags on each page.

[hadley]

I'll move that to a new issue.