Ex_doc: Provide full text search

[josevalim]

To provide full text search, we can use a tool such as lunr.js.

We can build an index off-line when we generate the docs and just use the index on the website - this has the downside of requiring the person generating the docs to have node installed (we can fallback to the old search if they don't have one).

Thoughts?

[josevalim]

@ericmj has mentioned that Elixir docs have 700kb which would very likely make the index very large. There is a chance this approach is not viable at all.

[aphillipo]

I think this would excellent and was looking at the same solution.

700kb for search of all of Elixir isn't too bad, we could choose to load it and cache it after a first interaction with the search bar?

[josevalim]

it is very likely the tool already caches it for us. But given we have concerns with the size, the next step is probably to generate the index and see how long does it take and how large it is.

[ericmj]

700kb is the size of the markdown sources and it only includes module and function docs. I would imagine that the index is larger than the data it indexes. But we need to measure to be sure.

[aphillipo]

I'm knocking up a nasty bit of JS to give us a rough index size. Will let you know the results for full Elixir docs in a bit...

[aphillipo]

Okay so I've stripped the html and the size of the index comes down to 1.6mb before gzip, which is 205kb.

We can statically build the gzip file I suppose...

[ericmj]

@aphillipo Thanks for investigating.

We shouldn't gzip when building because we want this to work locally also and we can configure the CDN to gzip. 205kb isn't beyond the realm of possibility, how long does it take to for lunr to parse the index and return the search result?

Loading https://hexdocs.pm/elixir/Kernel.html fetches 209kb so it would double the page size but only when searching and I guess we can lazy load it?

[josevalim]

Master has already changed to have a specific search.html page so we will

only load it when needed.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[tmbb]

Maybe it makes more sense for hex.pm to provide a search API instead? Just dumping the text into a PostgreSQL database and using the native search function is probably good enough

[ericmj]

@tmbb We are looking into that approach as well and one does not eliminate the other. Javascript search is a lot less development work and will work for offline docs. Search powered by hexdocs.pm can be more powerful because you can search across multiple packages and versions and we can use better search engines. But it also requires more work and requires infrastructure to support it.

Help on either approach would be much appreciated.

[olivernn]

Let me know if you have any questions on getting Lunr to work for you. It sounds like index size is the main question at this point, maybe I can help trying to make that smaller?

The issue of requiring node at build time for docs sites like this has come up before. There is nothing stopping someone from creating a backend for Lunr in something other than JavaScript, it just needs some time. A WIP schema for the generated index is published and I'd be happy to work with anyone wanting to take on an Elixir backend.

[peillis]

I've been doing some research with this to get an idea of the time it takes lunr.js to generate the index.
First I've generated a json with the contents. Basically extending the actual sidebar_items-(digest).js file, adding a "content" key with the docs. I've done a very crappy and basic extraction using Poison to encode the contents, this can be surely improved a lot.
In one of the Elixir docs pages (the biggest docs I've found), I've indexed the whole json object in lunr and get the times in the console.
It takes around 1 second in my laptop, which is a MBP 3.3GHz and using Chrome v63.
I don't know how much time could it take in a slower computer, but I think it's a very reasonable time. Besides, once the index is done it could be serialized and stored in a sessionStorage object.

[josevalim]

@peillis thanks. Assuming we can do it only when search is actually used, I do agree it is quite reasonable.

At the same time, if we can store it in the sessionStorage, we should also be able to precompute it. If that's the case, then I would definitely prefer it, as it yields a better user experience.

[peillis]

Before going any further with this. Is it ok to add Poison as a dependency of ExDoc?

[josevalim]

While that would be OK, I am more inclined to generate the index offline,
if possible. This means we would need node installed (or a js runtime) when
generating the docs... although I think it would be the best user

experience.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[peillis]

Anyway the json is needed to index the contents. Maybe an option could be to generate the index offline if node is available, and do it online when no index is found.

[josevalim]

Anyway the json is needed to index the contents.

I see. Yeah, that makes sense.

Maybe an option could be to generate the index offline if node is available, and do it online when no index is found.

Seems to be more code to maintain though. :) Let's start with the offline one for now and see how it goes!

[peillis]

I've made some progress with this. Please check this out http://demo.doofinder.com.s3.amazonaws.com/enrique/doc/readme.html to see if I'm in the right direction.
It's the Plug documentation working with lunr.js

[aphillipo]

Seems to work great and gave me awesome results (the most correct things were first!). How big is the index size here? Would be worth checking the indexing on the Elixir codebase due to it being likely to be the largest index! Ta.

[josevalim]

This looks really great @peillis! :heart:

A couple of things, sometimes the markdown gets mixed up: http://demo.doofinder.com.s3.amazonaws.com/enrique/doc/search.html?q=example

I am afraid this is not an easy problem to solve? Unless we print to HTML and then we strip the tags? Or maybe the earmark engine has a mechanism to strip only text.

Btw, how should we show the overall information? Should it be something like:

MODULE (OR PAGE)
Entry name (section, type, function, macro, callback, etc)
...part where we matched...

Looking forward to this!

[tmbb]

Stripping the HTML tags seems simpler than messing with the Markdown

[peillis]

I hadn't noticed the markdown issue, I'll try to find a solution for that.
About how to show the information, what I've done so far is:
If there is a match in module, type, function, etc, then show just:

Module.Function

If the match is in the docs, sections, etc, then:

PAGE (or MODULE.FUNCTION...)
...parts with match...

If anyone has a better idea about this please tell me.

[josevalim]

@peillis I found the current match to not have enough information, that's why I proposed the other one:

MODULE (OR PAGE)
Entry name (section, type, function, macro, callback, etc)
...part where we matched...

We treat "MODULE / PAGE" as the title, we treat the "section / function" name as the subtitle and the matched part if any. Would that be possible?

[josevalim]

Examples:

is_atom/1 (function)
...matches an atom...

NAMING CONVENTIONS
Function names (section)
...starts with underscore...

[peillis]

I see. Ok, I think it's possible yes.

[peillis]

Hi, here is a new version with the commented improvements: http://demo.doofinder.com.s3.amazonaws.com/enrique/doc2/readme.html

[DeLongShot]

I became aware that this issue was being worked on last week. I'd like to help. If there is something I can do to help, let me know. Otherwise, I'll jump in where I can.

[eksperimental]

hi @peillis,
One thing I noticed, is the If i type the name of the Module ending with a period. let's say "Enum." it should list all the functions under that module, but here it will give me no matches

[peillis]

Hi, I don't see why people would search with a period at the end. If you search using just the module, for example "Plug.Test" in the Plug example you'll see all the functions of that module.
Anyway, I don't think the point of the search is to show all the functions of a module, for that I would go to the module page in question. I think the search should return the most relevant results.

[josevalim]

@peillis sorry for the delay on this. I think this looks excellent! :heart:

Here is some feedback:

1. The snippets we are showing in the search are not very helpful:

   

   Should we show only a single snippet and increase the radius?

2. Should we include the URL under the match? I think it can help developers decide if they got the correct match. Here is an example (please ignore the style, it is just an example):

   

3. For the pages (such as API reference), you are using the page id ("api-reference") as the header in the search results instead of the page title ("API Reference"). You can see this in the screenshot above since we have "readme" and not "README"

4. As you can see in the screenshot above in the readme results, it seems we are showing unescaped expressions in results, such as &quote

5. If you search for Plug.Conn, you can see the Plug.Conn module appears twice, one for the moduledoc match and later one including each of its functions. Maybe if we the reason of a match is the module name (or the moduledoc), we do not need to show one entry for each function? Just the module itself?

6. If you have an exact match on the subtitle, such as the function name, no snippet is shown. Maybe we could show the first sentence of the doc? Would this be possible?

I am really excited about this!

[milmazz]

@peillis Hi, are you still working on this? Do you need some help?

[peillis]

Hi, I couldn't work on it lately. Any help would be very much appreciated. Here is my work so far https://github.com/peillis/ex_doc/tree/full-text-search

[peillis]

Another step with this issue.

See demo here: http://demo.doofinder.com.s3.amazonaws.com/enrique/doc3/readme.html

I've decided not to group the results under modules for multiple reasons:

• complexity
• many times the results of a module where scattered due to different scorings
• this new way of displaying the results seems more informative, showing also the type of result (function, module, macro, callback...)
• a filter could be added in the future to display only one of those types, i.e. filter by 'functions'

@josevalim about your suggestions:

1. The snippets we are showing in the search are not very helpful:
   Should we show only a single snippet and increase the radius?

It's done. Tell me if you think the radius should be bigger.

1. Should we include the URL under the match? I think it can help developers decide if they got the correct match.

I've tried and imho doesn't seem to be very helpful. Usually it's just the title with some .html# chars.

1. For the pages (such as API reference), you are using the page id ("api-reference") as the header in the search results instead of the page title ("API Reference"). You can see this in the screenshot above since we have "readme" and not "README"

I've changed that. Actually it's not README what it's displayed but the title, "Plug" in this case.

1. As you can see in the screenshot above in the readme results, it seems we are showing unescaped expressions in results, such as &quote

I think those expressions come from code examples, but I'm not completely sure yet, and I'm not sure if we should clean the code examples from the search either.

1. If you search for Plug.Conn, you can see the Plug.Conn module appears twice, one for the moduledoc match and later one including each of its functions. Maybe if we the reason of a match is the module name (or the moduledoc), we do not need to show one entry for each function? Just the module itself?

This is solved with the new disposition of results. It could be argued that the functions of the module shouldn't be results of that search, but if I remove the module name from those items I think we lose accuracy in many other kind of queries, specially in projects with many modules and with queries with multiple terms.

1. If you have an exact match on the subtitle, such as the function name, no snippet is shown. Maybe we could show the first sentence of the doc? Would this be possible?

This is done as well.

[josevalim]

@peillis this is awesome. :heart: :green_heart: :blue_heart: :yellow_heart: :purple_heart: I have no further notes! What is the next step? :)

[peillis]

Great! I have some things to clear up in the code and some small improvements pending. After that I'll make a pull request so you could check the code. Then I guess I should add some tests.

[axelson]

Here is a link to the merged PR: https://github.com/elixir-lang/ex_doc/pull/974

:confetti_ball: :tada: