Zola: Search improvement

[Keats]

See https://zola.discourse.group/t/search-improvement/344/2

We still to discuss what improvements make sense but imagine you have a site with thousands of pages; adding all of that to the search index will result in a huge JS file that is not usable. Being able to select which field would help for example.

[jasikpark]

might look at https://github.com/jameslittle230/stork

i only recommend it because it looks cool and #rust

but it also seems promising!!

_{Sent with GitHawk}

[TheSirC]

Would Tantivy be a possibility for the search ?

It is what powers the search of lib.rs. It is written in Rust and might allow easy integration and, it seems, pretty good performance.

As far as my search goes, the website seems to implement its search logic here, if that can help for reference.

[jasikpark]

It is closer to Apache Lucene than to Elasticsearch or Apache Solr in the sense it is not an off-the-shelf search engine server, but rather a crate that can be used to build such a search engine.

— Tantivy

Stork is built with Rust, and the Javascript library uses WebAssembly behind the scenes. It’s built with content creators in mind, in that it requires little-to-no code to get started and can be extended deeply. It’s perfect for JAMstack sites and personal blogs, but can be used wherever you need a search interface.

— Stork

Based on the goals of the projects, I feel like stork might be more user-friendly for static sites, though Tantivy might offer better opportunity to integrate with Zola +Tera templates?

Idk what people really need in terms of customization though.

_{Sent with GitHawk}

[Keats]

I've received an email of someone having a fork with Tantivy for search with Zola but I don't have their GH handle if they have one :(

To test which search engine/lib to use, the main thing to test is how good are the results. I don't really care about whether it's in Rust or JS like right now, as long as the results are decent and that you can do a usable search input with a little bit of JS

[Th3Whit3Wolf]

@Keats I found them here.

Edit: Here's his article if anyone was interested in it.
Another Edit: update first link from his github to self hosted gitea which has more commits.

[Th3Whit3Wolf]

Does there have to be only option? Could a user choose which one through a flag, config.toml, or even a build feature?

Tantivy and stork look like they have two different but very valid use cases for a workflow that includes zola.

[Keats]

Tantity requires a backend so the output becomes something different from a static site. Zola is a SSG so it won't get an official option that is not completely static.

[Th3Whit3Wolf]

Could zola support a generic index building? With Tantivy you can load schema from json so if zola could take a config file or option where a user can describe what information you want to be indexed and output a json with that information that would be super helpful. Similarly stork asks you to store that information in a toml file and stork will build and index from that information. It would be nice if you could also optionally ask zola to run some command after it indexes on build as well.

Stork asks that you run . . .

stork --build index.toml

to build the index. Obviously since stork is written in rust you could pull in some of it's source code to build it as a part of zola but I think having the ability to run a single shell command would be more flexible for a wider array of search engines.

[Keats]

Another request: https://github.com/getzola/zola/issues/975

[Th3Whit3Wolf]

I made a fork that is based of off zola 10.1 with the changes made by Jonathon Strong here that allow zola to build a tantivy index.

It adds a new subcommand index that takes a n index type -t that can be either elasticlunr or tantivy and optionally an output directory for the index (defaults to public

Here is the output of zola index -h

Create a search index as a stand-alone task, and with additional options

USAGE:
    zola-tantivy index [FLAGS] [OPTIONS] --index-type <index_type>

FLAGS:
        --drafts     Include drafts when loading the site
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -t, --index-type <index_type>    what kind of search index to build [possible values: elasticlunr, tantivy]
    -o, --output-dir <output_dir>    Outputs the generated search index files into the provided dir. Note: Tantivy
                                     indexing produces a directory instead of a file, which will be located at output-
                                     dir/tantivy-index [default: public]

I know @Keats said

Tantity requires a backend so the output becomes something different from a static site. Zola is a SSG so it won't get an official option that is not completely static.

So I was unsure if this pull request would be wanted but I have been using Jonathon Strong's fork for awhile and it feels rude to not send a pull request.

Please let me know what you think.

---

As a side note I have also made a server, you can check it out here, that can

1. query tantivy index
2. send the user to a random page by reading the sitemap zola produces

It uses Actix-web for the web server and Tera for templates so it's pretty easy to make templates based of your zola templates.

For anyone that is self hosting their zola website and wants to use tantivy for their search.

[Keats]

It can be mentioned in the docs but won't be accepted as a PR

[maxdobeck]

Lunr has worked well for me in the past. https://lunrjs.com/ Seems to be maintained still but hard to tell

[Keats]

Zola uses elasticlunr which is based on lunr