Docusaurus: Support local search

[yanzhiwei147]

Is this a bug report?

NO

Have you read the [Contributing Guidelines on issues]

YES

Environment

None

Steps to Reproduce

None

Expected Behavior

None

Actual Behavior

None

Reproducible Demo

None

Request new feature

I have a website in my intranet, it generated use gitbook. Now i perfer to use the excellent website generator, and deprecated gitbook. But the tool miss local search capability while migrate process. So we have possible to provide local search(like hexo next theme)?

[cuongnv-ibl]

Do you mean Algoria Search locally ?

[ntlex]

In my company we are looking for a similar solution using Lunrjs, where you can simply index your article locally. At the moment it looks pretty impossible to workaround reactjs and extend the navigation bar to allow the search field to appear without an algolia setup.

[endiliey]

@ntlex
Hmm, thats a good point. It is indeed not possible to extend the navigation bar to allow your own search field (if not algolia) to appear without submitting a PR to Docusaurus itself.

Things will change in Docusaurus v2 though 😊.

[ntlex]

@endiliey when is Docusaurus v2 coming ?

[JoelMarcey]

@ntlex Hi. We are in the planning stages now.

[altafakhai]

We have been evaluating a few tools to generate an internal help system in our company. The help system will be hosted behind the firewall and will not be exposed to public. To use the search feature, we need to have Algolia enabled, which wants the site to be public. Is there a way we can use the Search feature without Algolia? I really like Docusaurus and want to go ahead with it. However, the Search limitation is pulling us back. Will there be an immediate solution to this?

[JoelMarcey]

Hi @altafakhai - I think we just responded to your tweet as well. We do not have a simple mechanism to support search behind a firewall at this point. We are looking at options for v2. There is probably a way to embed a search feature on your custom pages, but that is not optimal, of course.

I believe that Algolia has an offline feature, but I have never used it and I don't think it would easily integrate in the current version of Docusaurus.

cc: @s-pace

[s-pace]

👋 @altafakhai

You can use DocSearch on your internal network if you run it on your own. You will need to create your own configuration and then run it. We do not feature login so far but we are PR welcome. It is based on top of scrapy

You can inspire the configuration from the Docusarus's one.

Happy to help if you need anything else.

[endiliey]

@s-pace I have a question.

If we run the scraper on our own, it will still push the the crawled results to the Algolia index (which is not our own network), and it's not "offline" search. Its just not requiring the website to be publicly accessible (as in to be crawled).

Is this correct ?

[s-pace]

@endiliey Indeed.

You can still use algolia offline but regarding the use-case, it is what we will recommend.

[yanzhiwei147]

Thank everyone provide so much suggests. Now i express my solution:

1. Generate the index.xml to root directory while build website.
2. Add search bar on website.
3. Search from index.xml when search bar text input or change.

Information for your reference.

[altafakhai]

Thanks for the help, @JoelMarcey @s-pace

I will try to work with DocSearch and see if we get the desired results. Will keep you all posted with the progress.

Eagerly waiting for v2. 😃

[JoelMarcey]

@sbansal3096 is planning to work on this either for v1 or v2.

@endiliey @yangshun - are we planning on offering a local search implementation for v2. Or just a pluggable infrastructure for search?

[endiliey]

Just pluggable search box. I think he can try it on v1, the main implementation part is on building the docs index.

[JoelMarcey]

I think he can try it on v1, the main implementation part is on building the docs index.

@endiliey Do you mean he should only index *.md files and not worry about trying to index React pages, etc? If so, I think that is ok. I think @sbansal3096 should use an off the shelf indexer like lunr.js and run with that.

[cryptoads]

Any update on this?

[pauljamescleary]

Any update? Currently looking at static site generators for documentation. Like most things about Docusaurus except for the search. This is a nice project :)

[the-darc]

I have the same issue here on a Internal project and solved it with a server side search using @weixsong/elasticlunr.js.

I would like to send a PR with this solution at some time. But Unfortunately I do not have time to do it right now. For now it is available in @the-darc/Docusaurus.

[cryptoads]

Just implemented your fix. Looks good!! Cheers.

[haslam22]

+1 Local search would be an amazing addition. My use case is that I'd like to use Docusaurus for documentation hosted on an intranet and I can't pass any information to a third-party.

[lashomb]

Same use case as @haslam22 for me.

[SmolinPavel]

Waiting for this feature to be implemented too!

[joetidee]

... it's like being handed the keys to a Porsche .... with no engine :(

[JohnSeals]

Any update on this? Or do I just have to build my own? :-)

Thanks!

[yangshun]

Just to set expectations, the Docusaurus core team won't be working on this anytime soon. Our priority is getting feature parity with Docusaurus v1.

[JohnSeals]

Thanks for the quick reply @yangshun! So if I need to get local search running now is there a path you recommend? There are several options listed above but none of them seem to be really ready for primetime.

[hatzopoulos]

@JohnSeals

For what it's worth I got @the-darc's fork working for me with some modifications and it didn't take very long.

https://github.com/facebook/docusaurus/issues/776#issuecomment-480805342

[JohnSeals]

@hatzopoulos I tried it here and got it working too. Thanks for the tip and thanks to @the-darc!

[cmfcmf]

I have created a plugin for Docusaurus v2.0.0-alpha.40 that enables local search using lunr.js. The source is at https://github.com/cmfcmf/docusaurus-search-local
and you can try it live at https://cmfcmf.github.io/OpenWeatherMap-PHP-API.

If you have any feedback or issues, feel free to open an issue in my repository.
Be aware that search only works in the production build of your documentation (this is due to current limitations of the v2 plugin lifecycle apis).

[tomrav]

I have created a plugin for Docusaurus v2.0.0-alpha.40 that enables local search using lunr.js. The source is at https://github.com/cmfcmf/docusaurus-search-local
and you can try it live at https://cmfcmf.github.io/OpenWeatherMap-PHP-API.
...

@cmfcmf I just integrated your plugin to my v2 project, works out-of-the-box and as expected - AWESOME. Sadly it's on a closed system, so I can't show off.

Thank you for sharing!

[JoelMarcey]

I can't wait to try this out!

cc @yangshun @jordwalke

[rcline]

@cmfcmf You are my hero! I was just about to do this and found you had just posted this solution only an hour earlier!! ❤️

Tested it out and works great! 🎉

[jordwalke]

Very cool @tomrav

[lelouch77]

I created an offline search plugin that looks similar to algolia search using lunr.js Check it out ..
https://github.com/lelouch77/docusaurus-lunr-search

Demo is available at https://lelouch77.github.io/docusaurus-lunr-search-demo/

[yanzhiwei147]

Great! @endiliey Can we merge it into v2.

[JoelMarcey]

@yanzhiwei147 Endi passed away last month 😔

@yangshun - is this something to be merged into v2?

[yangshun]

I don't think we'll make this a core of v2 especially since we don't own the code. It could be left as a community-supported plugin.

[maximousblk]

I don't think we'll make this a core of v2 especially since we don't own the code. It could be left as a community-supported plugin.

@yangshun What can be possible problems if you make this a core feature? And what code are you talking about, the plugin or lunrjs?

---

As an alternative to lunsjs, I would like to suggest FlexSearch. It's super fast, less resource hungry and requires no dependencies. This can be turned into a plugin similar to @lelouch77's solution.

---

problem with the local full text solutions is the the index directly from the source code. This this makes them index unnecessary objects like comments, URLs, code snippets, etc. which add to the final index and look bad when someone searches through that. Algolia on the other hand indexes directly from the final build which removes all the unwanted elements.

What I'm trying to say is, if you need a proper local search, you need to first filter out the unnecessary elements from the raw markdown and then index it.

If you can implement this solution as a core feature in V2, this can be a very strong consideration point when choosing a documentation builder. Currently Docusaurus is behind Vuepress (personal opinion) because Vuepress at least supports native header search assisted by tags.

---

If Docusaurus got a native full-text search feature, that would be the best thing that happened to documentations. just saying

[JoelMarcey]

cc @slorber

[yangshun]

since we don't own the code.

The fact that the plugin is built by the community. The scenario I'm afraid of here is that we require patches to the community plugin but the maintainer is no longer active, we would then have to fork the project in order to make updates to it. We might relax the stance here because we're using remark-admonitions now which likewise is built by the community.

If you can implement this solution as a core feature in V2, this can be a very strong consideration point when choosing a documentation builder.

What's wrong with using Algolia and why is there a need for local search? Is it because the docs are behind a VPN?

[maximousblk]

What's wrong with using Algolia

Nothing. Don't remove algolia. It's just that a local alternative would be super helpful.

why is there a need for local search?

For proprietary software documentation. The index is stored on the algolia servers which can be a problem if ever it gets breached, all the documentation will be public. Though that is a very narrow scenario.

Another case would be if the documentation is stored on a network totally cut off from the internet.

And it will help with PWAs as the index would be available offline.

This would also help beginners as docsearch denies indexing example projects. And not everyone publishes their test projects.

No other documentation builders have proper search other than the ones hosted on their servers. Docsify has a native search but it has the filtering problems I mentioned earlier.

I'm very new to programming in general so I don't really know about the problems that occur while maintaining such a project.

[cmfcmf]

What's wrong with using Algolia and why is there a need for local search? Is it because the docs are behind a VPN?

The reason why I didn't want to use Algolia was mainly to avoid having to read through their terms of service and privacy policy to figure out if and what data they collect from visitors of my website when I load their JavaScript SDK and connect to their servers.

As an alternative to lunsjs, I would like to suggest FlexSearch. It's super fast, less resource hungry and requires no dependencies. This can be turned into a plugin similar to @lelouch77's solution.

Thank you for bringing up FlexSearch! This is certainly an interesting alternative. I will see if using that instead of lunrjs would work for my plugin.

[danforbes]

Thanks to the comments on this issue, I have found many helpful links for local search with Docusaurus v2, but I am currently on v1 and curious if anyone can help me out there. Is anyone aware of any established patterns for using local search with Docusaurus v1?

[JoelMarcey]

@danforbes - just curious, is there anything technical stopping you from moving to v2?

[danforbes]

Just like general time commitments and stuff...this is something I would like to get taken care of in a week's time. Is migrating to v2 fairly straightforward? How much of an impact will it have on the look-and-feel of our site?

[s-pace]

It is pretty quick and easy yes https://v2.docusaurus.io/docs/migrating-from-v1-to-v2/

[s-pace]

Impact is neat with the new features such as dark mode and interactive code snippet

[s-pace]

And you will soon have a new search UX like on https://docsearch.algolia.com/

[JoelMarcey]

@danforbes Yes, depending on the features your v1 site uses, it could be a straightforward conversion. We may have folks who might be able to help convert if you would like a pull request on it.

[danforbes]

Thank you guys I am going to take a look on my end :pray:

[danforbes]

I have prepared a naive attempt based on the migration guide. At the moment, I'm getting some errors related to missing React dependencies, which I think may be related to some information I found on the migration guide. I'd appreciate any suggestions regarding the React issue as well as any general comments on the branch.

[danforbes]

@JoelMarcey any chance someone could take a peek at the above?

[slorber]

@danforbes I can help you.

Let's talk about your migration on your repo (can you open an issue or PR and ping me?) or on Discord instead.

[danforbes]

Thank you, @slorber! Are you able to add comments to this commit in GitHub?

[danforbes]

Updated commit range, @slorber https://github.com/substrate-developer-hub/substrate-developer-hub.github.io/compare/df/chore/upgrade

[danforbes]

@slorber I've also created a draft PR https://github.com/substrate-developer-hub/substrate-developer-hub.github.io/pull/563

[dhitimedin]

I have the same issue here on a Internal project and solved it with a server side search using @weixsong/elasticlunr.js.

I would like to send a PR with this solution at some time. But Unfortunately I do not have time to do it right now. For now it is available in [@the-darc/Docusaurus](https://github.com/the-darc/Docusaurus

I am looking for local search support for Hindi language. Any suggestions or a sulution?

[lelouch77]

@dhitimedin You can make use of this https://github.com/MihaiValentin/lunr-languages/pull/6

[dhitimedin]

@ lelouch77 When I try to do that I get series of warnings. While running "npm run swizzle docusaurus-lunr-search SearchBar", I get exception.

Trace pasted below:

npm WARN deprecated [email protected]: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN @cmfcmf/[email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @docusaurus/[email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @docusaurus/[email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @docusaurus/[email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @docusaurus/[email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @docusaurus/[email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @docusaurus/[email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @docusaurus/[email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @docusaurus/[email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @docusaurus/core@^2.0.0 but none is installed. You must install peer dependencies yourself.

Component "SearchBar" doesn't exists. "SearchBar" is swizzled instead of "SearchBar".
Error: SearchBar is an internal component, and have a higher breaking change probability. If you want to swizzle it, use the "--danger" flag.

[praveenn-vbi]

@dhitimedin Thats happens on version of dosucarus ... Try running npm run swizzle docusaurus-lunr-search SearchBar --- --danger Thanks

[slorber]

These swizzle error messages are tracked in https://github.com/facebook/docusaurus/issues/3408
It's definitively a bug but you should be able to swizzle using the --danger cli flag

[weareoutman]

Hi everyone, I just created https://github.com/easyops-cn/docusaurus-search-local, which is originally forked from cmfcmf/docusaurus-search-local, but later fully rewritten to reach our goal: _up to date_, _maintainable_, _robustness_, _pretty look_, and mostly language of _Chinese_ supported. We have already using it in our production (but not public, that's why we need local search, : ).).

[dhitimedin]

https://github.com/easyops-cn/docusaurus-search-local

Great! I will try right away. Does it support Hindi as a language?

[dhitimedin]

Hi everyone, I just created https://github.com/easyops-cn/docusaurus-search-local, which is originally forked from cmfcmf/docusaurus-search-local, but later fully rewritten to reach our goal: _up to date_, _maintainable_, _robustness_, _pretty look_, and mostly language of _Chinese_ supported. We have already using it in our production (but not public, that's why we need local search, : ).).

UI does not work properly on mobile.

[weareoutman]

@dhitimedin Hope it will help you. It currently supports all lunr-languages + Chinese, but Hindi is not in it yet.

I actually have not tested it with mobile devices. But it can be fixed easily by apply css media queries. I'll fix it later.

[dhitimedin]

@dhitimedin Hope it will help you. It currently supports all lunr-languages + Chinese, but Hindi is not in it yet.

I actually have not tested it with mobile devices. But it can be fixed easily by apply css media queries. I'll fix it later.

Great! Would be helpful. My users use my site mostly from mobile. Do keep me posted on the steps required to rectify.

[dhitimedin]

Hi everyone, I just created https://github.com/easyops-cn/docusaurus-search-local, which is originally forked from cmfcmf/docusaurus-search-local, but later fully rewritten to reach our goal: _up to date_, _maintainable_, _robustness_, _pretty look_, and mostly language of _Chinese_ supported. We have already using it in our production (but not public, that's why we need local search, : ).).

Getting the following error while installing it:

node-pre-gyp WARN Using needle for node-pre-gyp https download
node-pre-gyp WARN Tried to download(404): https://github.com/yanyiwu/nodejieba/releases/download/2.4.1/nodejieba-v2.4.1-node-v72-darwin-x64.tar.gz
node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (node-v72 ABI, unknown) (falling back to source compile with node-gyp)
No receipt for 'com.apple.pkg.CLTools_Executables' found at '/'.

No receipt for 'com.apple.pkg.DeveloperToolsCLILeo' found at '/'.

No receipt for 'com.apple.pkg.DeveloperToolsCLI' found at '/'.

gyp: No Xcode or CLT version detected!
gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack at ChildProcess.emit (events.js:315:20)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)
gyp ERR! System Darwin 19.5.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/Users/nishit/Documents/GitHub/docusaurus3/node_modules/nodejieba/build/Release/nodejieba.node" "--module_name=nodejieba" "--module_path=/Users/nishit/Documents/GitHub/docusaurus3/node_modules/nodejieba/build/Release" "--napi_version=6" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v72"
gyp ERR! cwd /Users/nishit/Documents/GitHub/docusaurus3/node_modules/nodejieba
gyp ERR! node -v v12.18.0
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/usr/local/bin/node /usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/Users/nishit/Documents/GitHub/docusaurus3/node_modules/nodejieba/build/Release/nodejieba.node --module_name=nodejieba --module_path=/Users/nishit/Documents/GitHub/docusaurus3/node_modules/nodejieba/build/Release --napi_version=6 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v72' (1)
node-pre-gyp ERR! stack at ChildProcess. (/Users/nishit/Documents/GitHub/docusaurus3/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack at ChildProcess.emit (events.js:315:20)
node-pre-gyp ERR! stack at maybeClose (internal/child_process.js:1021:16)
node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
node-pre-gyp ERR! System Darwin 19.5.0
node-pre-gyp ERR! command "/usr/local/bin/node" "/Users/nishit/Documents/GitHub/docusaurus3/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /Users/nishit/Documents/GitHub/docusaurus3/node_modules/nodejieba
node-pre-gyp ERR! node -v v12.18.0
node-pre-gyp ERR! node-pre-gyp -v v0.14.0
node-pre-gyp ERR! not ok
Failed to execute '/usr/local/bin/node /usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/Users/nishit/Documents/GitHub/docusaurus3/node_modules/nodejieba/build/Release/nodejieba.node --module_name=nodejieba --module_path=/Users/nishit/Documents/GitHub/docusaurus3/node_modules/nodejieba/build/Release --napi_version=6 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v72' (1)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: node-pre-gyp install --fallback-to-build
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/nishit/.npm/_logs/2020-10-21T08_08_50_492Z-debug.log

[weareoutman]

@dhitimedin I think this is not the right place to trouble shooting this error. Could you please fire an issue at easyops-cn/docusaurus-search-local?

[weareoutman]

@dhitimedin I just moved nodejieba from dependencies to peerDependencies (which is only required for language of zh), and a new version has published (v0.7.0), you can try it. It means that you will not get nodejieba installed by default, and the install error above would be no longer an issue.

For further questions, please fire issues at easyops-cn/docusaurus-search-local, thank you.