Mkdocs-material: Search does not work in french language

[kevung]

__I checked that...__

• [x] ... the documentation does not mention anything about my problem
• [x] ... the problem doesn't occur with the default MkDocs template
• [x] ... the problem is not in any of my customizations (CSS, JS, template)
• [ ] ... there are no open or closed issues that are related to my problem

Description

Search word in with mkdocs-material fir french language does not work properly

Expected behavior

Simple word such as 'plage' should be found.
Accented words are not consistently found.

Actual behavior

'plag' is found, but not 'plage'

Steps to reproduce the bug

1. Install mkdocs-material (5.2.2)
2. Use this mkdocs.yml file:

site_name: My Docs
theme:
  name: material
  language: fr

plugins:
  - search

and this ./docs/index.md

Cet été ensoleillé, nous irons à la plage.

1. serve mkdocs (python -m mkdocs serve) and search for plage.

Package versions

• Python: `3.8.3
• MkDocs: 1.1.2
• Material: 5.2.2

Project configuration

site_name: My Docs
theme:
  name: material
  language: fr

plugins:
  - search

System information

• OS: Linux 5.6.15-arch1-1 x86_64 GNU/Linux
• Browser: firefox

[squidfunk]

As already noted in #419, it's very likely related to #1603 and #1675. Please, see the comments on these issues.

[kevung]

@squidfunk Thank you very for your reply. I will just put here some simple tests here.
Here are the following tests I made. I use each time the same ./docs/index.md file:

Cet été ensoleillé, nous irons à la plage.

I want to search the word plage.

• With this docs.yml, the search is correct:

site_name: My Docs
theme:
  name: material
  language: en

plugins:
  - search

• With this docs.yml, plag is found but plage is not found:

site_name: My Docs
theme:
  name: material
  language: fr

plugins:
  - search

[squidfunk]

As noted in the comments, it's not fixable without losing the type-ahead experience.

[kevung]

I use each time the same ./docs/index.md file:

Cet été ensoleillé, nous irons à la plage.

With this docs.yml:

site_name: My Docs
theme:
  name: material
  language: en

plugins:
  - search:
      lang:
        - fr

the word ensoleilléis properly found, but à is not. Maybe it is because à begins with an accent.
(same result with language: en)

[kevung]

I use each time the same ./docs/index.md file:

Cet été ensoleillé, nous irons à la plage.

With this docs.yml:

site_name: My Docs
theme:
  name: material
  language: en

plugins:
  - search:
      lang:
        - en

the word irons à is properly found,
but when activating the french option in search plugin:

site_name: My Docs
theme:
  name: material
  language: en

plugins:
  - search:
      lang:
        - fr

the word irons à is not found any more.
And strangely, with this docs.yml

site_name: My Docs
theme:
  name: material
  language: fr

plugins:
  - search:
      lang:
        - en

the word irons à is also not found.

It seems that there is a tight intrication between language option from theme, and lang from the search plugin

[squidfunk]

I took some time on this Sunday and think I have a fix in place as part of 36e1e3db.

It seems like swapping the pipeline initialization and also resetting the search pipeline (something that was added in lunr 2.0) seems to do the trick for most of the languages. I tried to test as many languages as possible and added some fixes to the default initialization.

[squidfunk]

Note that search is super tricky to get right for 40 languages. This may have introduced new problems I'm not aware of. I tried to test as many languages as I could, but unfortunately, my time is limited.

[squidfunk]

Released as part of 5.2.3

[kevung]

@squidfunk Thank you so much for your help, and for your work!
According to search plugin doc page, the language chosen for the theme sets the language for the search.

I tested a minimal example on the last Mkdocs-material release (5.2.3) with:

sudo pip install mkdocs-material --upgrade

In this minimal example:

• ./docs/index.md:

Cet été ensoleillé, nous irons à la plage.

• mkdocs.yml:

site_name: My Docs
theme:
  name: material
  language: fr

plugins:
  - search

nav:
  - index.md

the word "nous" connot be found.

screenshot

[squidfunk]

This is because "nous" is considered a stopword in the French language and filtered out. See:
https://github.com/MihaiValentin/lunr-languages/blob/a62fec97fb1a62bb4581c9b69a5ddedf62f8f62f/lunr.fr.js#L699

[kevung]

Mmm, I closed mkdocs serve, refresh, did a lot of things... and in exactely the same config as https://github.com/squidfunk/mkdocs-material/issues/1745#issuecomment-640232868
I can now find "nous" !
Haha I am a bit confused :)
screenshot

The search functionality is very important for a web doc, so maybe id it wiser to mark the multi-language search as experimental?

I think I will set the language as english by default as it might give the most stable results. Thank you again for help. If I can be of any support for testing, I'd be happy to help.