Fightpandemics: Post Search results are not matching the search criteria.

[rabbi08]

1. Open STG, and navigate to Help Board.
2. Enter search text 'HelpNextDoor.net' and hit to search

You will observe that 7 records are displayed. But only 2 post have actual searched text. All other post have no matching text. Refer below GIF.

More test data :

• Wrong text is matched with the search text, I search for 'The novel' but it also search the words as 'further', 'Other', refer below screenshot
• Also while searching multiple words, search is performed as each single word like in the screenshots
  

[joshmorel]

Yeah I guess this is based on how we want to preference our search whole words or not.

The HelpNextDoor.net one is actually maybe okay? It's splitting on . and the 2nd one matches Frenalacurva.net

But generally, I'm not sure if this okay enough for now (with future refinement) or we want to fix before merging into prod.

[Zaydme]

@rabbi08 I believe all of the issues listed here were fixed in https://github.com/FightPandemics/FightPandemics/pull/1695
Please confirm so I can close this issue. (just to keep what's fixed and what's not clear.)

_please note that we don't share the same db with staging, so the posts are different_

[rabbi08]

@Zaydme

1. For non-logged in User: While searching for 'HelpNextDoor.net' I got 2 post for the search result, where first one is expected but in the second one, the matched text i.e. '.net' is not highlighted. Please see below screenshot.

For logged in user: I got single search result only with highlighted text the exact match.

1. Two word search: Search is performed for words which contains more than 2 letters:
   

2. the word 'here' is searched in 'there'. Refer below GIF
   

[Zaydme]

@rabbi08 if we changed 2. then 3. will get worse.

This how it works now:

• if using NON english letters, all words above 2 characters length are searched, with partial matching: example: "щв" will match entries containing "шв" and "швц".
• if using english letters there are two cases:
  
  
  
  • words less than 3 chars, will only use full matching, so "he" will only match posts with "he" and not with "the" or "help".
  
  
  • words more than 3: will use full and partial matching, so "mask" will match "masks" and "mask".
  
  

---

When the user is not-logged in, none of this applies, in this case the search is always full match because when user is not logged in we use mongodb full $text search which doesn't have partial matching. This explains 1, we have no control over how mongodb does the search, but I can guarantee it will always return the most relevant results.

[rabbi08]

@rabbi08 if we changed 2. then 3. will get worse.

This how it works now:

• if using NON english letters, all words above 2 characters length are searched, with partial matching: example: "щв" will match entries containing "шв" and "швц".

• if using english letters there are two cases:

  • words less than 3 chars, will only use full matching, so "he" will only match posts with "he" and not with "the" or "help".
  • words more than 3: will use full and partial matching, so "mask" will match "masks" and "mask".

When the user is not-logged in, none of this applies, in this case the search is always full match because when user is not logged in we use mongodb full $text search which doesn't have partial matching. This explains 1, we have no control over how mongodb does the search, but I can guarantee it will always return the most relevant results.

@Zaydme
This explanation make sense, thanx a lot for the details.

I have below query:
for using English letters there are two cases:

• words less than 3 chars, will only use full matching, so "he" will only match posts with "he" and not with "the" or "help".
  User should be able to find 'find me' both words together?

• words more than 3: will use full and partial matching, so "mask" will match "masks" and "mask".
  Can it be like for English, we should perform search matching from the beginning of the word?

One last ques :)

• in first screenshot, in second post for non logged in user, '.net' is searched but not highlighted?

[Zaydme]

@rabbi08

• yes, when searching "find me" user will see posts with "find me" and both words are included in the search and will be highlighted, but as another example posts with "find medical help" in this case only the keyword "find" is used and "medical" will not be highlighted.

• Yes, we can force the matching to start from the beginning of the word, and that's actually a good idea, so for example "here" will not match "there" and that solves one of our problems.

[rabbi08]

But for 'find me' keyword search -> both searched word are exact match but both are not highlighted. Please see below screenshot.

[Zaydme]

@rabbi08 For the highlighting, when the user is logged in it should work perfectly.
Because the highlighting functions knows what words were used in the regex (logged in) search, and it highlights them.
But when the user is not logged in, we said that we use mongodb full $text search, the highlighting function doesn't know how mongodb will perform that search, since we have no control over it, so it doesn't know what words to highlight, and it uses the same rules as "logged-in", in this case it didn't highlight ".net" because it's only 3 (net) characters long, and it doesn't fully match the word, so it thinks it wasn't used in the search, but mongodb actually used it.
But I think we can "fix" this, and make it so when the user is not logged in tell the highlighting function to also highlight words under 3 characters length even if they don't fully match, that will solve the issue for "most" cases, but the highlighting will never be perfect like the logged-in, but good enough.

[Zaydme]

But for 'find me' keyword search -> both searched word are exact match but both are not highlighted. Please see below screenshot.

@rabbi08
I just looked at the code again and I see that I also made it if using english letters it only counts keywords strictly longer than 2, so:

• anything less or equal to 2 is completely excluded.
• anything above 2 but less than 3 is using full match.
• anything strictly above 3 is using partial/full match.

Do you think we should remove the "less or equal to 2 is completely excluded" and maybe change it to 1?

[rabbi08]

But for 'find me' keyword search -> both searched word are exact match but both are not highlighted. Please see below screenshot.

@rabbi08
I just looked at the code again and I see that I also made it if using english letters it only counts keywords strictly longer than 2, so:

• anything less or equal to 2 is completely excluded.
• anything above 2 but less than 3 is using full match.
• anything strictly above 3 is using partial/full match.

Do you think we should remove the "less or equal to 2 is completely excluded" and maybe change it to 1?

I would like to suggest that, we can make it 'less than 2'.

Also, we have very less posts data available in this feature branch i.e. even 'Sourced by FP' post are not there. Can we import some test data?

[Zaydme]

@rabbi08 Yes "less than 2" would be better.

For importing more posts there is a script that does that but I can't run it, I think @mannykary can help us with that maybe.