Addons-server: Improve relevancy of search results
Have a quick look at the analyzers and filtering code we're using to see if we could improve relevancy of results. Maybe tweak boosts a little, etc. Draw inspiration from Marketplace.
diox
All 36 comments
I just found https://blog.algolia.com/algolia-v-elasticsearch-relevance/, not completely relevant but it's at least a good source of inspiration and mentiones ElasticSearch quite a bit (despite being obviously about Algolia but who can blame them)
EnTeQuAk
on 16 May 2017
I like the extensive definitions of custom analyzers which we might want to look at https://github.com/mozilla/zamboni/blob/master/mkt/search/indexers.py#L168-L227
EnTeQuAk
on 15 Aug 2017
might even be worth a separate issue but I stumbled upon the fact that we define boost factors of 0.X (e.g 0.8 for summary matches, 0.3 for description matches etc). But given that all these boost factors are by default getting multiplied we actively reduce the score for matches in these fields. This isn't documented like that so I feel like this was an accident. @diox any idea why we do that?
tag matches are also one factor where we effectively reduce the score.
For ES 1.7: https://www.elastic.co/guide/en/elasticsearch/reference/1.7/query-dsl-function-score-query.html#_using_function_score
For ES 5.X: https://www.elastic.co/guide/en/elasticsearch/reference/5.5/query-dsl-function-score-query.html#query-dsl-function-score-query
Both have the default of multiply so this is weird. Has also been the default since 0.x so that never changed.
EnTeQuAk
on 15 Aug 2017
Another thing: We also do not have any exact match preferring rule in there that would immediately boost exact add-on name matches. I feel like that could be a useful feature, especially if people are actively searching for an add-on that isn't used that much used or potentially not as SEO described as others but insists he wants that because a friend of a friend recommended it.
EnTeQuAk
on 15 Aug 2017
Regarding the boost values, I have no idea, it does feel like an accident. Do check the _score in results to confirm your findings, it seems like a weird oversight...
In any case, yes, a rule for exact matches would be great (there is one in Marketplace, it needs a raw, non analyzed version of the name.)
diox
on 16 Aug 2017
Regarding the boost values, I have no idea, it does feel like an accident. Do check the _score in results to confirm your findings, it seems like a weird oversight...
yeah, I am currently doing that. Especially reading through _explain is tiring but from what I see it seems that it's more an accident since we essentially lowering the score.
I'm writing tests for that though to verify my findings.
EnTeQuAk
on 16 Aug 2017
When attempting to add to a collection, I often have to enter the numeric ID of an extension (after failure to find with all name-based attempts). Should I add an example to this issue? Or raise a new issue in this repo?
grahamperrin
on 27 Aug 2017
More data never hurt to get more information about search behavior, so
an example is greatly appreciated!
On Sun, Aug 27, 2017, at 12:51 AM, Graham Perrin wrote:
When attempting to add to a collection, I often have to enter the
numeric ID of an extension (after failure to find with all name-based
attempts). Should I add an example to this issue? Or raise a new issue
in this repo?> — You are receiving this because you commented. Reply to this email
directly, view it on GitHub[1], or mute the thread[2].>
Links:
EnTeQuAk
on 27 Aug 2017
Collections later.
First, here's a more basic failure:
https://addons.mozilla.org/en-Gb/firefox/search/?q=british+spell&appver=55.0&cat=1%2C0 fails to find either British English Dictionary or British English Dictionary (Marco Pinto):
Those two are found only with find as you type (FAYT) with same search phrase in the _search for add-ons_ field without keying Enter or Return:
The requirement to use FAYT alone is frustrating because Control-Click on any result causes all results to disappear.
grahamperrin
on 30 Aug 2017
https://addons.mozilla.org/en-Gb/firefox/search/?q=british+spell&appver=55.0&cat=1%2C0 fails to find either British English Dictionary or British English Dictionary (Marco Pinto):
That's because the default search only searches through extensions. If you select "All Add-ons" it works.
diox
on 30 Sep 2017
… only searches through extensions. If you select "All Add-ons" …
https://addons.mozilla.org/en-Gb/firefox/search/?q=british+spell&appver=55.0&cat=1%2C0 does already show 'All Add-ons':
Is that the problem? Is what's shown, top left, misleading and if so, where else should All Add-ons be selected?
grahamperrin
on 1 Oct 2017
It's a bug in the frontend, the wrong option is pre-selected. If you re-click on it once it correctly shows All Add-ons. It should go away in the new frontend that we'll push to production soon.
diox
on 1 Oct 2017
Thanks, I found it necessary to click more than once.
… the new frontend …
Is that what's currently known as the _Mobile Site_? (I've had the blue favicon, rotated 45°, in lieu of the green, for the non-mobile view for quite a while now.)
grahamperrin
on 1 Oct 2017
... the new frontend …
Is that what's currently known as the Mobile Site?
Yes. Eventually the new frontend will power both mobile and desktop, since it's responsive. Currently it's only enabled if you click the "View Mobile Site" link in the footer or if you visit with a mobile user-agent string.
(I've had the blue favicon, rotated 45°, in lieu of the green, for the non-mobile view for quite a while now.)
That has nothing to do with it, the favicon was changed to the blue/turquoise one across the board.
diox
on 1 Oct 2017
… We also do not have any exact match preferring rule …
I'm always/often surprised when I type exactly and don't get an exact match.
(That's one of the reasons for me sometimes pleading with developers to not have two, three or more names for a single add-on.)
grahamperrin
on 15 Oct 2017
Is it my imagination, or has exact matching improved (or been implemented) in the past three or four days?
(I was preparing to add an example of something that I recall being not matched … no example at the time of writing.)
_tab center redux_
Not found: (about:addons in Firefox 56.0.1):
First and foremost in the front-end, which I know will change:
grahamperrin
on 19 Oct 2017
Does _categorisation_ affect relevance in search results?
I ask because it's easy to see (for example) many things that are miscategorised under _tabs_.
grahamperrin
on 30 Oct 2017
Does categorisation affect relevance in search results?
It doesn't affect relevance as we don't boost on categories, we do have a very low boost on tags though.
EnTeQuAk
on 6 Nov 2017
https://addons.mozilla.org/search/?q=Tabmarks&type=extension&sort=name&page=35 finds _Tabmarks_ on page 35 (at the time of writing). Related, again:
https://addons.mozilla.org/search/?q=Tabmarks&type=extension&sort=relevance does not find the relevant extension.
Use case
https://mozilla.logbot.info/firefox/20171118#c13877033 today,
Is there a way to search extensions marked experimental? I'm pretty sure I read through every extension in the tab section and didn't see this, and it doesn't even show up when you search for Tabmarks.
grahamperrin
on 18 Nov 2017
Seeking _Scrollmap_ fails to find Scrollmap.
grahamperrin
on 29 Dec 2017
https://addons.mozilla.org/firefox/search/?q=Tab+Data+(%2BMemory+usage) should place Tab Data (+Memory usage) in first place.
Instead, it's eighth.
https://addons.mozilla.org/firefox/search/?q=Tab+Data – note, the installed extension is named _Tab Data_ – fails to place the extension on the first page of results.
grahamperrin
on 29 Dec 2017
Seeking Scrollmap fails to find Scrollmap.
That add-on is not compatible with Firefox 57 or higher and it's also marked as experimental, both things which severely reduces its chances to appear in the first pages of results. You'd find it if you were on Firefox 56 or lower, in page 5 or so.
Off-topic here, but funny enough I made a similar extension that works with Firefox 57+: https://addons.mozilla.org/en-US/firefox/addon/minimap-scroller-sidebar/
Also expected since it's not a WebExtension.
diox
on 29 Dec 2017
Scrollmap found on page 4.
It's _nearly always_ infuriating when, for _any_ reason, seeking an exact word does not treat the exact match as most relevant. Three pages of irrelevance 👎 can't be good UX.
grahamperrin
on 29 Dec 2017
I started a Search PRD late last year with these type of improvements in mind. This is not a final draft, but it may help frame some thoughts or clarify priorities around how we can improve AMO search.
https://docs.google.com/document/d/1mu4f8VLlkcJSzFRcNk0LUMu4cHQfeX58OeF2H7wmYa8/edit#
devaneymoz
on 11 Jan 2018
Awesome, that helps a lot I think
EnTeQuAk
on 11 Jan 2018
Could you make the doc public? I've been following the development of the server and the new frontend and there are some non-public docs linked in issue and pr discussions. We end up missing a lot of context when that happens.
dessant
on 11 Jan 2018
@dessant sure, anyone can view it now.
devaneymoz
on 11 Jan 2018
Thanks @devaneymoz! 😋
dessant
on 11 Jan 2018
This should have been significantly better with https://github.com/mozilla/addons-server/pull/7303 which will land this Thursday and also contains one more experiment behind a flag that we can use to test more scoring improvements. So I'm removing the milestone, un-assigning myself and leave this open for general discussions, e.g for the PRD. I could argue that we should open a new issue once the PRD get's to a more finalized state since it contains quite a lot of search-scenarios that we do not implement these days.
Opinions?
EnTeQuAk
on 30 Jan 2018
Search performed with Firefox 58.0, for _Vertical Tabs Reloaded_
– five pages of results, none of which is https://addons.mozilla.org/en-GB/firefox/addon/vertical-tabs-reloaded/
Am I missing something again?
grahamperrin
on 10 Feb 2018
Looks like it's an issue with your platform - it works fine on the legacy website using your link, or on the new site using "linux" as the platform instead of "FreeBSD". I'd file a bug over at addons-frontend, if there isn't one already.
diox
on 10 Feb 2018
… works fine on the legacy website using your link …
Confirmed, thanks.
With the classic site, and the same link, still with Firefox on FreeBSD-CURRENT, the required extension is first and foremost.
grahamperrin
on 10 Feb 2018
It's half-funny :-) that a search for _font_ finds _Drudge-Fakenews_ – https://addons.mozilla.org/addon/drudge-fakenews/reviews/1120505/. On one hand, I love occasional serendipity.
On the other, I _would_ love results to be more relevant by default …
grahamperrin
on 13 May 2018
… tag matches …
Results of tag-based searches were sometimes more useful – more relevant, e.g.
https://addons.mozilla.org/search/?tag=sessionhttps://addons.mozilla.org/search/?tag=sessions
– but I suspect that use of (or application of) tags is somewhat deprecated.
grahamperrin
on 23 Sep 2018
I've started looking into search relevancy issues on and off while I'm at Paris-Web conference this week. I've got a branch that adds the score and explanation in results, I'll clean it up and submit a PR to add this to the API soon. Meanwhile... digging into this has already helped me confirm a few things @EnTeQuAk suspected:
- Some of our boosts are lowering the score for some matches. Essentially all the rules in
secondary_should_rules()inSearchQueryFilterare wrong, because the boosts they apply are between 0 and 1, and we're multiplying the score for each term it's applied to with each boost, so the final score is lowered. The final score is a sum of all intermediary values, but those values are the ones being lowered. - On dev/stage/prod we have multiple elasticsearch shards and that's severely affecting scoring, because each shard has its own statistics for term frequencies. That makes some results show up higher that they should be compared to the rest, depending on how lucky they are (because they are the most relevant for their shard)
Adding the score & explanation to the API and fixing those 2 things should significantly improve things. Then, we can worry about things like better handling of stopwords, improving language-specific analyzers, and looking more extensively at the queries users are making and tweaking specific scenarios. Also worth noting, we will need to switch back the similarity algorithm away from classic mode eventually as it has been deprecated.
diox
on 5 Oct 2018
Closing this issue because it doesn't have anything actionable in it, we have a top-level project to track search relevancy improvements and a lot of individual issues about specific problems we need to fix.
diox
on 24 Oct 2018
Related issues
jvillalobos
·
3Comments
Juraj-Masiar
·
5Comments
kewisch
·
7Comments
Juraj-Masiar
·
6Comments
ValentinaPC
·
4Comments
Most helpful comment
Another thing: We also do not have any exact match preferring rule in there that would immediately boost exact add-on name matches. I feel like that could be a useful feature, especially if people are actively searching for an add-on that isn't used that much used or potentially not as SEO described as others but insists he wants that because a friend of a friend recommended it.