Sanity: Better support for ranked full text search

[bjornwang]

Is your feature request related to a problem? Please describe.
Many solutions developed with Sanity require full text search of some sort. Seeing as Sanity is using Elasticsearch under the hood, it seems a waste to have to set up an external search engine to index Sanity content for full text search.

The GROQ 'match' operator is a good starting point, but is lacking with regards to ranked sorting of the result set. Haven't found any clear documentation, but 'match' seems to operate as a GROQ filter that doesn't affect sort order any particular way. This is as expected.

Comparing Sanity to other "content management" competitors, search is an area where Sanity feels a bit lacking currently. Better free text search options would be great.

Describe the solution you'd like
Ranked search across joined document types is a non-trivial problem, but small extensions to GROQ might go a long way.

1. The most important thing would be to allow sorting result sets after text search relevance, with the best hits first. This might entail making the Lucene / Elasticsearch scoring available somehow so the query could explicitly specify sorting by relevance, or it could be the default sort order if nothing is specified.

2. As with the experimental settings for Studio search, it would be great to be able to tune GROQ search by specifying boost value for specific fields. For example, a hit in a title field should probably give a better relevance score than a score in a general body text field.

I expect scoring GROQ queries with joins will be the toughest part, depending on how your Elasticsearch indexes are set up, so I'll refrain from suggesting actual solutions and rather stick to just specifying the need.

Describe alternatives you've considered

1. Setting up an external search engine. Feels a bit wasted knowing that Sanity builds on Elasticsearch. Also involves setting up schemas that feel duplicate, pushing data around and keeping it in synch.

1. Hacking scoring in GROQ. We've faked improved scoring by looking at which field gives a hit and sorted the result set based on this. Helps a bit, but not enough.

2. Doing search client-side, returning all data from Sanity to the client. This is what we've opted for currently, as a quick and dirty solution, as we're searching a limited set of documents.

[evenwestvang]

Thank you for your suggestion @bjornwang! This is indeed something that we are looking to implement and we will be treating this as a cogent & vocal +1 for a more comprehensive and weighted full-text search.

[bjornwang]

@evenwestvang Any progress on this one? We're still postponing setting up alternative indexing in our end hoping something might come out of Sanity.

In our project, we've split our core objects into multiple doc types in Sanity (to restrict access to parts of the objects), so a good approach for us would be the ability to configure GROQ queries to be indexed for free text search. The GROQs would then do all the convoluted joins and serve nice, flattened result sets for Elasticsearch to index which would then also be much easier to rank intelligently.

Such configured queries could possibly also be handled as "views" (like in an SQL database), which would allow for them to be referenced in GROQ queries for further joining. But this would of course lead back to the original problem of how to rank free text hits intelligently across complex joins.

[snozwoz]

Hi, Any progress on this - I realise this feedback is now a year old and maybe is no longer an issue. I am new to Sanity and this feature would be an issue for our requirements

[AHBruns]

Just adding my support for this. I'm unsure if I should build out my own search functionality on-top of Sanity or wait for this feature. Some guidance on the timeline would be appreciated.

[snozwoz]

Just spotted this experimental feature (now 1 year old) https://www.sanity.io/docs/studio-search-config - @evenwestvang does this solve this issue? But what does the future hold for this experimental feature, will it be deprecated or make the final cut?

[AHBruns]

Just spotted this experimental feature (now 1 year old) https://www.sanity.io/docs/studio-search-config - @evenwestvang does this solve this issue? But what does the future hold for this experimental feature, will it be deprecated or make the final cut?

That is for in-studio search (e.g. the search bar in the Sanity dashboard). It doesn't provider search via a GROQ query.

[bryangruneberg]

Also throwing support in here! Having index and ranked search as part of the platform and service would be super.

[bjornwang]

Adding another comment on this on the financial aspect. This is actually stopping Sanity from being adopted on internal projects, as when comparing costs to other platforms search becomes an extra expense.

CC: @evenwestvang, @kmelve