Uwazi: Improve search query validation and UX

[fnocetti]

Uwazi currently accepts the Lucene syntax for search queries. A search will fail if Lucene reserved characters are input to a query that doesn't match the syntax. This is particularly inconvenient in searches were every new character triggers a new Search API call (e.g. searching for snippets within a document), which results in a bad user experience since the UI will show an error until the user forms a correct Lucene expression.
We should evaluate:

• If triggering a search for each character is a good approach for those cases and, if it is, evaluate if we need a less disruptive way to inform the error on the UI.
• If we should validate the query and decide how to escape the reserved characters before requesting to the backend.
  https://www.elastic.co/guide/en/elasticsearch/reference/current/search-validate.html might be a good option.

[mfacar]

@txau, @fnocetti, @RafaPolit

_UPDATE:
Currently, the search is only triggered when the user hits enter or click on the search icon._

If a search term has an invalid syntax the client is still having an Elasticsearch error. This is because we are using a query_string full-text search query.

According to this warning in the Elastic documentation "Because it returns an error for any invalid syntax, we don’t recommend using the query_string query for search boxes.", and given that we need to take advantage of the strict query of "query_string", what do you think about handling the search errors caused by an invalid syntax, changing the type of the full-text search to "simple_query_string" only for this cases? I tried this on the code and it works very acceptably, although the search is not as precise as with "query_string", it is returning a good set of results and doesn't break the client.

With this solution when the query doesn't have any syntax errors it works, as usual, giving super strict results, but when for any reason the search term is not valid, we still have approximate results.

[fnocetti]

@mfacar will simple_query_string tolerate malformed syntax like opening a Lucene operator without closing it?

If that's the case, I would say always try with query_string first and if it fails because of a syntax error (I think we can differentiate it, but not sure) fallback to simple_query_string.
It will for sure have some performance implications, but even validating the query will have them. Maybe we can throw both searches in parallel in use the more strict one that have results. Just thinking out loud.

[mfacar]

Thanks @fnocetti.

@mfacar will simple_query_string tolerate malformed syntax like opening a Lucene operator without closing it?

Yes, it does

If that's the case, I would say always try with query_string first and if it fails because of a syntax error (I think we can differentiate it, but not sure) fallback to simple_query_string.

that's the idea

It will for sure have some performance implications, but even validating the query will have them. Maybe we can throw both searches in parallel in use the more strict one that have results. Just thinking out loud.

I agree, I'm testing with two approaches, the first one is to search with query_string and if it fails do the search with simple_query_string, and the second is to validate the search_term before the search and prevent a fail, I'll try to compare the performance impact with these two options.

[mfacar]

I've made some performance tests and although the validation called alone is taken among 10 and 15 ms, it seems that the search call is not being affected significantly when it is validating the query before to execute it. With the second option, however, the invalid queries have a notable delay in the execution, about 40ms.
Based on these results, I'm opening a PR with the first approach.