Jetpack: Search: Integrate Search Suggestions (Auto correct)

[gibrown]

The primary purpose here is to handle spelling mistakes in searches. I think we should use the Elasticsearch phrase suggester (not perfect, but good enough for now) on every search request when this feature is turned on (should default to off for now).

Suggest Logic

I think the following pseudocode makes sense and covers the various cases:

results = get_search_results_from_api( query = user_query, get_suggestions = true )
if ( results.hits are not empty ) {
  display whatever results we have
  if ( results.suggestions are not empty ) {
    display the suggestion
  }
} elseif ( results.suggestions are not empty ) {
  results = get_search_results_from_api( query = suggested_query, get_suggestions = false )
  display a message saying no results for user_query, showing results for suggested_query
  display whatever results we have
} else {
  display no results
}

Suggest Display

For the display of the suggestions there are two cases to display (X = user query, Y = suggestion):

• "Did you mean: Y" - X is a link
• "Nothing found for X. Showing results for: Y" - no links

This should be shown at the top of the search results page and should work in any theme. It should also be in a div so it can be easily styleable.

This page doesn't have the correct logic, but does show it:

Admin Interface

A toggle for turning this feature on should be added to Calypso. It should be marked experimental. I am not confident that the query below will work well on the global search index. I am pretty sure it works ok on VIP indices.

Suggest Query

Here is the suggest portion of the query I'd suggest we use:

  "suggest": {
    "main-suggest": {
      "text": "huson bay",
      "phrase": {
        "field": "mlt_content",
        "direct_generator":[{
          "field":"mlt_content",
          "prefix_length":2,
          "suggest_mode":"missing"
        }]
      }
    }
  },

Notes on development: you may run into some API issues with implementing this. Please feel free to ping me on this thread, in WP.org slack, or anywhere else to get some help.

[gibrown]

Supporting this feature in any theme is complicated by there being no good way to output the results without modifying the theme by customizing search.php. One possible method would be to insert a fake post into the stream. May be a can of worms though.

[drywall]

No progress on this can of worms, eh?

[gibrown]

@drywall not yet unfortunately

[drywall]

@gibrown That's okay. Mostly I'm wondering if using Jetpack Pro for search we can manually implement something like this in custom code...

[gibrown]

@drywall sorry for the slow response. Yes in theory. Let me verify that I can unblock the suggest portion of the API for you. May take a few days.

You can steal some code from wpcom-elasticsearch VIP plugin if you search for "suggest". But I'd suggest using the params I mentioned above and using the content field instead of mlt_content (mlt_content is undocumented because it will probably change).

[drywall]

@gibrown You're my new hero, thank you!

[gibrown]

@drywall OK, i've unblocked the suggest params and added some tests. You should be able to start experimenting with it.

Some thoughts:

• We don't have any fields that have shingles on them (as in the ES docs for the phrase suggester: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/search-suggesters-phrase.html), but I think it may still work ok.
• The Jetpack Search index is shared (https://greg.blog/2014/02/06/scaling-elasticsearch-part-2-indexing/) so on the shard that has your site on it, there will also be 10s of thousands of other sites. Because suggest works on a per shard basis, this means you may get suggestions that will not actually match the data on your site. This may be good or bad. It allows you to correct for misspellings even if that misspelling will not match anything on your site. I think if you auto-rerun the search as suggested above it is probably good. If you find that it is problematic then let me know what you are seeing and I have a few other ideas.
• Because the index is shared, which field you choose to suggest on matters in different ways. "content.en" is very likely to only have words that occur on English sites. "content.default" will have words from all languages.

I'm excited to see how it goes for you. Let me know if there is anything at all I can help with.

[gibrown]

Rather than inserting a "fake post", let's try just using the loop_start action to echo out the necessary html before any posts are output: https://github.com/WordPress/wordpress-develop/blob/8df8cf2df14fe26174f97af5bb17d63f2e867231/src/wp-includes/class-wp-query.php#L3226

[anaemnesis]

Requested 1600099-zen

[csonnek]

Requested 2174715-zen

[gibrown]

The referenced pull request does implement this, but there are still some issues blocking merging it. Right now we are working on implementing it in another way, but it will end up as a bigger change.

[jonathanstegall]

I noticed in #10023 that it mentions a "good version of this getting launched." Did this ever happen? If so, is it documented anywhere how we can activate/use it? I host on VIP Go, but I realize this may or may not be relevant still.

[gibrown]

Hi @jonathanstegall ya so this been launched in the new version of JP Search: https://jetpack.com/upgrade/search

It looks like this:

And has a fairly different architecture that makes auto corrections actually work efficiently.

It is in a different product with different pricing and not currently included in VIP by default though it does work there. I think you would need to discuss with them what the best way to add this would be.

Happy to answer more questions here about it too. I'm going to close this issue though since I had apparently forgotten to do so and we don't have any plans to port this back to the version of Jetpack Search that is included in the Pro plan.