Elasticpress: Custom Analyzer

[GhostToast]

This is definitely a "would be nice" item:

To be able to associate relevant terms from within WP dashboard. What I mean is terms which have synonymous meaning within the context of a particular site and would help users find content based on the potential intention of their words.

As an example, let's say there is a restaurant review site which writes about a bar in Chicago. Only, they insist on referring to it as being in "Chi-Town" or "The Windy City" or "Second City" and never actually call Chicago by the real name. A user wants to find reviews for "Chicago Bars" and no results are found. I know this is an edge case, but one can find other uses, such as abbreviations. (New York City: "NYC").

Would be amazing to someday plug in these pieces of info directly into a WP dashboard. I may investigate doing this manually with a custom analyzer for an upcoming project which deals with medical issues (and their common abbreviations).

[GhostToast]

Related reading: http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-analyzers.html

[tlovett1]

Definitely an idea. Please post the code you use to add the custom analyzer. Thanks!

[GhostToast]

More related reading: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-synonym-tokenfilter.html

[GhostToast]

Some basic synonym stuff:

``````

namespace SECRET\CLIENT\EP;

/**

• Custom mapping for ElasticPress
  *
• @param $mapping
  *
• @return mixed
  */
  function elasticpress_config_mapping( $mapping ) {
  if ( ! isset( $mapping ) || ! is_array( $mapping ) ) {
  return false;
  }

if ( ! isset( $mapping['settings']['analysis']['filter'] ) || ! is_array( $mapping['settings']['analysis']['filter'] ) ) {
    return false;
}

if ( ! isset( $mapping['settings']['analysis']['analyzer']['default']['filter'] ) || ! is_array( $mapping['settings']['analysis']['analyzer']['default']['filter'] ) ) {
    return false;
}

$mapping['settings']['analysis']['filter']['secret_client_synonym_filter'] = array(
    'type' => 'synonym',
    'synonyms' => array(
        'nyc,new york,new york city',
    ),
);

$mapping['settings']['analysis']['analyzer']['default']['filter'][] = 'secret_client_synonym_filter';

return $mapping;

}
add_filter( 'ep_config_mapping', 'SECRET\CLIENT\EP\elasticpress_config_mapping', 10, 1 );```

``````

So this burrows into the default analyzer. Alternatively, one could write multiple analyzers but I believe you would have to decide which one you're using at read and/or write time, so I just globbed onto this one. This is somewhat greedy, however, as now if I search for "nyc" I do find posts about "New York City" but I also just find posts about "new" or "city" as well. I will do further research to see if the expansion can be controlled more.

[tlovett1]

Awesome start!

[tlovett1]

@GhostToast perhaps it's worth adding the custom analyzer thing to the wiki?

[GhostToast]

Added https://github.com/10up/ElasticPress/wiki/Synonym-Analyzer @tlovett1

[msh134]

@GhostToast I have used the Synonym Analyzer code but unfortunately not getting the expected results. For an example, if people searches with "WHO" or "World Health Organization", the result should show posts about "WHO" and "World Health Organization". Here is my code used in the filter you have shown.

$mapping['settings']['analysis']['filter']['secret_client_synonym_filter'] = array( 'type' => 'synonym', 'synonyms' => array( "who,world health organization" ), );

I have also tried with "who => world health organization" but the result is the same.

I am using ElasticPress 3.1.1 and Elasticsearch 6.3.1 (Amazon Elasticsearch Service). Please let me know if I need to make any changes to get the expected result. Thanks.

[tott]

@msh134 did you make sure that you also did a full reindex of the site either through the admin interface or in wp-cli with the --setup flag. If you did, can you share the code you used to alter the search behavior

[msh134]

@tott sorry for the delayed response. Yes, I have run the full reindex from the admin interface. Here is the code I used. It is similar to the provided solution just changed the synonym list.

 function configMapping($mapping){
        // bail early if $mapping is missing or not array
        if ( ! isset( $mapping ) || ! is_array( $mapping ) ) {
            return false;
        }

        // ensure we have filters and is array
        if (! isset( $mapping['settings']['analysis']['filter'] )
            || ! is_array( $mapping['settings']['analysis']['filter'] )) {
            return false;
        }

        // ensure we have analyzers and is array
        if (! isset( $mapping['settings']['analysis']['analyzer']['default']['filter'] )
            || ! is_array( $mapping['settings']['analysis']['analyzer']['default']['filter'] )) {
            return false;
        }

        // define the custom filter
        $mapping['settings']['analysis']['filter']['secret_client_synonym_filter'] = array(
            'type' => 'synonym',
            'synonyms' => array(
                  'who, world health organization',
                  'ac, acme corporation'
            ),
        );

        // tell the analyzer to use our newly created filter
        $mapping['settings']['analysis']['analyzer']['default']['filter'][] = 'secret_client_synonym_filter';
        return $mapping;
    }
add_filter('ep_config_mapping', 'configMapping', 10, 1);

[msh134]

@GhostToast I have used the Synonym Analyzer code but unfortunately not getting the expected results. For an example, if people searches with "WHO" or "World Health Organization", the result should show posts about "WHO" and "World Health Organization". Here is my code used in the filter you have shown.

$mapping['settings']['analysis']['filter']['secret_client_synonym_filter'] = array( 'type' => 'synonym', 'synonyms' => array( "who,world health organization" ), );

I have also tried with "who => world health organization" but the result is the same.

I am using ElasticPress 3.1.1 and Elasticsearch 6.3.1 (Amazon Elasticsearch Service). Please let me know if I need to make any changes to get the expected result. Thanks.

Could you please help me get out of this issue?

[GhostToast]

@msh134 you also have to add the filter itself. to a list of filters to include. so for example in my code I have:

// Define the custom filter.
$mapping['settings']['analysis']['filter']['synonym_filter'] = [
    'type'     => 'synonym',
    'synonyms' => [
        'tv, tvs, television, hdtv',
    ],
];

and then also:

// Tell the analyzer to use our newly created filter.
$mapping['settings']['analysis']['analyzer']['default']['filter'][] = 'synonym_filter';

it is not enough to add the filter if it isn't added to a list of filters to be used. this adds it to the default list, but you may want to add it to a different list (but then you'd need to, elsewhere, tell EP which filterset you want to use).

[msh134]

@tott sorry for the delayed response. Yes, I have run the full reindex from the admin interface. Here is the code I used. It is similar to the provided solution just changed the synonym list.

 function configMapping($mapping){
        // bail early if $mapping is missing or not array
        if ( ! isset( $mapping ) || ! is_array( $mapping ) ) {
            return false;
        }

        // ensure we have filters and is array
        if (! isset( $mapping['settings']['analysis']['filter'] )
            || ! is_array( $mapping['settings']['analysis']['filter'] )) {
            return false;
        }

        // ensure we have analyzers and is array
        if (! isset( $mapping['settings']['analysis']['analyzer']['default']['filter'] )
            || ! is_array( $mapping['settings']['analysis']['analyzer']['default']['filter'] )) {
            return false;
        }

        // define the custom filter
        $mapping['settings']['analysis']['filter']['secret_client_synonym_filter'] = array(
            'type' => 'synonym',
            'synonyms' => array(
                  'who, world health organization',
                  'ac, acme corporation'
            ),
        );

        // tell the analyzer to use our newly created filter
        $mapping['settings']['analysis']['analyzer']['default']['filter'][] = 'secret_client_synonym_filter';
        return $mapping;
    }
add_filter('ep_config_mapping', 'configMapping', 10, 1);

@GhostToast I had added the filter to the default list and ran full reindex from admin interface but did not get the expected results. Please see my code above and let me know if I need to look into anything.

[GhostToast]

@msh134 I'm not sure - I'm still using an older version of ElasticPress and Elasticsearch, so it's possible the pathways have changed. You may need to read up on how synonym filters work for your version of Elasticsearch, and do some debugging later in EP to see what the mappings look like on their way out from within EP. I also wonder if your synonyms are too short to unfold. Can you try something more robust? For example try synonymizing "apple" to "banana" as a test.

[GhostToast]

Possibly helpful: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/analysis-synonym-tokenfilter.html

[scooterlord]

Hello, this is an old post, but did you manage to make this work?

[brown-a2]

Hello, I'm having a lot of trouble trying to modify the default analyzer.

If I try to replace or add to the default analyzer array, ie
$mapping['settings']['analysis']['analyzer']['default']['filter'][] = 'secret_client_synonym_filter';

I get "Error: Mapping failed". I'm hooking into ep_config_mapping.

When I var_dump the array (see below), I can see it has been added to the array, the mapping is just failing.

array(5) {
  [0]=>
  string(18) "ewp_word_delimiter"
  [1]=>
  string(9) "lowercase"
  [2]=>
  string(4) "stop"
  [3]=>
  string(12) "ewp_snowball"
  [4]=>
  string(19) "secret_client_synonym_filter"

I can add my own custom analyzer, but I'd like to add to the default one as this seems to be the predominate one the plugin uses. Or am I approaching this the wrong way around, should I be overriding the default analyzer in some way?

Any suggestions? Thanks

[brown-a2]

Just an update on my above comment. Using the ep_config_mapping_request filter, I was able to see the response coming back from Elasticsearch and it seems that using theewp_word_delimiter and synonyms are incompatible. At least in the above order. I think I'd still the to know the correct approach here, should I even be trying to modify the default analyzer or should I just create my own custom one, and then use that for search queries? Thanks

[scooterlord]

Just an update on my above comment. Using the ep_config_mapping_request filter, I was able to see the response coming back from Elasticsearch and it seems that using theewp_word_delimiter and synonyms are incompatible. At least in the above order. I think I'd still the to know the correct approach here, should I even be trying to modify the default analyzer or should I just create my own custom one, and then use that for search queries? Thanks

You are right, it probably has to do with the order of the filter. I managed to get the synonyms filter to work by directly changing the includes/mappings/post/7-0.php file and placing the synonym filter first.

I was too busy doing other things in the meantime, but at some point I'll have to deal with this issue properly using a wordpress function. If anyone happens to write a similar function in the meantime, please share!

[brown-a2]

So, if it helps others, my final solution to the above was to use array_merge() to add my synonym filter to the default analyzer, but in the first position of the array. So as an example, using the ep_config_mapping hook, and the following code:

$default_filter_array = $mapping['settings']['analysis']['analyzer']['default']['filter'];

$mapping['settings']['analysis']['analyzer']['default']['filter'] = array_merge(
            ['secret_client_synonym_filter'],
            $default_filter_array
        );

@christianc1 that's great you're adding this feature in 👍

[scooterlord]

Hello, I tried to combine the answers above, but still haven't found a way to make this work:

function configMapping($mapping){
     $default_filter_array_analyzer = $mapping['settings']['analysis']['analyzer'];

     $mapping['settings']['analysis']['analyzer'] = array_merge(
         ['synonym'],
         $default_filter_array_analyzer
     );

     $mapping['settings']['analysis']['analyzer']['synonym'] = array(
         'tokenizer'   => 'standard',
         'filter'      => array('synonym_filter'),
      );

     $default_filter_array = $mapping['settings']['analysis']['analyzer']['default']['filter'];

     $mapping['settings']['analysis']['analyzer']['default']['filter'] = array_merge(
         ['my_synonym_filter'],
         $default_filter_array
     );

     $mapping['settings']['analysis']['analyzer']['default']['filter']['my_synonym_filter'] = array(
         'type' => 'synonym',
         'synonyms' => array(
             'oneword,otherword'
         ),
     );
     return $mapping;
}
add_filter('ep_config_mapping', 'configMapping', 10, 1);

Can someone help me out? What am I doing wrong? Is there a way to debug this on-screen?

Edit: Answering my own question... I was trying to add the synonym_filter inside the analyzer - which was obviously wrong - and I also realized by carefully examining the code that I don't need to create a new analyzer, but rather use the default one instead.

Here is the final code for anyone that wants a working answer to the same potential question:

function configMapping($mapping){
     $default_filter_array = $mapping['settings']['analysis']['analyzer']['default']['filter'];

     $mapping['settings']['analysis']['analyzer']['default']['filter'] = array_merge(
         ['synonym_filter'],
         $default_filter_array
     );

     $mapping['settings']['analysis']['filter']['synonym_filter'] = array(
         'type' => 'synonym',
         'synonyms' => array(
             'λαστιχα,lastixa'
         ),
     );
     return $mapping;
}