Searchkick: Allow customized filters on query (`where: {}` argument)

Created on 25 Aug 2017 · 2Comments · Source: ankane/searchkick

Hey @ankane, thanks again for this incredible lib, and how this is still up to date to the most recent Elasticsearch changes! :clap:

As the title says, this issue is about how Searchkick transforms its options within the where: {} clause in filters. I got stuck trying to heavily customize my filters when executing some complex queries.

Setup

My example: I have the following Searchkick setup, allowing Elasticsearch to create its native "nested" type, so I can execute nested aggregations:

# On my `Offer` model
searchkick index_name: -> { ['super-custom-name', model_name.plural, Rails.env].join('_') },
         merge_mappings:      true,
         mappings:            {
           offer: {
             properties: {
               tech_specs: {
                 type:       :nested,
                 properties: {
                   name:               { type: :keyword },
                   position:           { type: :integer },
                   value:              { type: :keyword }
                 }
               }
             }
           }
         }

And here it is my aggregations for faceted filtering (which are later deep merged in the payload by the body_options).

{
  body_options: {
    aggs: {
      tech_specs: {
        nested: {
          path: :tech_specs
        },
        aggs:   {
          names: {
            terms: {
              field: :'tech_specs.name',
              order: {
                position: :asc
              }
            },
            aggs:  {
              position: {
                max: {
                  field: :'tech_specs.position'
                }
              },
              value:    {
                terms: {
                  field: :'tech_specs.value'
                }
              }
            }
          }
        }
      }
    }
  }
}

Problem

As I navigated through the Searchkick::Query source code, I didn't find anything about nested types, and I needed to filter out offers by my facet filtering selections. On a raw Elasticsearch query, this is easy!

{
    "query": {
        "bool": {
            "filter": [{
                "bool": {
                    "must": [{
                        "nested": {
                            "path": "tech_specs",
                            "query": {
                                "bool": {
                                    "must": [{
                                            "term": {
                                                "tech_specs.name": {
                                                    "value": "Color"
                                                }
                                            }
                                        },
                                        {
                                            "term": {
                                                "tech_specs.value": {
                                                    "value": "Red"
                                                }
                                            }
                                        }
                                    ]
                                }
                            }
                        }
                    }]
                }
            }]
        }
    }
}

Cool, now the real issue: using Searchckick, how do I add my nested filters to the where: {} clause, besides using the predefined key/values at the Searchkick::Query#where_filters?

https://github.com/ankane/searchkick/blob/3dc4776bb69b5f2c59d29f48fd7edf618d887ee9/lib/searchkick/query.rb#L732

My first naive approach was to use the body_options to do it so, but later, deep_merge just destroyed my searchkick filters array. What I needed was to append the filters array in the payload! Aha!

Workaround

As you describe in the docs: Note: If you use a custom mapping, you'll need to use custom searching as well., so I ended up using a mix of body_options and customizing the payload through the "Advanced Search" body yield syntax, but that ended up with the following "dangerous" override:

# `options` is my series of hash merges to customize Searchkick options (+ body_options)
search = Offer.search(@query, options) do |body|
  filters = body.dig(:query, :bool, :filter) # Yeah, digging it up
  filters << { { bool: { must: {} } } # Hidden here for brevity's sake. Also, if I wanted, I could do some safe programming here and `Array.wrap` it

  body[:query][:bool][:filter] = filters # annnnddd this. If I dig and don't find it, ok! But I cannot ensure the chain `body[:query][:bool][:filter]` will always exist
end

As you can see, it would be "safer" and "cleaner" to use a custom filter on the where: {} clause instead of overriding the payload.

As I wrote on issue #976, using the body_options works for all my customized aggregations, but deep_merge the query + filters array is not possible.

Ideas and Suggestions

Well, after that, I had some ideas about the where clause:

Create a mapping for Elasticsearch nested filters in the Searchkick::Query#where_filters method
- Pro: keeps everything within the "searchkick" way of passing parameters
- Con: since the #where_filters is a recursive call against key/values, this wouldn't allow any further customization on the given filter
Allow it to have a special key, where the user could build any customized JSON to send as a payload
- Example: Model.search('query', where: { custom_filters: [{ foo: :bar }, { xpto: :macarena }] })
- Pro: this would allow including an array of filters into the payload
- Con: if anyone using Searchkick has a custom_filters mapping on their model, this would break their query, causing backwards compatibility issues (naming is hard ¯\_(ツ)_/¯)
Expose the Searchkick::Query filters array
- Pro: allows to add as many custom filters as the user wants
- Con: This would require to rerun the Searchkick::Query#prepare command, and also would require the execute: false, since this is the only way to access the Searchkick::Query object instance

So, what do you think? :) I would be happy to open a PR on any conclusion we may reach here :)

Update: just found out about issue https://github.com/ankane/searchkick/issues/953, as @marcnit faces the same issue as me with ES nested ttypes

Source

tiagoamaro

👍2

Most helpful comment

Hey @tiagoamaro, thanks for the suggestions 👍 On a high level, I'd prefer not to mix Searchkick options with Elasticsearch options. For those who understand Elasticsearch enough to customize queries, I'd recommend using the Elasticsearch DSL.