Searchkick: Searchkick 5

[ankane]

Timeline: when Elasticsearch 8 is released

New

• [ ] Add support for Elasticseach 8
• [ ] (may wait until 5.1) Active Record-like query building - Product.search("apples").where(in_stock: true).limit(10).offset(50) - see #1395 and relation branch (still need to merge certain options intuitively, show friendly error message when trying to call AR scope on Searchkick relation)
• [ ] Support mode: :async and mode: :queue with bulk reindexing - reindex_refactor branch
• [ ] (maybe) Use search_as_you_type for instant search - search_as_you_type branch (on hold due to misspellings)
• [ ] Use rank_features type for conversions - conversions_v2 branch https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-rank-feature-query.html (todo: upgrade path)

Breaking

• [ ] Drop support for Elasticsearch 6
• [ ] Drop support for NoBrainer and Cequel
• [ ] Lazy load queries - relation branch
• [ ] Have relation reindex remove records when should_index? is false - #1424
• [ ] Raise an error for unpermitted parameters (like Active Record)
• [ ] Raise ArgumentError instead of RuntimeError for unknown operators
• [ ] Add .+ to non-anchored regular expressions
• [ ] Add support for case insensitive regular expressions - #1461
• [ ] Raise ArgumentError (instead of warning) for invalid regular expression modifiers
• [ ] Raise error when search called on relations
• [ ] Fix or remove wordnet option (could remove first and re-add later)
• [ ] Drop support for deprecated aws_signers_v4 (prefer aws_sigv4 instead)
• [ ] Remove hashie dependency - less_deps branch
• [ ] Use like: [{_index: ..., _id: ...}] for similar records
• [ ] Replace japanese with japanese2

Other

• [ ] Decide how to handle fields that use Searchkick options like word_start and custom settings with merge_mappings (throw error?) - #1410
• [ ] Use ActiveSupport.on_load(:mongoid) for Mongoid
• [ ] Choose better name for HashWrapper class - Result, Hit, Document (could still subclass HashWrapper for backward compatibility)
• [ ] For partial reindexing, move to reindex(update: :search_method) instead of reindex(:search_method) (warn on deprecated style)
• [ ] Drop support for Ruby < 2.6 and Active Record < 5.2

Maybe

• [ ] Better notifications - log data on store and update and add filtering for sensitive data (similar to Active Record filter_attributes)
• [ ] Have relation.reindex load all records - use relation.in_batches.reindex to load in batches
• [ ] reindex_now/reindex_later methods
• [ ] Add with_score method to Searchkick::Results
• [ ] Take model pagination into account
• [ ] Only create analyzers and filters that are used
• [ ] Make misspellings below the default (pros: less bad matches, better performance for correctly spelled queries, cons: worse performance for misspelled queries due to additional query)
• [ ] Deprecate (or remove) scope_results in place of load

Before 5 (Searchkick 4.4)

• [x] Reloadable, multi-word, search-time synonyms https://www.elastic.co/blog/boosting-the-power-of-elasticsearch-with-synonyms

[jpaas]

What do you think about mirroring ES versions? i.e. When you support ES 8 bump version to 8.x.

[ankane]

Hey @jpaas, thanks for the suggestion. I don't see a strong reason to mirror Elasticsearch versions. The plan is for the latest version of Searchkick to always support the 2 most recent Elasticsearch versions.

[SteveC]

Some ideas from recent use of searchkick (thanks!!!)

1. multi-word synonyms would be super useful
2. bulk reindex nested object caching. For example when doing MyObject.reindex if it belongs_to OtherObject and I have a model search_data pulling in the nested data, then reindex() will reindex 1,000 records at a time but try to load the nested OtherObject with 1,000 SELECT statements. This is suboptimal if there aren't that many OtherObjects. I've gotten around this with my own little hash cache in the model that for MyObject search_data() goes to my hash of objects instead of through activerecord for the OtherObject when doing any indexing. It helps in my case that OtherObject rarely changes, and maybe rails builtin caching would be better..... but anyway, some sort of built in caching would be awesome for reindexing millions of rows, unless I somehow missed it and it's already there.
3. convention-based reindex column. searchkick could have a convention of 'needs_reindex' column on objects for when the standard methods fail. For example, the ActiveRecord Import gem allows me to import hundreds of thousands of records very quickly but it bypasses searchkick. While, yes, I can and do track this (well, kinda) and do my own partial reindex, if I could just set a column flag which gets reindexed in some cron (or whatever) job then that would be more awesome, perhaps?

[ankane]

Hey @SteveC, thanks for the ideas!

1. Would love to support multi-word synonyms, but not sure there's an easy way to do this with the current analyzers. If someone is able to figure it out, happy to review a PR.
2. The search_import scope should help with N+1 queries for bulk reindexing if you haven't tried that.
3. It's an interesting idea. I'm not sure I want to add more reindexing patterns right now, but someone could create a gem that wraps Searchkick to do this.

Edit: Think I found a way to do multi-word synonyms (synonyms_v2 branch), so will try to make this part of Searchkick 5.

[ankane]

No point in waiting for Searchkick 5. Reloadable, multi-word, search time synonyms are now available in Searchkick 4.4 :tada:

https://github.com/ankane/searchkick#synonyms