Searchkick: Feature request: proper HTML indexing using ES native html_strip char filter

[feliperaul]

I think a very common use scenario for ElasticSearch/SearchKick is use it to search HTML content generated by WYSIWYG editors like Summernote, CK Editor, so on.

After debugging for 1 hour, I found out that the default settings are terrible for indexing this kind of content.

I had a HTML post with "decadência" in both the title (simple string) and content (string with HTML) fields. If I searched Block.search "decadência", it would return a match, but Block.search "decadência", fields: [:content] would return no hits.

Using the debug notes in the readme, I ran the query trough the tokenizer:

Block.search_index.tokens("decadência", analyzer: "searchkick_index")
=> ["decadente"]

And then compared to the instance's content field trough the same tokenizer to see why they don't match:

Block.search_index.tokens(Block.last.content, analyzer: "searchkick_index")
=> HUGE array, including, amongst others, `strongdecadandecia`

As you can see, the decadência was bold (<strong>), so the tokenizer messed it all up.

My current solution is a before_save filter that runs ActionController::Base.helpers.strip_tags(self.content.squish) and saves the HTML-free string in a duplicate database field, but that solution is far from ideal.

As far as I see, ES supports this natively using a char filter:
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-htmlstrip-charfilter.html

But my ES skills prevent me from issuing a pull request yet.

---

PS: I strongly suggest adding a note to the Readme about this use case, even before a fix is made, because it took me hours to find this out and other people might just assume it's 'broken'.

PS2: the See the complete list of analyzers link in the readme is pointing to a wrong file/line.

[ankane]

Hey @feliperaul, thanks for the feedback. I've updated the list of analyzers link. You should be able to use strip_tags inside of the search_data method so you don't need a duplicate database field. I'm sorry to hear it took you a few hours, but expecting a library to strip HTML is probably not a good assumption in most cases.

[tirdadc]

@feliperaul You can resolve this pretty easily by doing what @ankane suggested. Just remember that strips_tags and the other related methods are ActionView helpers, so you have to call them like this:

def search_data
  {
    ActionController::Base.helpers.strip_tags(content)
  }
end

This works well to remove formatting, links and what not.

Edit: I would recommend using ActionView::Base.full_sanitizer.sanitize.