Elasticsearch-rails: Automatically update elasticsearch index

[m2omou]

I'm trying to update elasticsearch index after updating a field in my db.

Create and delete works fine, the index is correctly updated, but not when I try to update a field.

I have included Elasticsearch::Model::Callbacks

Any idea ? Thank you.

[karmi]

@m2omou When you include the Elasticsearch::Model::Callbacks module, changes to the model are intercepted via ActiveRecord hooks, and the index is updated. Uncomment this line https://github.com/elasticsearch/elasticsearch-rails/blob/master/elasticsearch-model/examples/activerecord_article.rb#L57 in the example, and try it out.

Alternatively, you might want to do the indexing in the background, add a custom logic, etc -- please see the relevant chapter in the README.

[m2omou]

Hi @karmi, Thanks for the information !

I already included the elastisearch callbacks, but for some reason it still doesn't work for me.

Here is my code so far:

• Model

require 'elasticsearch/model'

class Article < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  def self.suggest(query)
    Article.__elasticsearch__.client.suggest(:index => Article.index_name, :body => {
        :suggestions => {
            :text => query,
            :completion => {
                :field => 'suggest'
            }
        }
    })
  end

  def as_indexed_json(options={})
    {
        :name => self.title,
        :suggest => {
            :input => [self.title, self.content],
            :output => self.title,
            :payload => {
                :id => self.id,
                :content => self.content
            }
        }
    }
  end
end

• Controller

class ArticlesController < ApplicationController
  def update
    @article = Article.find(params[:id])

    if @article.update_attributes(article_params)
       render :json => @article
    else
       render :json => @article.errors
    end
  end
end

Do I need to include anything in the controller ?

Thank you.

[karmi]

@m2omou No, you don't need to include anything in the controller. Been playing with it a it with the linked example file, and it works as excepted, even when I add a custom as_indexed_json method in the model.

Can you isolate the problem by using that example? Can you also turn on logging for the Elasticsearch client in your application (see README) and verify what is and isn't sent to Elasticsearch?

[m2omou]

@karmi I turned on the logs, and found out something interesting. The parameters doc was empty.

> {"doc":{}}

Then I tried to comment the as_indexed_json block, and was finally able to see my parameters inside doc.

Do I need to remove as_indexed_json to make it work ?

I'm trying to implement the completion suggester:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html

And according to the doc, I need to index my data in a such way:

curl -X PUT 'localhost:9200/music/song/1?refresh=true' -d '{
    "name" : "Nevermind",
    "suggest" : {
        "input": [ "Nevermind", "Nirvana" ],
        "output": "Nirvana - Nevermind",
        "payload" : { "artistId" : 2321 },
        "weight" : 34
    }
}'

That's what I'm trying to do with the as_indexed_json

def as_indexed_json(options={})
    {
        :name => self.title,
        :suggest => {
            :input => [self.title, self.content],
            :output => self.title,
            :payload => {
                :id => self.id,
                :content => self.content
            }
        }
    }.as_json
  end

Is there another way to do the indexing with or without as_indexed_json ? Thank you !

[karmi]

It's perfectly valid to have a custom as_indexed_json, of course. Eg. for the suggester, or for other reasons. What confuses me is that I can make it work with the example, without problems. Can you check out the example application (see Rails templates), so we have a shared codebase with which we can work?

[m2omou]

humm that's strange, that doesn't work for me. I also tried with the rails template, and still facing the same issue.

class Article < ActiveRecord::Base
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  __elasticsearch__.client = Elasticsearch::Client.new :log => true

  # elasticsearch settings mapping
  settings :index => { :number_of_shards => 1 } do
    mappings :dynamic => 'false' do
      indexes :name, :type => 'string'
      indexes :suggest, :type => 'completion', :index_analyzer => 'simple', :search_analyzer => 'simple', :payloads => true
    end
  end

  # elasticsearch suggest
  def self.suggest(query)
    Article.__elasticsearch__.client.suggest(:index => Article.index_name, :body => {
        :suggestions => {
            :text => query,
            :completion => {
                :field => 'suggest'
            }
        }
    })
  end

  # elasticsearch indexing
  def as_indexed_json(options={})
    {
        :name => self.title,
        :suggest => {
            :input => [self.title],
            :output => self.title,
            :payload => {
                :id => self.id
            }
        }
    }.as_json()
  end
end

rails console

irb(main):001:0> Article.import
  Article Load (0.3ms)  SELECT  "articles".* FROM "articles"   ORDER BY "articles"."id" ASC LIMIT 1000
2015-01-13 18:50:09 +0100: POST http://localhost:9200/articles/article/_bulk?type=article [status:200, request:0.017s, query:0.010s]
2015-01-13 18:50:09 +0100: > {"index":{"_id":1}}
{"name":"Updatedsaasdasdd!","suggest":{"input":["Updatedsaasdasdd!"],"output":"Updatedsaasdasdd!","payload":{"id":1}}}
{"index":{"_id":2}}
{"name":"Two","suggest":{"input":["Two"],"output":"Two","payload":{"id":2}}}
{"index":{"_id":3}}
{"name":"Three","suggest":{"input":["Three"],"output":"Three","payload":{"id":3}}}
{"index":{"_id":4}}
{"name":"Four","suggest":{"input":["Four"],"output":"Four","payload":{"id":4}}}
{"index":{"_id":5}}
{"name":"Five","suggest":{"input":["Five"],"output":"Five","payload":{"id":5}}}

2015-01-13 18:50:09 +0100: < {"took":10,"errors":false,"items":[{"index":{"_index":"articles","_type":"article","_id":"1","_version":6,"status":200}},{"index":{"_index":"articles","_type":"article","_id":"2","_version":3,"status":200}},{"index":{"_index":"articles","_type":"article","_id":"3","_version":3,"status":200}},{"index":{"_index":"articles","_type":"article","_id":"4","_version":3,"status":200}},{"index":{"_index":"articles","_type":"article","_id":"5","_version":3,"status":200}}]}
=> 0

Now trying to update a title:

irb(main):002:0> Article.first.update_attribute :title, 'my title'         
  Article Load (0.5ms)  SELECT  "articles".* FROM "articles"   ORDER BY "articles"."id" ASC LIMIT 1
   (0.1ms)  begin transaction
  SQL (0.7ms)  UPDATE "articles" SET "title" = ?, "updated_at" = ? WHERE "articles"."id" = 1  [["title", "my title"], ["updated_at", "2015-01-13 17:50:33.806050"]]
   (1.8ms)  commit transaction
2015-01-13 18:50:33 +0100: POST http://localhost:9200/articles/article/1/_update [status:200, request:0.009s, query:n/a]
2015-01-13 18:50:33 +0100: > {"doc":{}}
2015-01-13 18:50:33 +0100: < {"_index":"articles","_type":"article","_id":"1","_version":7}
=> true

As you can see the {"doc":{}} is still empty :/

Maybe I'm doing someting wrong...

[m2omou]

Found a solution, I directly update the index in my controller:

       @article.__elasticsearch__.index_document

Thanks ! :)

[orlando]

I think this happens when you have custom methods in your as_indexed_json method. For example, I have:

mapping dynamic: false do
  indexes :start_at, type: 'date'
  indexes :end_at, type: 'date'
end

but start_at and end_at are not fields in the database, are methods that I use as helpers.

Since update_document checks for changed_attributes code, those fields will never appear (even when the data that generate those fields changes), that's why I'm getting an empty {"doc":{}} just like @m2omou

So instead of using the default update_document, you should implement your own, or use index_document

[zapnap]

Just stumbled on this myself. Thanks for the explanation @orlando

Perhaps this should be explicitly mentioned in the project README for callbacks? It really had me confused. I'm indexing updated_at in my case so I can easily expose recently updated articles, and that field never ends up triggering an update even though it's returned correctly in my custom as_indexed_json method.

The solution is simply adding a custom callback to handle the reindexing on update.

[zapnap]

Alternatively, would you accept a patch that optionally includes updated_at if it's found within the indexed json for the document?

[mguterl]

We're seeing the same issue, is there a best practice for working around this issue?

[orlando]

@mguterl Using a custom callback instead of the default update_document I think is the best approach, that way you can control when to update your index.

[karmi]

I'm a bit rusty on the original background here, but absolutely, please, consider the "automatic callbacks" just a starting point, just a convenience for somebody who wants to "try it out" -- and write your own logic using the building blocks for anything significant (as @orlando suggests)...

[dngo]

In in my model I added an aftersave and it fixes the problem. I think this is better than doing it in a controller

after_save :reindex

def reindex
   __elasticsearch__.index_document
end