Elasticsearch-rails: Explictly map inner objects that are not 'parent/child' or 'nested' types

[twigbranch]

Is this possible with elasticsearch-rails persistence?

Based on previous tire implementations, I thought something like the following would work:

Class Tweet
...
mappings do
  indexes :message, type: 'string'
  indexes :tags, type: 'string',
  indexes :lists do 
    indexes :name, type: 'string',
    indexes :description, type: 'string'
    indexes :members do 
      indexes :identifier, type: 'string'
      indexes :name, type: 'string'
    end
  end
end

But, the resulting mappings show 'lists' of type 'string' and I get an error when trying to import data (org.elasticsearch.index.mapper.MapperParsingException: failed to parse [lists]). Here's the full error:

org.elasticsearch.index.mapper.MapperParsingException: failed to parse [lists]
    at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:416)
    at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:557)
    at org.elasticsearch.index.mapper.object.ObjectMapper.serializeNonDynamicArray(ObjectMapper.java:688)
    at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:607)
    at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:492)
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:542)
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:491)
    at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:413)
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:435)
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:150)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:512)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: unknown property [name]
    at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateFieldForString(StringFieldMapper.java:331)
    at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:277)
    at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:406)
    ... 14 more

What am I missing?

[twigbranch]

As in the examples, I had been defining 'mapping' before 'attributes.'

In doing so, the call to 'attributes' caused the results of my prior call to 'mapping' to be overridden.

This is somewhat confusing behavior, especially with the way the examples are written with separate calls to mapping and attributes in that order.

[twigbranch]

Perhaps in the call to mapping, the "default" type of 'string' should only be assigned if mapping has not yet been defined? Or there should be a warning that mapping is being redefined... or at least made clear in the examples and docs, that the call to attributes will override any prior separate call to mapping for a given property.

[karmi]

I have fixed your example to be fully executable and tried to understand the issue:

require 'elasticsearch/persistence/model'

class Tweet
  include Elasticsearch::Persistence::Model  
  mappings do  
    indexes :message, type: 'string'    
    indexes :tags, type: 'string'    
    indexes :lists do     
      indexes :name, type: 'string'      
      indexes :description, type: 'string'      
      indexes :members do       
        indexes :identifier, type: 'string'        
        indexes :name, type: 'string'        
      end        
    end      
  end    
end;  

Tweet.mapping.to_hash
# => {:tweet=>
#  {:properties=>
#    {:created_at=>{:type=>"date"},
#     :updated_at=>{:type=>"date"},
#     :message=>{:type=>"string"},
#     :tags=>{:type=>"string"},
#     :lists=>
#      {:type=>"object",
#       :properties=>
#        {:name=>{:type=>"string"},
#         :description=>{:type=>"string"},
#         :members=>{:type=>"object", :properties=>{:identifier=>{:type=>"string"}, :name=>{:type=>"string"}}}}}}}}

Tweet.create_index! force: true
# => {"acknowledged"=>true}

puts Net::HTTP.get URI.parse('http://localhost:9200/tweets/_mapping?pretty')
# {
#   "tweets" : {
#     "mappings" : {
#       "tweet" : {
#         "properties" : {
#           "created_at" : {
#             "type" : "date",
#             "format" : "dateOptionalTime"
#           },
#           "lists" : {
#             "properties" : {
#               "description" : {
#                 "type" : "string"
#               },
#               "members" : {
#                 "properties" : {
#                   "identifier" : {
#                     "type" : "string"
#                   },
#                   "name" : {
#                     "type" : "string"
#                   }
#                 }
#               },
#               "name" : {
#                 "type" : "string"
#               }
#             }
#           },
#           "message" : {
#             "type" : "string"
#           },
#           "tags" : {
#             "type" : "string"
#           },
#           "updated_at" : {
#             "type" : "date",
#             "format" : "dateOptionalTime"
#           }
#         }
#       }
#     }
#   }
# }

Can you run the isolated example again and point me specifically to the place you see some kind of error?

[twigbranch]

Thanks for checking on this. As I mentioned in my follow-up comment, I had been setting 'attribute' after 'mappings,' like the following:

class Tweet
  include Elasticsearch::Persistence::Model  
  mappings do  
    indexes :message, type: 'string'    
    indexes :tags, type: 'string'    
    indexes :lists do     
      indexes :name, type: 'string'      
      indexes :description, type: 'string'      
      indexes :members do       
        indexes :identifier, type: 'string'        
        indexes :name, type: 'string'        
      end        
    end      
  end  

  attribute :message, String
  attribute :tags, String
  attribute :lists, Array[List]

end

class List
  include Elasticsearch::Persistence::Model  
  attribute :name, String
  attribute :description, String
  attribute :members, Array[Member]
end

class Member
  include Elasticsearch::Persistence::Model 
  attribute :identifier, String
  attribute :name, String
end

The result of this is,

Tweet.mapping.to_hash
=> {:tweet=>{:properties=>{:created_at=>{:type=>"date"}, :updated_at=>{:type=>"date"}, :message=>{:type=>"string"}, :tags=>{:type=>"string"}, :lists=>{:type=>"string"}}}}

As you can see, 'lists' is mapped to 'string' instead of 'object.'

If 'attribute' is called before 'mappings' its ok.

Obviously I was confused as to the proper way to set attributes and mapping, partly because of the way the album/artists persistence example used both mappings and attribute, and called mappings before attribute.

Based on this, I made the following suggestion (above):

Perhaps in the call to mapping, the "default" type of 'string' should only be assigned if mapping has not >yet been defined? Or there should be a warning that mapping is being redefined... or at least made >clear in the examples and docs, that the call to attributes will override any prior separate call to mapping >for a given property.

[karmi]

Not for me:

Tweet.mapping.to_hash
=> {:tweet=>
  {:properties=>
    {:created_at=>{:type=>"date"},
     :updated_at=>{:type=>"date"},
     :message=>{:type=>"string"},
     :tags=>{:type=>"string"},
     :lists=>
      {:type=>"object",
       :properties=>
        {:name=>{:type=>"string"},
         :description=>{:type=>"string"},
         :members=>{:type=>"object", :properties=>{:identifier=>{:type=>"string"}, :name=>{:type=>"string"}}}}}}}}

But importantly, why are you using the indexes method at all for this?

Why not:

require 'elasticsearch/persistence/model'

class List
  # This shouldn't be Elasticsearch::Persistence::Model I think, but eg. Virtus
end

class Tweet
  include Elasticsearch::Persistence::Model

  attribute :message, String
  attribute :tags, String
  attribute :lists, Array[List], mapping: {
    type: 'object',
    properties: {
      name: { type: 'string' }
      # ...
    }
  }
end

puts Tweet.mapping.to_hash
# {:tweet=>{:properties=>{:created_at=>{:type=>"date"}, :updated_at=>{:type=>"date"}, :message=>{:type=>"string"}, :tags=>{:type=>"string"}, :lists=>{:type=>"object", :properties=>{:name=>{:type=>"string"}}}}}}

[twigbranch]

1) Why use indexes?
Because it wasn't clear I shouldn't based on the README and the examples (particularly album/artist, though it looks like one issue was changed).

2) We are getting different results for Tweet.mapping.to_hash - type of 'lists' is still 'string' when calling attribute after mappings.

class Member
  include Elasticsearch::Persistence::Model  #or Virtus
  attribute :identifier, String
  attribute :name, String
end

class List
  include Elasticsearch::Persistence::Model  #or Virtus
  attribute :name, String
  attribute :description, String
  attribute :members, Array[Member]
end

class Tweet
  include Elasticsearch::Persistence::Model  
  mappings do  
    indexes :message, type: 'string'    
    indexes :tags, type: 'string'    
    indexes :lists do     
      indexes :name, type: 'string'      
      indexes :description, type: 'string'      
      indexes :members do       
        indexes :identifier, type: 'string'        
        indexes :name, type: 'string'        
      end        
    end      
  end  

  attribute :message, String
  attribute :tags, String
  attribute :lists, Array[List]

end

Tweet.mapping.to_hash
=> {:tweet=>{:properties=>{:created_at=>{:type=>"date"}, :updated_at=>{:type=>"date"}, :message=>{:type=>"string"}, :tags=>{:type=>"string"}, :lists=>{:type=>"string"}}}}

[karmi]

I need you to run the code I have posted and verify that you're seeing the same output as I do.

Then, and only then, we can progress to your code -- we need a common baseline.

[jogaco]

Glad I found this issue, as it is not documented how to specify mapping for object attributes.
I should be made explicitly with an example in README, IMO.

[vbrazo]

I did the same but when I try to merge a nested association inside an index and do a save, I get a which is an instance of Hash(#70352839692580).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.