Elasticsearch-rails: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes

[SchlauFuchs]

Hi,

I am trying a bulk import using an ActiveRecord class including ElasticSearch::Model using the function import.
I can see a call
2016-07-27 12:52:24 +1200: POST http://localhost:9200/myindex/myclass/_bulk [status:200, request:0.159s, query:0.150s] 2016-07-27 12:52:24 +1200: > {"index":{"_id":794719072}}
and lots of json lines, then
lots of
2016-07-27 12:52:24 +1200: < {"took":150,"errors":true,"items":[{"index":{"_index":"myindex","_type":"myclass","_id":"794719072","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}}}},

On the elasticsearch log side I can see:
[2016-07-27 12:52:24,427][DEBUG][action.bulk ] [Hybrid] [myindex][4] failed to execute bulk item (index) index {[myindex][myclass][794720051], source[_na_]} MapperParsingException[failed to parse]; nested: NotXContentException[Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes]; at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:154) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309) at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:580) at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:559) at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:212) at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:224) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:327) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:120) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271) at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes at org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:85) at org.elasticsearch.common.xcontent.XContentHelper.createParser(XContentHelper.java:50) at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:90) ... 18 more

This happens on ElasticSearch 2.3.4 using elasticsearch-model 0.1.9.

[karmi]

Hi, I'm not familiar with the error, but this looks like a problem with the mapping. Can you verify that you mapping is correct and potentially paste it here for other to inspect?

[SchlauFuchs]

Hi, It is not the mapping as when I import them doc by doc using as
statement like

MyEntity.find_each{|h|h.__elasticsearch__.index_document}
everything is processing fine.
Which doesn't mean that my mapping is perfect yet.

On 1 August 2016 at 03:39, Karel Minarik [e-mail] wrote:

Hi, I'm not familiar with the error, but this looks like a problem with
the mapping. Can you verify that you mapping is correct and potentially
paste it here for other to inspect?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/elastic/elasticsearch-rails/issues/606#issuecomment-236436853,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA6unbzeagZUkwJ2buZs3E9H1VVmMo5Jks5qbMHMgaJpZM4JVvXN
.

```

[mtomov]

Hello,

This error actually occurs if you pass in a string instead of a json document to the body[n]data key.

Maybe something in the bulk api code transforms the string into something else and then sends it to elastic, while it doesn't do the same when using index_document. index_document works even with a plain string.

This is the string document for reference:

"{\"id\":1,\"name\":\"Ruby on Rails Tote\",\"description\":\"Molestiae recusandae reiciendis cumque qui sapiente dolores. Illo cum nostrum numquam mollitia et ex sequi error. Aut voluptas dolores quam est odio repudiandae.\",\"slug\":\"ruby-on-rails-tote\",\"shipping_category_id\":1,\"master\":{\"id\":1,\"sku\":\"ROR-00011\",\"price\":\"15.99\",\"display_price\":\"£15.99\",\"in_stock\":true,\"total_on_hand\":9,\"backorderable\":false,\"option_values\":[],\"images\":[{\"position\":1,\"small_url\":\"/spree/products/21/small/ror_tote.jpeg?1478727074\"},{\"position\":2,\"small_url\":\"/spree/products/22/small/ror_tote_back.jpeg?1478727074\"}]},\"variants\":[],\"product_properties\":[{\"value\":\"Tote\",\"property_name\":\"Type\"},{\"value\":\"15\\\" x 18\\\" x 6\\\"\",\"property_name\":\"Size\"},{\"value\":\"Canvas\",\"property_name\":\"Material\"}],\"classifications\":[{\"taxon\":{\"id\":3,\"name\":\"Bags\",\"parent_id\":1,\"permalink\":\"categories/bags\",\"description\":null,\"meta_description\":null,\"meta_title\":null,\"taxons\":[{\"id\":1,\"name\":\"Categories\",\"parent_id\":null,\"permalink\":\"categories\",\"description\":null,\"meta_description\":null,\"meta_title\":null}]}},{\"taxon\":{\"id\":10,\"name\":\"Rails\",\"parent_id\":2,\"permalink\":\"brand/rails\",\"description\":null,\"meta_description\":null,\"meta_title\":null,\"taxons\":[{\"id\":2,\"name\":\"Brand\",\"parent_id\":null,\"permalink\":\"brand\",\"description\":null,\"meta_description\":null,\"meta_title\":null}]}}]}"

If I do JSON.parse(..) on that string, and supply that to the data key on the bulk call { index: { _id: model.id, data: JSON.parse(...) } } - then it works..

[geekpete]

Looks like a fix was merged for this:

Binary fields indexed into an upgraded (from 1.x -> 2.x) index hit "Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes" exception

https://github.com/elastic/elasticsearch/pull/21440

[karmi]

Thanks for the link, @geekpete! @SchlauFuchs, is it OK to close the issue?

[chadfurman]

~This issue is present in the source I pulled following the tutorial here:~

~https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html~

Was an issue in my template being brought in when following the tutorial here:
https://documentation.wazuh.com/1.1/ossec_elk_elasticsearch.html#ossec-elk-elasticsearch

Rather than running cd ~/ossec_tmp/ossec-wazuh/extensions/elasticsearch/ && curl -XPUT "http://localhost:9200/_template/ossec/" -d "@ossec-elastic-template.json"

I should run:

cd ~/ossec_tmp/ossec-wazuh/extensions/elasticsearch/ && curl -XPUT "http://localhost:9200/_template/ossec/" -d "@wazuh-elastic5-template.json"

[Laykou]

Note: @mtomov 's answer above helped us. In our case we were sending nil. So we just need to make sure to always send a hash :) Thx

[estolfo]

Closing as this has been resolved. See the comments above for solutions.

[xiao-ma-nong]

用实体类存就可以了