Elasticsearch: Term vectors API crashes for documents that have no term vectors in nested fields

[rafaelglater]

Elasticsearch version: 6.3.2

JVM version: 1.8.0_181

OS version: KDE Neon 5.13.4

Description of the problem including expected versus actual behavior:
When requesting the termvectors of a document with empty terms in a nested field, we get a null pointer exception.

Steps to reproduce:
1.
PUT /my-index { "settings": { "number_of_shards": 1, "similarity": { "default": { "type": "BM25" } }, "analysis": { "filter": { "bigram_filter": { "type": "shingle", "min_shingle_size": 2, "max_shingle_size": 2, "output_unigrams": false }, "stopwords": { "type": "stop", "stopwords": "_english_" }, "kstemmer": { "type": "stemmer", "language": "kstem" }, "possessive": { "type": "stemmer", "language": "possessive_english" } }, "analyzer": { "bigram_analyzer": { "type": "custom", "tokenizer": "standard", "filter": [ "lowercase", "stopwords", "possessive", "kstemmer", "bigram_filter" ] }, "standard": { "tokenizer": "standard", "filter": [ "lowercase", "stopwords", "possessive", "kstemmer" ] } } } } }

2.
PUT /my-index/_mapping/my_doc { "my_doc": { "properties": { "text": {"type":"text", "analyzer":"standard", "term_vector": "yes", "fields": { "bigrams": { "type": "text", "term_vector": "yes", "analyzer": "bigram_analyzer" } } }, "links": {"type":"nested", "properties":{ "page_id": {"type":"text", "analyzer":"standard"}, "page_name": {"type":"text", "analyzer":"standard", "fields": { "bigrams": { "type": "text", "analyzer": "bigram_analyzer" } } } } } } } }

3.
Insert documents with different terms configurations:

PUT /my-index/my_doc/111 { "text": "a bigram text", "links": [{"page_id": "1", "page_name":"two terms"}, {"page_id": "2", "page_name":"unique"} ] }

PUT /my-index/my_doc/222 { "text": "unigram", "links": [{"page_id": "3", "page_name":"one"}, {"page_id": "2", "page_name":"unique"} ] }

PUT /my-index/my_doc/333 { "text": "", "links": [{"page_id": "4", "page_name":""}] }

PUT /my-index/my_doc/444 { "text": "", "links": [{"page_id": "4", "page_name":""}, {"page_id": "2", "page_name":"unique"} ] }

4.
If I request the termvectors of documents 111 and 222 it's ok. But when requesting termvectors of document 333 and 444, I get null pointer exception because I have an empty nested field "page_name":

GET /my-index/my_doc/444/_termvectors { "fields" : ["text", "text.bigrams", "links.page_name", "links.page_name.bigrams"], "offsets" : false, "payloads" : false, "positions" : false, "term_statistics" : true, "field_statistics" : true }

Provide logs (if relevant):
[2018-08-06T16:04:28,002][DEBUG][o.e.a.t.TransportTermVectorsAction] [T3sBXHE] null: failed to execute [org.elasticsearch.action.termvectors.TermVectorsRequest@307d2b7b]
org.elasticsearch.transport.RemoteTransportException: [T3sBXHE][127.0.0.1:9300][indices:data/read/tv[s]]
Caused by: org.elasticsearch.ElasticsearchException: failed to execute term vector request
at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:150) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:77) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:89) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:43) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:287) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:280) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:259) [x-pack-security-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:317) [x-pack-security-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:664) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.lang.NullPointerException
at org.elasticsearch.action.termvectors.TermVectorsWriter.setFields(TermVectorsWriter.java:82) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TermVectorsResponse.setFields(TermVectorsResponse.java:361) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:146) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:77) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:89) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:43) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:287) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:280) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:259) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:317) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:664) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
[2018-08-06T16:04:28,008][WARN ][r.suppressed ] path: /my-index/my_doc/444/_termvectors, params: {index=my-index, id=444, type=my_doc}
org.elasticsearch.transport.RemoteTransportException: [T3sBXHE][127.0.0.1:9300][indices:data/read/tv[s]]
Caused by: org.elasticsearch.ElasticsearchException: failed to execute term vector request
at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:150) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:77) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:89) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:43) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:287) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:280) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:259) [x-pack-security-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:317) [x-pack-security-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:664) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.lang.NullPointerException
at org.elasticsearch.action.termvectors.TermVectorsWriter.setFields(TermVectorsWriter.java:82) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TermVectorsResponse.setFields(TermVectorsResponse.java:361) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:146) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:77) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:89) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:43) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:287) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:280) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:259) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:317) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:664) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]

[elasticmachine]

Pinging @elastic/es-search-aggs

[colings86]

@mayya-sharipova would you be able to have a look at this?

[kramarz]

I was running into a similar issue without "nested" in 6.3.2 But I wrote a script to reproduce it and after switching to 6.4.0 the problem is not there anymore :laughing: . Then I also tested it with nested feature and it also seems to work. The script is here if anyone wants to check it : https://gist.github.com/kramarz/20844c1cb48fc745d806dff0c0016525

[rafaelglater]

@kramarz I was also having trouble with non-nested fields (which I realized later, after opening this bug). Since my needs couldn't wait for this bug fix, I changed my code to request term vectors field by field separately and assigning "zero" in the cases where I got an error... This was time-consuming, by the way... Good to know it's fixed in version 6.4. I'll give a try. Thank you for letting me know!