Elasticsearch: Null pointer exception with script query on indexed text field

[tonyyanga]

Elasticsearch version (bin/elasticsearch --version):

Version: 6.3.0, Build: default/zip/424e937/2018-06-11T23:38:03.357887Z, JVM: 1.8.0_171

Plugins installed: None

JVM version (java -version):

openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

OS version (uname -a if on a Unix-like system):

Linux tony-laptop 3.16.0-4-amd64 #1 SMP Debian 3.16.51-3 (2017-12-13) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

I created a mapping with a text field, with only one document of valid values. Then I run a script query on using _source field, filtering null values. The query fails with null pointer exception and returns HTTP 500.

I expect to receive one hit. I am able to obtain the expected behavior on version 6.0.0 but failed with 6.1.4, 6.2.4 and 6.3.0.

Steps to reproduce:

Create mappings:

PUT /test_schema_1

{
  "settings":{
    "index":{
      "number_of_shards":1,
      "number_of_replicas":0
    }
  }
}

PUT /test_schema_1/_mapping/test_table_1

{"test_table_1":{"properties":{"text_field":{"type":"text"}}}}

Populate with one document:

POST /_bulk?refresh=true

{ "index" : { "_index" : "test_schema_1", "_type" : "test_table_1" } }
{"text_field":"string_value_0"}

Query:

POST /test_schema_1/test_table_1/_search?scroll=300000ms

{
  "from" : 0,
  "size" : 4000,
  "query" : {
    "bool" : {
      "must" : [
        {
          "script" : {
            "script" : {
              "source" : "(def) (params._source.text_field == null ? null : 1 == 1)",
              "lang" : "painless"
            },
            "boost" : 1.0
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  }
}

Actual behavior with version 6.3.0:

{
    "error": {
        "root_cause": [
            {
                "type": "script_exception",
                "reason": "runtime error",
                "script_stack": [
                    "(def) (params._source.text_field == null ? null : 1 == 1)",
                    "                     ^---- HERE"
                ],
                "script": "(def) (params._source.text_field == null ? null : 1 == 1)",
                "lang": "painless"
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "test_schema_1",
                "node": "KxD0i-SOSvieYNucLKBC-g",
                "reason": {
                    "type": "script_exception",
                    "reason": "runtime error",
                    "script_stack": [
                        "(def) (params._source.text_field == null ? null : 1 == 1)",
                        "                     ^---- HERE"
                    ],
                    "script": "(def) (params._source.text_field == null ? null : 1 == 1)",
                    "lang": "painless",
                    "caused_by": {
                        "type": "null_pointer_exception",
                        "reason": null
                    }
                }
            }
        ]
    },
    "status": 500
}

Expected behavior obtained with version 6.0.0:

{
    "_scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAIWN1FxY04yMG5RSWlORnhJdzN5aTlGZw==",
    "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 1,
        "hits": [
            {
                "_index": "test_schema_1",
                "_type": "test_table_1",
                "_id": "BAY4GmQBHwt3Z431yPc8",
                "_score": 1,
                "_source": {
                    "text_field": "string_value_0"
                }
            }
        ]
    }
}

Provide logs (if relevant):

Console output:

[2018-06-19T15:46:18,138][WARN ][r.suppressed             ] path: /test_schema_1/test_table_1/_search, params: {scroll=300000ms, index=test_schema_1, type=test_table_1}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:527) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1188) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1172) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:385) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:341) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: org.elasticsearch.script.ScriptException: runtime error
    at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94) ~[?:?]
    at org.elasticsearch.painless.PainlessScript$Script.execute((def) (params._source.text_field == null ? null : 1 == 1):8) ~[?:?]
    at org.elasticsearch.index.query.ScriptQueryBuilder$ScriptQuery$1$1.matches(ScriptQueryBuilder.java:188) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:240) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:266) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:324) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:357) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0]
    ... 9 more
Caused by: java.lang.NullPointerException
    at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:202) ~[?:?]
    at org.elasticsearch.painless.PainlessScript$Script.execute((def) (params._source.text_field == null ? null : 1 == 1):22) ~[?:?]
    at org.elasticsearch.index.query.ScriptQueryBuilder$ScriptQuery$1$1.matches(ScriptQueryBuilder.java:188) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:240) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:266) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:324) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:357) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0]
    ... 9 more
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:324) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:357) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0]
    ... 9 more
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:324) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:357) ~[elasticsearch-6.3.0.jar:6.3.0]
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0]
    ... 9 more

More info:

The above error case uses script:

(def) (params._source.text_field == null ? null : 1 == 1)

I get null pointer exception with (def) (params._source== null ? null : 1 == 1) as well. However, using (def) (params == null ? null : 1 == 1) I can get the right result.

Is there a change with painless script syntax from 6.0 to 6.1 that I am not aware of?

Thank you!

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

Accessing _source in a filter script is no longer possible (removed in https://github.com/elastic/elasticsearch/pull/26003). This is orders of magnitude slower than accessing doc values. If you want to filter on a field existing, try using an exists query.

Additionally, please ask questions on our forum. We use github for feature requests and confirmed bug reports.

[juliusecker]

@rjernst

Your docs are out of date: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html

That page still lists params._source as a valid, if slow, way to access data. If that is not available anymore, __please update your docs__. This is not the place I should find out a feature was removed.