Kibana: [Discover] Compile error when filtering on scripted fields

Created on 21 Sep 2016 · 7Comments · Source: elastic/kibana

To repro:

Go into the logstash-* index pattern
Click the scripted fields tab
Click add scripted field
Create field with name: "Inverse bytes"
Select language 'Painless'
Type in Script input: doc["bytes"].value * -1
Click Save
Click over to Discover
Ensure scripted fields are shown in the field list and in the documents
Ensure filters are enabled for all scripted fields in list and in documents
Filter by a script value

After following the above steps and attempt to filter by a script value, I run into a Discover: compile error with the following error message:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","script_stack":["... [\"bytes\"].value * -1) == value","                             ^---- HERE"],"script":"(doc[\"bytes\"].value * -1) == value","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetch","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-0","node":"yb6x_Ou6SdK969VVMixU1w","reason":{"type":"query_shard_exception","reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"match_all\" : {\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -3531\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : 1474211056294,\n            \"to\" : 1474470256294,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"format\" : \"epoch_millis\",\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"must_not\" : [\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -5173\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : 0\n            }\n          },\n          \"boost\" : 1.0\n        }\n      }\n    ],\n    \"disable_coord\" : false,\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}","index_uuid":"qjdZ3IWlRO-hqD_GhTIxdA","index":"logstash-0","caused_by":{"type":"script_exception","reason":"compile error","caused_by":{"type":"illegal_argument_exception","reason":"Variable [value] is not defined."},"script_stack":["... [\"bytes\"].value * -1) == value","                             ^---- HERE"],"script":"(doc[\"bytes\"].value * -1) == value","lang":"painless"}}}],"caused_by":{"type":"query_shard_exception","reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"match_all\" : {\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -3531\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : 1474211056294,\n            \"to\" : 1474470256294,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"format\" : \"epoch_millis\",\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"must_not\" : [\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -5173\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : 0\n            }\n          },\n          \"boost\" : 1.0\n        }\n      }\n    ],\n    \"disable_coord\" : false,\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}","index_uuid":"qjdZ3IWlRO-hqD_GhTIxdA","index":"logstash-0","caused_by":{"type":"script_exception","reason":"compile error","caused_by":{"type":"illegal_argument_exception","reason":"Variable [value] is not defined."},"script_stack":["... [\"bytes\"].value * -1) == value","                             ^---- HERE"],"script":"(doc[\"bytes\"].value * -1) == value","lang":"painless"}}},"status":400}
KbnError@http://localhost:5601/bundles/commons.bundle.js?v=14300:63:25990
RequestFailure@http://localhost:5601/bundles/commons.bundle.js?v=14300:63:27183
http://localhost:5601/bundles/kibana.bundle.js?v=14300:26:28446
http://localhost:5601/bundles/commons.bundle.js?v=14300:75:22358
map@[native code]
map@http://localhost:5601/bundles/commons.bundle.js?v=14300:75:21681
processQueue@http://localhost:5601/bundles/commons.bundle.js?v=14300:38:23623
http://localhost:5601/bundles/commons.bundle.js?v=14300:38:23900
$digest@http://localhost:5601/bundles/commons.bundle.js?v=14300:39:2364
$apply@http://localhost:5601/bundles/commons.bundle.js?v=14300:39:5044
done@http://localhost:5601/bundles/commons.bundle.js?v=14300:37:25033
completeRequest@http://localhost:5601/bundles/commons.bundle.js?v=14300:37:28710
onload@http://localhost:5601/bundles/commons.bundle.js?v=14300:37:29649

compileerror

compileerroroverview

bug v5.0.0

Source

stacey-gammon

Most helpful comment

Confirmed with ES that the issue is on our side (See https://github.com/elastic/elasticsearch/issues/20885). It appears like the request should send
"inline":"(-1 * doc['bytes'].value) == params.value"

instead of

"inline":"(-1 * doc['bytes'].value) == value"

stacey-gammon on 12 Oct 2016

👍2

All 7 comments

can you check if you are using the "expression" language when creating this field;

it works as an expression, but not as a painless script. Not sure if there's some invalid syntax in that example of inverse bytes

thomasneirynck on 21 Sep 2016

below is an example of an ES search request that triggers this error. It appears we get an error when using the script as part of a filter.

POST logstash-0/_search
{  
   "size":500,
   "sort":[  
      {  
         "@timestamp":{  
            "order":"desc",
            "unmapped_type":"boolean"
         }
      }
   ],
   "highlight":{  
      "pre_tags":[  
         "@kibana-highlighted-field@"
      ],
      "post_tags":[  
         "@/kibana-highlighted-field@"
      ],
      "fields":{  
         "*":{  

         }
      },
      "require_field_match":false,
      "fragment_size":2147483647
   },
   "query":{  
      "bool":{  
         "must":[  
            {  
               "query_string":{  
                  "query":"*",
                  "analyze_wildcard":true
               }
            },
            {  
               "script":{  
                  "script":{  
                     "inline":"(-1 * doc['bytes'].value) == value",
                     "lang":"painless",
                     "params":{  
                        "value":0
                     }
                  }
               }
            },
            {  
               "script":{  
                  "script":{  
                     "inline":"(-1 * doc['bytes'].value) == value",
                     "lang":"painless",
                     "params":{  
                        "value":-2724
                     }
                  }
               }
            },
            {  
               "range":{  
                  "@timestamp":{  
                     "gte":1474479018497,
                     "lte":1474479918497,
                     "format":"epoch_millis"
                  }
               }
            }
         ],
         "must_not":[  

         ]
      }
   },
   "aggs":{  
      "2":{  
         "date_histogram":{  
            "field":"@timestamp",
            "interval":"30s",
            "time_zone":"America/New_York",
            "min_doc_count":1
         }
      }
   },
   "stored_fields":[  
      "*"
   ],
   "_source":true,
   "script_fields":{  
      "inverse bytes":{  
         "script":{  
            "inline":"-1 * doc['bytes'].value",
            "lang":"painless"
         }
      }
   },
   "docvalue_fields":[  
      "utc_time",
      "relatedContent.article:modified_time",
      "@timestamp",
      "relatedContent.article:published_time"
   ]
}

Response:

{  
   "error":{  
      "root_cause":[  
         {  
            "type":"script_exception",
            "reason":"compile error",
            "script_stack":[  
               "... * doc['bytes'].value) == value",
               "                             ^---- HERE"
            ],
            "script":"(-1 * doc['bytes'].value) == value",
            "lang":"painless"
         }
      ],
      "type":"search_phase_execution_exception",
      "reason":"all shards failed",
      "phase":"query_fetch",
      "grouped":true,
      "failed_shards":[  
         {  
            "shard":0,
            "index":"logstash-0",
            "node":"apPgvtcuTbSA2V4HWRUWww",
            "reason":{  
               "type":"query_shard_exception",
               "reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"query_string\" : {\n          \"query\" : \"*\",\n          \"fields\" : [ ],\n          \"use_dis_max\" : true,\n          \"tie_breaker\" : 0.0,\n          \"default_operator\" : \"or\",\n          \"auto_generate_phrase_queries\" : false,\n          \"max_determined_states\" : 10000,\n          \"lowercase_expanded_terms\" : true,\n          \"enable_position_increment\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"locale\" : \"und\",\n          \"escape\" : false,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(-1 * doc['bytes'].value) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : 0\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(-1 * doc['bytes'].value) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -2724\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : 1474479018497,\n            \"to\" : 1474479918497,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"format\" : \"epoch_millis\",\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"disable_coord\" : false,\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}",
               "index_uuid":"2Gr8UT6HTTCyaJWywRhrCQ",
               "index":"logstash-0",
               "caused_by":{  
                  "type":"script_exception",
                  "reason":"compile error",
                  "caused_by":{  
                     "type":"illegal_argument_exception",
                     "reason":"Variable [value] is not defined."
                  },
                  "script_stack":[  
                     "... * doc['bytes'].value) == value",
                     "                             ^---- HERE"
                  ],
                  "script":"(-1 * doc['bytes'].value) == value",
                  "lang":"painless"
               }
            }
         }
      ],
      "caused_by":{  
         "type":"query_shard_exception",
         "reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"query_string\" : {\n          \"query\" : \"*\",\n          \"fields\" : [ ],\n          \"use_dis_max\" : true,\n          \"tie_breaker\" : 0.0,\n          \"default_operator\" : \"or\",\n          \"auto_generate_phrase_queries\" : false,\n          \"max_determined_states\" : 10000,\n          \"lowercase_expanded_terms\" : true,\n          \"enable_position_increment\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"locale\" : \"und\",\n          \"escape\" : false,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(-1 * doc['bytes'].value) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : 0\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(-1 * doc['bytes'].value) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -2724\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : 1474479018497,\n            \"to\" : 1474479918497,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"format\" : \"epoch_millis\",\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"disable_coord\" : false,\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}",
         "index_uuid":"2Gr8UT6HTTCyaJWywRhrCQ",
         "index":"logstash-0",
         "caused_by":{  
            "type":"script_exception",
            "reason":"compile error",
            "caused_by":{  
               "type":"illegal_argument_exception",
               "reason":"Variable [value] is not defined."
            },
            "script_stack":[  
               "... * doc['bytes'].value) == value",
               "                             ^---- HERE"
            ],
            "script":"(-1 * doc['bytes'].value) == value",
            "lang":"painless"
         }
      }
   },
   "status":400
}

thomasneirynck on 21 Sep 2016

instead of

"inline":"(-1 * doc['bytes'].value) == value"

stacey-gammon on 12 Oct 2016

👍2

Users can theoretically write groovy and other language scripts if they've manually enabled them in ES, so we should probably check those as well.

Bargs on 12 Oct 2016

I think I found the culprit and have a fix, at least for painless & expression. I'm not sure about the other script types. Kibana will show additional options in the language dropdown depending on ES settings? How can I adjust those settings to verify if it works?

stacey-gammon on 12 Oct 2016

You can enable groovy in your grunt esvm config by adding the following code to your grunt esvm (esvm.js) dev config block:

script: {
  engine: {
    groovy: {
      inline: true
    }
  }
}

Bargs on 12 Oct 2016

Good call on checking Groovy - indeed it is not happy with params.value. Checking with @clintongormley to see if this is expected.

stacey-gammon on 12 Oct 2016

Was this page helpful?

0 / 5 - 0 ratings

Related issues

convert json of a kibana visualization to an elastic query

socialmineruser1 · 3Comments

Emoji Support

cafuego · 3Comments

Use the visualizeLoader for dashboards

timroes · 3Comments

No 'currency' option for number field format in 4.1.1

bradvido · 3Comments

Rename dashboards

MaartenUreel · 3Comments