Elasticsearch: Painless Strings do not export the getBytes() function

[josephDunne]

Elasticsearch version: 5.1

Plugins installed: default

JVM version:

OS version: Centos 6

Description of the problem including expected versus actual behavior:

Painless Strings do not export the getBytes() method.

Provide logs (if relevant):

[2017-01-17T02:52:27,514][DEBUG][o.e.a.s.TransportSearchAction] [Z8qG7Y5] [nap-gn-2017.01.17][0], node[Z8qG7Y5IScyM2r06BN0Dbg], [P], s[STARTED], a[id=sKArGnzwQb6rCU1BTubCEQ]: Failed to execute [SearchRequest{searchType=QUERY_AND_FETCH, indices=[nap-gn-2017.01.17], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='1484620334897', requestCache=null, scroll=null, source={
  "size" : 500,
  "query" : {
    "bool" : {
      "must" : [
        {
          "query_string" : {
            "query" : "*",
            "fields" : [ ],
            "use_dis_max" : true,
            "tie_breaker" : 0.0,
            "default_operator" : "or",
            "auto_generate_phrase_queries" : false,
            "max_determined_states" : 10000,
            "lowercase_expanded_terms" : true,
            "enable_position_increment" : true,
            "fuzziness" : "AUTO",
            "fuzzy_prefix_length" : 0,
            "fuzzy_max_expansions" : 50,
            "phrase_slop" : 0,
            "analyze_wildcard" : true,
            "locale" : "und",
            "escape" : false,
            "boost" : 1.0
          }
        },
        {
          "range" : {
            "TS" : {
              "from" : 1484619701937,
              "to" : 1484621501937,
              "include_lower" : true,
              "include_upper" : true,
              "format" : "epoch_millis",
              "boost" : 1.0
            }
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "_source" : {
    "includes" : [ ],
    "excludes" : [ ]
  },
  "stored_fields" : "*",
  "docvalue_fields" : [
    "TS"
  ],
  "script_fields" : {
    "search_token" : {
      "script" : {
        "inline" : "def json = '{';\nif (doc['SRC_PORT'][0] != 0){\n  json += \"\\\"lp\\\": \" + doc['SRC_PORT'][0] + \",\";\n}\nif (doc['DST_PORT'][0] != 0){\n  json += \"\\\"hp\\\": \" + doc['DST_PORT'][0] + \",\";\n}\nif (doc['VLANID'][0] != 0){\n  json += \"\\\"vl\\\": \" + doc['VLANID'][0] + \",\";\n}\njson += \"\\\"hs\\\": \" + doc['KEY'][0] + \",\";\njson += \"\\\"ts\\\": \" + doc['TS'][0];\njson += '}';\ndef encodedBytes = Base64.getEncoder().encodeToString(json.getBytes());\nreturn encodedBytes",
        "lang" : "painless"
      },
      "ignore_failure" : false
    }
  },
  "sort" : [
    {
      "TS" : {
        "order" : "desc",
        "unmapped_type" : "boolean"
      }
    }
  ],
  "aggregations" : {
    "2" : {
      "date_histogram" : {
        "field" : "TS",
        "time_zone" : "Europe/London",
        "interval" : "30s",
        "offset" : 0,
        "order" : {
          "_key" : "asc"
        },
        "keyed" : false,
        "min_doc_count" : 1
      }
    }
  },
  "highlight" : {
    "pre_tags" : [
      "@kibana-highlighted-field@"
    ],
    "post_tags" : [
      "@/kibana-highlighted-field@"
    ],
    "fragment_size" : 2147483647,
    "require_field_match" : false,
    "fields" : {
      "*" : { }
    }
  },
  "ext" : { }
}}]
org.elasticsearch.transport.RemoteTransportException: [Z8qG7Y5][127.0.0.1:9300][indices:data/read/search[phase/query+fetch]]
Caused by: org.elasticsearch.script.ScriptException: runtime error
        at org.elasticsearch.painless.ScriptImpl.convertToScriptException(ScriptImpl.java:177) ~[?:?]
        at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:124) ~[?:?]
        at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:52) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:358) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$9(SearchTransportService.java:291) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:548) [elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:504) [elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.0.jar:5.0.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: java.lang.IllegalArgumentException: Unable to find dynamic method [getBytes] with [0] arguments for class [java.lang.String].
        at org.elasticsearch.painless.Def.lookupMethodInternal(Def.java:202) ~[?:?]
        at org.elasticsearch.painless.Def.lookupMethod(Def.java:230) ~[?:?]
        at org.elasticsearch.painless.DefBootstrap$PIC.lookup(DefBootstrap.java:142) ~[?:?]
        at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:196) ~[?:?]
        at org.elasticsearch.painless.Executable$Script.execute(def json = '{'; ... @ <inline script>:404) ~[?:?]
        at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:121) ~[?:?]
        at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:52) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:358) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$9(SearchTransportService.java:291) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:548) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:504) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.0.0.jar:5.0.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_101]
[2017-01-17T02:52:27,517][DEBUG][o.e.a.s.TransportSearchAction] [Z8qG7Y5] All shards failed for phase: [query_fetch]
org.elasticsearch.transport.RemoteTransportException: [Z8qG7Y5][127.0.0.1:9300][indices:data/read/search[phase/query+fetch]]
Caused by: org.elasticsearch.script.ScriptException: runtime error
        at org.elasticsearch.painless.ScriptImpl.convertToScriptException(ScriptImpl.java:177) ~[?:?]
        at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:124) ~[?:?]
        at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:52) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:358) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$9(SearchTransportService.java:291) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:548) [elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:504) [elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.0.jar:5.0.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: java.lang.IllegalArgumentException: Unable to find dynamic method [getBytes] with [0] arguments for class [java.lang.String].
        at org.elasticsearch.painless.Def.lookupMethodInternal(Def.java:202) ~[?:?]
        at org.elasticsearch.painless.Def.lookupMethod(Def.java:230) ~[?:?]
        at org.elasticsearch.painless.DefBootstrap$PIC.lookup(DefBootstrap.java:142) ~[?:?]
        at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:196) ~[?:?]
        at org.elasticsearch.painless.Executable$Script.execute(def json = '{'; ... @ <inline script>:404) ~[?:?]
        at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:121) ~[?:?]
        at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:52) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:161) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:358) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$9(SearchTransportService.java:291) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:548) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:504) ~[elasticsearch-5.0.0.jar:5.0.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.0.0.jar:5.0.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_101]

[clintongormley]

@jdconrad @nik9000 we seem to be getting a few requests for things like base64 encoding and hashing. Should we consider adding support for this in Painless?

[jdconrad]

hashCode is included as part of the existing String and anything else that extends Object right now (so any class that is introduces non-static methods/members). I'm hesitant to introduce the getBytes method on String simply because of the character set implications without knowing more about the specific use case. You can get the codepoints for a String right now using int codePointAt(int)

[nik9000]

@jdconrad @nik9000 we seem to be getting a few requests for things like base64 encoding and hashing. Should we consider adding support for this in Painless?

I'm happy to add something for base64 encoding. If we don't want to add support for getBytes we can use that.

[jdconrad]

I'm fine with adding base64 encoding as @nik9000 is suggesting.

[ndtreviv]

Getting bytes would be REALLY handy for MD5-ing the ID of my v2.3 documents as I'm trying to reindex ids that are apparently too long to go into 5.6 😫