Elasticsearch: Issue with inline script using Joda DateTimeZone.

[deanobarnett]

Using the offical Elasticsearch Docker image for 2.3.1

Elasticsearch version: 2.3.1

JVM version: java:8-jre

OS version: debian jessie

Description of the problem including expected versus actual behavior:

Actual

Using DateTimeZone with an inline script results in an error.

java.io.IOException: Resource not found: "org/joda/time/tz/data/America/New_York"

Expected

The Correct DateTimeZone instance to be returned.

Steps to reproduce:

1. Start elasticsearch 2.3.1.
2. Pass the startup param -Des.script.engine.groovy.inline.aggs=true
3. Create index and mapping with a date_time_no_millis field.
4. Run a search query like the one in Appendix: A
5. Notice an error like The datetime zone id 'America/New_York' is not recognised
6. Run a Date Histogram query like the one in Appendix: B
7. The same error will occur
8. Restart the elasticsearch instance
9. Re-run Appendix A and B, they now both work.
   

   Appendix

A: Aggregate Query

{
  "size": 0,
  "aggs": {
    "group_by_hour": {
      "terms": {
        "script": "def opDate = new DateTime(doc['operation_date'].date); opDate.withZone(DateTimeZone.forID('America/New_York')).getHourOfDay()",
        "order": {
          "_term": "asc"
        }
      }
    }
  }
}

B: Date Histogram Query

{
  "size": 0,
  "aggs": {
    "group_by_hour": {
      "date_histogram": {
        "field": "date_field",
        "interval": "hour",
        "format": "H",
        "time_zone": "America/New_York"
      }
    }
  }
}

[clintongormley]

Whether this fails or not depends on the order of requests, eg given this document:

PUT t/t/1
{
  "operation_date": "2001/10/10"
}

If you run a non-scripting agg which refers to the time zone first:

GET _search
{
  "aggs": {
    "NAME": {
      "date_histogram": {
        "field": "operation_date",
        "interval": "hour",
        "time_zone": "America/New_York"
      }
    }
  }
}

then this request succeeds, and so do the scripting request:

GET _search
{
  "size": 0,
  "aggs": {
    "group_by_hour": {
      "terms": {
        "script": "def opDate = new DateTime(doc['operation_date'].date); opDate.withZone(DateTimeZone.forID('America/New_York')).getHourOfDay()",
        "order": {
          "_term": "asc"
        }
      }
    }
  }
}

If you reverse the order of the searches, then both fail.

[gmoskovicz]

If you don't have an aggregation, there is currently no way to get timezones and work with scripts. Any ETA on fixing this? Any workaround that can be defined inside the script?

I tried modifying the java.policy file but doesn't look like it works. Isn't this related to #14524? Is there any workaround other than performing a fake search operation before running the script?

[jasontedor]

Modifying the java.policy file will not help, the issue is in Joda Time.

[gmoskovicz]

Thanks for the quick answer @jasontedor . So there isn't a real workaround? Is it related to JodaOrg/joda-time#327? Until this is not fixed, one cannot do this operation?

Is there something that we can do for this? Looks like the Joda Time issue has been opened for a while already.

[jasontedor]

I opened JodaOrg/joda-time#375. When we can incorporate a new release of Joda Time that contains this into Elasticsearch we will be able to close this bug out.

[gmoskovicz]

@jasontedor is this going to be added in a 2.3.x release for ES?

[jasontedor]

is this going to be added in a 2.3.x release for ES?

I don't know; we have to see if JodaOrg/joda-time#375 is accepted into Joda Time first, and if it is, the timeline under which a bug fix release of Joda Time is made that includes it that we can incorporate into Elasticsearch.