Kibana: Kibana error - ElasticSearchParseException related to date field

[abennet]

Hello,

I would appreciate some help with the following date parse error.

I have a date mapping defined with the @Field annotation as follows:

@Field(type = org.springframework.data.elasticsearch.annotations.FieldType.Date, format = DateFormat.custom, pattern="yyyy-MM-dd HH:mm:ss")
private String loggedTime;

The data gets inserted into elastic search just fine, but when I load kibana3 with an table to display fields I get the following error:

Oops! ElasticSearchParseException[failed to parse date field [2013-12-07T01:02:31.880Z], tried both date format [yyyy-MM-dd HH:mm:ss], and timestamp number]

The date mentioned in the error [2013-12-07T01:02:31.880Z] is part of the custom time filter enabled (I don't have control over its format, enabled it using the kibana interface).

The other relevant kibana configuration are the time picker settings - which refer to the loggedTime field with the mapping above, and the default relative and auto-refresh options mentioned.
I tried the kibana index settings with "day" and "hour", "none" timestamping and the index pattern to [anoraklogs-]YYYY-MM-DD HH:mm:ss (since that matches the time stamp mapping above).
I do not have the _timestamp field enabled in my index.

Is the error message related to a bad configuration in elastic search or the kibana dashboard? Seems like it's trying to parse the time filter range with the format defined in the mapping… How else can I specify the kibana time filter to comply?

Thanks,
Andra

[rashidkpc]

Sounds like your date isn't being mapped correctly. I'm not sure what you mean by custom time filter, but you should take a look at http://localhost:9200/_mapping and ensure your field is being correctly mapped to a date type with the correct format.

[abennet]

Hi Rashid,

Thank you for your reply. I checked my mapping and looks correct ("loggedTime":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}) - as expected.

By "custom time filter" I meant that I chose to define "Custom" (as seen below) - in the "Set a time filter" dashboard settings.

So I have two time filters defined:

time must
field : loggedTime
from : 2013-12-06T22:00:11.564Z
to : 2013-12-07T01:02:31.880Z
(shows date parse failures in the fields panel)

and

time must
field : loggedTime
from : now-30d
to : now

(shows date parse failures in the time series panel)

What is the relationship between my date format in the ES mapping and the date format from the Index Settings in the Kibana configuration? I played with the latter a bit, and convinced myself it should be timestamped daily with [myIndexName-]YYYY-MM-DDTHH:mm:ss.SSSZ
I must be missing something about this….

Thanks,
Andra

[rashidkpc]

[myIndexName-]YYYY-MM-DDTHH:mm:ss.SSSZ is definitely not right unless you're creating a new index every millisecond...which will not scale well :-)

Looking at your original ticket:
[2013-12-07T01:02:31.880Z], tried both date format [yyyy-MM-dd HH:mm:ss], and timestamp number]

^ that timestamp does not match your pattern. Though it shouldn't matter, since Kibana sends the time as a long, not iso8601, but have you checked that your data actually contains the field in the format you expect it in?

[neeleshs]

With elasticsearch 1.4 & kibana 4 beta. Going thru the tutuorial- added about 5 "blog" documents to "website" index in EC & hooked up Kibana and creates a historgram with "date" field aggregated daily against counts

[2014-11-22 00:28:03,226][DEBUG][action.search.type       ] [Spectra] [website][4], node[12Moo9TpTgeUtzO4E55Qbw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1e7d9bc8] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Americop][inet[/10.0.0.9:9301]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.ElasticsearchParseException: failed to parse date field [2014-11-22T08:13:03.150Z], tried both date format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd], and timestamp number
    at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:258)
    at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:82)
    at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:44)
    at org.elasticsearch.search.aggregations.support.format.ValueParser$DateMath.parseLong(ValueParser.java:96)
    at org.elasticsearch.search.aggregations.bucket.histogram.ExtendedBounds.processAndValidate(ExtendedBounds.java:52)
    at org.elasticsearch.search.aggregations.bucket.histogram.HistogramAggregator$Factory.create(HistogramAggregator.java:179)
    at org.elasticsearch.search.aggregations.bucket.histogram.HistogramAggregator$Factory.create(HistogramAggregator.java:138)
    at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:54)
    at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
    at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
    at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalArgumentException: Invalid format: "2014-11-22T08:13:03.150Z" is malformed at "-11-22T08:13:03.150Z"
    at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
    at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:250)
    ... 18 more
[2014-11-22 00:28:03,228][DEBUG][action.search.type       ] [Spectra] [website][2], node[nGAg-931RMqMb6evx-jgWg], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1e7d9bc8]
org.elasticsearch.ElasticsearchParseException: failed to parse date field [2014-11-22T08:13:03.150Z], tried both date format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd], and timestamp number
    at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:258)
    at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:82)
    at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:44)
    at org.elasticsearch.search.aggregations.support.format.ValueParser$DateMath.parseLong(ValueParser.java:96)
    at org.elasticsearch.search.aggregations.bucket.histogram.ExtendedBounds.processAndValidate(ExtendedBounds.java:52)
    at org.elasticsearch.search.aggregations.bucket.histogram.HistogramAggregator$Factory.create(HistogramAggregator.java:179)
    at org.elasticsearch.search.aggregations.bucket.histogram.HistogramAggregator$Factory.create(HistogramAggregator.java:138)
    at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:54)
    at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
    at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
    at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalArgumentException: Invalid format: "2014-11-22T08:13:03.150Z" is malformed at "-11-22T08:13:03.150Z"
    at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
    at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:250)
    ... 18 more

[neeleshs]

Also, the date values are not from the documents. The ones in the document are between 2014/01/01 to 2014/01/05. The failures on some current timestamps but I don't understand their source

[iamyogesh]

@rashidkpc
i have indexed data of elasticsearch in yyyy-MM-dd HH:mm:ss format.but when visulize data in kibana
if i create bucket of data and filter by daily am getting "failed to parse date field [1459449000000] with format [yyyy-MM-dd HH:mm:ss]", please help

[jackydan]

I've got exactly the same problem with the same date format.
Here is a piece of the mapping:
'event_time' => [ 'type' => 'date', 'format' => 'yyyy-MM-dd HH:mm:ss' ]