Elastalert: Parse date field error

[valentinmagrez]

Im using elastalert in a docker container and I got trouble with datetime format.
I have a rule file that trigger for new term:

name: Demo rule
type: new_term

index: my_index

fields:
 - "id"

timestamp_field: "my_time_field"
timestamp_type: custom
timestamp_format: "%Y-%m-%d %H:%M"

realert:
  minutes: 0
filter:
- term:
    type: "ERROR"

alert:
- "debug"

My index was created without @timestamp so I added:

timestamp_field: "my_time_field"
timestamp_type: custom
timestamp_format: "%Y-%m-%d %H:%M"

In this index I defined my_time_field like this:

                "my_time_field": {
                    "type": "date",
                    "format": "yyyy-MM-dd HH:mm"
                }

But when I launch elastalert I got this error:

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/elastalert/elastalert/elastalert.py", line 1477, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/opt/elastalert/elastalert/elastalert.py", line 1472, in main
    client = ElastAlerter(args)
  File "/opt/elastalert/elastalert/elastalert.py", line 93, in __init__
    self.conf = load_rules(self.args)Parse date field 
  File "elastalert/config.py", line 395, in load_rules
    raise EAException('Error loading file %s: %s' % (rule_file, e))
elastalert.util.EAException: Error loading file /opt/rules/demo_new.yaml: Error initializing rule Demo rule: Error searching for existing terms: RequestError(400, u'search_phase_execution_exception', {u'status': 400, u'error': {u'failed_shards': [{u'node': u'ocZ__DEqQ-SAP1gMNMDvbQ', u'index': u'my_index', u'reason': {u'caused_by': {u'reason': u'Parse failure at index [10] of [2017-02-05T14:35:04.720418Z]', u'type': u'illegal_argument_exception'}, u'reason': u'failed to parse date field [2017-02-05T14:35:04.720418Z] with format [yyyy-MM-dd HH:mm]', u'type': u'parse_exception'}, u'shard': 0}], u'root_cause': [{u'reason': u'failed to parse date field [2017-02-05T14:35:04.720418Z] with format [yyyy-MM-dd HH:mm]', u'type': u'parse_exception'}], u'caused_by': {u'caused_by': {u'reason': u'Parse failure at index [10] of [2017-02-05T14:35:04.720418Z]', u'type': u'illegal_argument_exception'}, u'reason': u'failed to parse date field [2017-02-05T14:35:04.720418Z] with format [yyyy-MM-dd HH:mm]', u'type': u'parse_exception'}, u'grouped': True, u'reason': u'all shards failed', u'phase': u'query_fetch', u'type': u'search_phase_execution_exception'}})

Moreover, the date _2017-02-05T14:35:04.720418Z_ correspond to none of my data.....

FYI:
If I change my config like this

                "my_time_field": {
                    "type": "date",
                    "format": "date_optional_time"
                }

And that I remove

timestamp_type: custom
timestamp_format: "%Y-%m-%d %H:%M"

Everything is ok. Am I doing domething something wrong in the first part ?

[Qmando]

This looks like a bug. This line (https://github.com/Yelp/elastalert/blob/master/elastalert/ruletypes.py#L557) is ignoring the custom timestamp_format. It's specific to the new_term rule. I'll fix this.