Elastalert: No mapping found for [@timestamp] in order to sort on]

[sandipanbasu]

I am consistently encountering an error while running ElastAlert's alert. The error states that there is no mapping for the "@timestamp" fields. The alert type that I am using is spike. After looking around, it seems apparent that "@timestamp" is an internal field in Logstash. I am yet to have a Logstash instance yet hence this error. This however brings up following questions --

1) Does ElastAlert always assumes that you have Logstash as data processor ?
2) The sort and range checks have "@timestamp". One can add to this list but cannot remove the "@timestamp" field. Is there any way to override this ?
3) Is there any option to override the @timestamp in a rule ?

[Qmando]

1) No, although it uses many of the same defaults, such as @timestamp
2 + 3) Yes, by setting the option timestamp_field. It should still be an ISO 8601 format timestamp, such as "2015-06-08T12:00:00Z", though other formats may work, as long as they can be parsed by dateutil.parser. Only ISO 8601 timestamps indexed as dateOptionalTime have been thoroughly tested.

[sandipanbasu]

Thanks !! Works now with the timestamp_field override.

[wybrono]

How to use "timestamp_field"

[Qmando]

@wybrono
did you run elastalert-create-index? If you did and still get this error, what is the name of the field in contains timestamps?

timestamp_field: name_of_field

[wybrono]

Thanks,I have fixed it

[islammohamed]

@wybrono sorry I didn't get it, how did you fixed the problem?

[mhepfer]

@islammohamed Add the line:
timestamp_field: name_of_field
to the rule you're trying to create. This is the example from the docs with timestamp field added:

```# From example_rules/example_frequency.yaml
es_host: elasticsearch.example.com
es_port: 14900
name: Example rule
type: frequency
timestamp_field: name_of_field
index: logstash-*
num_events: 50
timeframe:
hours: 4
filter:

• term:
  some_field: "some_value"
  alert:
• "email"
  email:
• "[email protected]"
  ```

[anjan-cele]

Hi @Qmando

When I added timestamp_field, error is coming as :

failed to parse date field [2017-08-14T15:51:59.716012Z] with format [yyyy-MM-dd HH:mm:ss]

while my timestamp_field is completiondt and,in log, format for this seems to be correct.

What could be the issue?

Thanks

[Qmando]

Try adding

timestamp_format: "%Y-%m-%d %H:%M:%S%z"
timestamp_type: custom

or maybe

timestamp_format: "%Y-%m-%d %H:%M:%SZ"
timestamp_type: custom

[sathishdsgithub]

@anjan-cele

In case if you're using graylog and getting the timestamp error. You can check https://github.com/Yelp/elastalert/pull/1022

[anjan-cele]

Hi @Qmando

When I added both the options in my .yaml rule file i.e

timestamp_format: "%Y-%m-%d %H:%M:%S%z"
timestamp_type: custom

or

timestamp_format: "%Y-%m-%d %H:%M:%SZ"
timestamp_type: custom

I am getting error like :
ERROR:root:Error running query: TransportError(400, u'search_phase_execution_exception', u'failed to parse date field [2017-08-15 18:53:14Z] with format [yyyy-MM-dd HH:mm:ss]')

@sathishdsgithub

I am using dotnetlog and I am unfamiliar with graylog.

Please suggest.

Thanks

[sathishdsgithub]

@anjan-cele

Can you try this https://github.com/Yelp/elastalert/pull/1022 and see if it fix the issue.

[anjan-cele]

HI @sathishdsgithub

I have tried adding this from #1022
timestamp_type: custom
timestamp_format: '%Y-%m-%dT%H:%M:%S.%fZ'
timestamp_format_expr: 'ts[:23] + ts[26:]'

But still facing the same issue.

Do I need to add any more lines somewhere to implement this #1022.

Thanks

[Qmando]

@anjan-cele
try

timestamp_format: "%Y-%m-%d %H:%M:%S"
timestamp_type: custom

TLDR: your Elasticsearch has @timestamp mapped to a strict format that doesn't accept timezones

[anjan-cele]

Hi @Qmando

It worked. Lot of thanks.
Regards
Anjan

[piellick]

H ieveryone ,

this conf working for index managed by graylog ("T" was removed from the format) :

timestamp_type: custom
timestamp_format: '%Y-%m-%d %H:%M:%S.%fZ'
timestamp_format_expr: 'ts[:23] + ts[26:]'