Elastalert: No mapping found for [@timestamp] in order to sort on]

[morganwildermuth]

I see that this person had the same issue (https://github.com/Yelp/elastalert/issues/122), but, unlike them, I am ingesting logs from logstash and my @timestamp is in the ISO8601 format. An example @timestamp in Elasticsearch that I have being 2015-08-13T23:29:56.000Z.

My error looks like this:

root@ip-10-0-20-114:/usr/local/elastalert# python elastalert/elastalert.py
ERROR:root:Error querying for last run: TransportError(400, u'SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; 
shardFailures {[8a3NL65NQiKa9wsxgwklyA][elastalert_status][0]: SearchParseException[[elastalert_status][0]: from[-1],size[-1]: 
Parse Failure [Failed to parse source [{"filter": {"term": {"rule_name": "whiteListedIPs"}}, "sort": {"@timestamp": {"order": "desc"}}}]]]; 
nested: SearchParseException[[elastalert_status][0]: from[-1],size[-1]: 
Parse Failure [No mapping found for [@timestamp] in order to sort on]]; }{[8a3NL65NQiKa9wsxgwklyA][elastalert_status][1]:
 SearchParseException[[elastalert_status][1]: from[-1],size[-1]: 

I've pretty much followed the documentation for getting started so I don't quite know how much more simple my test case could be. I did use a different folder for rules and created my own test case that basically just mimics the example_frequency.yaml.

Any ideas of how I should proceed?

[Qmando]

Did you run elastalert-create-index as in http://elastalert.readthedocs.org/en/latest/running_elastalert.html#setting-up-elasticsearch?

[morganwildermuth]

Yep.

root@ip-10-0-20-114:/usr/local/elastalert# elastalert-create-index
New index name? (Default elastalert_status)
Name of existing index to copy? (Default None)
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 9, in <module>
    load_entry_point('elastalert==0.0.55', 'console_scripts', 'elastalert-create-index')()
  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.0.55-py2.7.egg/elastalert/create_index.py", line 87, in main
    es.indices.create(index)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/indices.py", line 102, in create
    params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 89, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 105, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'IndexAlreadyExistsException[[elastalert_status] already exists]')

Ran it again just to make sure :)

[Qmando]

Hm. Strange. The error is because "@timestamp" doesn't exist as a sortable field in the elastalert_status index. elastalert-create-index should upload the mapping for that here https://github.com/Yelp/elastalert/blob/master/elastalert/create_index.py#L68.

If you run curl -XGET 'http://elasticsearch.example.com:14900/elastalert_status/_mapping/', what does it output?

[morganwildermuth]

Assuming I translated that right...

root@ip-10-0-20-114:/usr/local/elastalert# curl -XGET 'http://127.0.0.1:9200/elastalert_status/_mapping/'

{"elastalert_status":{"mappings":{"elastalert_error":{"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"data":{"properties":{"rule":{"type":"string"}}},"message":{"type":"string"},"traceback":{"type":"string"}}}}}}

[Qmando]

What version of Elasticsearch are you using? When I create a blank index by running elastalert-create-index and then that curl command, I get

{
  "elastalert_status": {
    "mappings": {
      "silence": {
        "properties": {
          "rule_name": {
            "type": "string",
            "index": "not_analyzed"
          },
          "until": {
            "type": "date",
            "format": "dateOptionalTime"
          }
        }
      },
      "elastalert_status": {
        "properties": {
          "@timestamp": {
            "type": "date",
            "format": "dateOptionalTime"
          },
          "rule_name": {
            "type": "string",
            "index": "not_analyzed"
          }
        }
      },
      "elastalert": {
        "properties": {
          "aggregate_id": {
            "type": "string",
            "index": "not_analyzed"
          },
          "match_body": {
            "type": "object",
            "enabled": false
          },
          "rule_name": {
            "type": "string",
            "index": "not_analyzed"
          }
        }
      },
      "elastalert_error": {
        "properties": {
          "data": {
            "type": "object",
            "enabled": false
          }
        }
      }
    }
  }
}

which is the full set of mappings. It appear that only the elastalert_error mapping was added to your index.

I'm not sure what would cause this, but you could try either:

• Recreate the index and run the script again
• Use the mapping API and to manually upload the mappings that I just posted. https://www.elastic.co/guide/en/elasticsearch/reference/1.6/indices-put-mapping.html

[morganwildermuth]

Fascinatingly bizarre. Thanks for helping me figure out what the underlying issue was!

root@ip-10-0-20-114:/usr/local/elastalert# elasticsearch -v
Version: 1.5.1, Build: 5e38401/2015-04-09T13:41:35Z, JVM: 1.7.0_79

[morganwildermuth]

After deleting the index and running it again everything worked; tinkering with my Chef recipes now to figure out how to fix this on start-up. Thanks again for your help.

[carlosvega]

Same problem here with last Elasticsearch 2.3.2

ERROR:root:Error while querying for alert silence status: TransportError(400, {u'failed_shards': [{u'node': u'GEXftzBpS2W-pilvUV3d4A', u'index': u'elastalert_status', u'reason': {u'reason': u'No mapping found for [until] in order to sort on', u'type': u'search_parse_exception'}, u'shard': 0}], u'root_cause': [{u'reason': u'No mapping found for [until] in order to sort on', u'type': u'search_parse_exception'}], u'grouped': True, u'reason': u'all shards failed', u'phase': u'query', u'type': u'search_phase_execution_exception'})

[Qmando]

@carlosvega Can you show the output of curl -XGET 'http://localhost:9200/elastalert_status/_mapping/ ?

The error here appears to be that the index did not getting full set when running elastalert-create-index. I would try deleting the elastalert_status index and running that again.

[carlosvega]

When running elastsalert I get the following error:

WARNING:elasticsearch:GET http://localhost:9200/elastalert_status/elastalert/_search?size=1000 [status:404 request:0.045s]
WARNING:elasticsearch:GET http://localhost:9200/elastalert_status/silence/_search?_source_include=until%2Cexponent&size=1 [status:404 request:0.002s]
ERROR:root:Error while querying for alert silence status: TransportError(404, {u'index': u'elastalert_status', u'resource.type': u'index_or_alias', u'root_cause': [{u'index': u'elastalert_status', u'reason': u'no such index', u'type': u'index_not_found_exception', u'resource.id': u'elastalert_status', u'resource.type': u'index_or_alias'}], u'reason': u'no such index', u'type': u'index_not_found_exception', u'resource.id': u'elastalert_status'})

And this is the mapping after it execution.

curl -XGET 'http://localhost:9200/elastalert_status/_mapping/'
{"elastalert_status":{"mappings":{"elastalert_error":{"properties":{"@timestamp":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"data":{"properties":{"rule":{"type":"string"}}},"message":{"type":"string"},"traceback":{"type":"string"}}},"elastalert":{"properties":{"@timestamp":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"alert_info":{"properties":{"output_file":{"type":"string"},"type":{"type":"string"}}},"alert_sent":{"type":"boolean"},"alert_time":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"match_body":{"properties":{"_id":{"type":"string"},"_index":{"type":"string"},"_type":{"type":"string"},"fecha":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"peticiones_pendientes_respuesta_tras_intervalo":{"type":"long"},"servicio":{"type":"string"}}},"rule_name":{"type":"string"}}},"elastalert_status":{"properties":{"@timestamp":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"endtime":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"hits":{"type":"long"},"matches":{"type":"long"},"rule_name":{"type":"string"},"starttime":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"time_taken":{"type":"double"}}}}}}%

[carlosvega]

Any news on this?

[ghost]

@carlosvega I had this problem until I realized I was missing an @timestamp property in my custom log data.

[Qmando]

Weird, how can it throw a index_not_found_exception for elastalert_status when you clearly just got the mapping for that index.

I also see that silence type is missing from your mapping. I think there is a race condition of sorts here, https://github.com/Yelp/elastalert/blob/master/elastalert/create_index.py#L116

But you would also get this error, albeit from a different line, if you don't have a @timestamp field in your actual data. You can change this by setting timestamp_field in the rule to a different field name.

[mpahuja]

thanks this resolved the issue for me

"But you would also get this error, albeit from a different line, if you don't have a @timestamp field in your actual data. You can change this by setting timestamp_field in the rule to a different field name."

[mattheworiordan]

Had this same issue with ElasticSearch 5, simply deleted the old mapping with curl -XDELETE 'localhost:9200/elastalert_status' and ran elastalert-create-index again to fix it.

[allantaylor81]

Had this same issue with ElasticSearch 2.4.4, simply deleted the old mapping with curl -XDELETE 'localhost:9200/elastalert_status' and ran elastalert-create-index again to fix it

ERROR:root:Error finding recent pending alerts: TransportError(400, u'search_phase_execution_exception', u'No mapping found for [alert_time] in order to sort on')

[Trustys]

Same issue with ES 2.4.2 (elastalert version is v0.1.12):

nested: SearchParseException[No mapping found for [@timestamp] in order to sort on];

I have the field @timestamp in ES:

curl -X GET 'http://localhost:9200/graylog_383/message/ee044f01-54d0-11e7-802f-0242ac120003?pretty'

{ "_index" : "graylog_383", "_type" : "message", "_id" : "ee044f01-54d0-11e7-802f-0242ac120003", "_version" : 1, "found" : true, "_source" : { "@timestamp" : "2017-06-19T09:23:18.000Z", "scoped" : "True", "full_message" : "blablabla", "event_prog" : "ASA_session", "hashcode2" : "0x0", "event_time" : "2017-06-19T11:23:18+02:00", "event_ip" : "1.2.3.4" } }

Mapping looks like OK:

curl -XGET 'http://127.0.0.1:9200/elastalert_status/_mapping/
{"elastalert_status":{"mappings":{"past_elastalert":{"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"aggregate_id":{"type":"string","index":"not_analyzed"},"match_body":{"type":"object","enabled":false},"rule_name":{"type":"string","index":"not_analyzed"}}},"elastalert_error":{"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"data":{"type":"object","enabled":false},"message":{"type":"string"},"traceback":{"type":"string"}}},"elastalert_status":{"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"endtime":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"hits":{"type":"long"},"matches":{"type":"long"},"rule_name":{"type":"string","index":"not_analyzed"},"starttime":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"time_taken":{"type":"double"}}},"elastalert":{"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"aggregate_id":{"type":"string","index":"not_analyzed"},"aggregation_key":{"type":"string"},"alert_exception":{"type":"string"},"alert_info":{"properties":{"fir_subject":{"type":"string"},"type":{"type":"string"}}},"alert_sent":{"type":"boolean"},"alert_time":{"type":"date","format":"dateOptionalTime"},"match_body":{"type":"object","enabled":false},"rule_name":{"type":"string","index":"not_analyzed"}}},"silence":{"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"exponent":{"type":"long"},"rule_name":{"type":"string","index":"not_analyzed"},"until":{"type":"date","format":"dateOptionalTime"}}}}}}

I deleted elastalert_status then ran elastalert-create-index again but the issue is still there ...

Please note that I see this error in elasticsearch logs, not on elastalert logs (but this is related)

Any other workaround ?

[Qmando]

Can you post the context of that error message? I don't think the error is coming from elastalert_status, but rather one of your graylog indexes. I'm not very familiar with graylog but if it's possible to have a message without @timestamp, one of the graylog_* indexes might be missing that field. elastalert_status mapping looks fine to me.

[Qmando]

@YogeeshKapila That's not coming from elastalert_status, that's coming from audit index. There is no @timestamp field in that index. Check what the proper time picker is and set timestamp_field in your rule.

[Qmando]

No @timestamp. I see "timestamp" instead. So add
timestamp_field: timestamp

[Qmando]

They are both "timestamp", so use timestamp_field: timestamp

[anjan-cele]

Hi @Qmando

I am using ES 5.0. When I am running my rule. Following error is coming.

ERROR:root:Error finding recent pending alerts: TransportError(400, u'search_phase_execution_exception', u'No mapping found for [alert_time] in order to sort on') {'sort': {'alert_time': {'order': 'asc'}}, 'query': {'bool': {'filter': {'range': {'alert_time': {'to': '2017-07-25T15:43:52.149157Z', 'from': '2017-07-23T15:43:52.149125Z'}}}, 'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}}}}
Traceback (most recent call last):
File "/home/watchman/elastalert/elastalert/elastalert.py", line 1387, in find_recent_pending_alerts
size=1000)
File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 73, in _wrapped
return func(args, params=params, *kwargs)
File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 623, in search
doc_type, '_search'), params=params, body=body)
File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 312, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_requests.py", line 90, in perform_request
self._raise_error(response.status_code, raw_data)
File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 125, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, u'search_phase_execution_exception', u'No mapping found for [alert_time] in order to sort on')

Even elastalert_index has been created correctly:

and it is like :

$ curl amdc2webl24:9200/elastalert_status/_mapping
{"elastalert_status":{"mappings":{"elastalert_error":{"properties":{"@timestamp":{"type":"date"},"data":{"properties":{"rule":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"message":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"traceback":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"elastalert_status":{"properties":{"@timestamp":{"type":"date"},"endtime":{"type":"date"},"hits":{"type":"long"},"matches":{"type":"long"},"rule_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"starttime":{"type":"date"},"time_taken":{"type":"float"}}}}}}

Do I need to include alert_time in the index. If so, then how I can include it.

Please help me.
Thanks

[sathieu]

FYI I had the problem with empty indices. To fix this, I deleted those indices.

[jgrevich]

I ran into this problem and found that one can recreate this issue if you run elastalert-create-index while elastalert is running. I was able to fix my problem by stopping elastalert, deleting the index, and creating a new one with elastalert-create-index. I figure our chef recipe must be starting the service before running the create index task.

[Bhairab123]

No @timestamp. I see "timestamp" instead. So add
timestamp_field: timestamp

Worked fine for me after mentioning the timestamp_field: in rule yaml file