Elastalert: Nested field support

[vikas8190]

I need to add a cardinality rule where the cardinality field is like "proc.name"(data from topbeat for system statistics) . However the rule doesnt seem to be working properly for nested field which have dot in its name. Any help please

[jgrmnprz]

Same problem here with a spike rule and query_key :

WARNING:elasticsearch:POST http://127.0.0.1:9200/elastalert_status/elastalert?op_type=create [status:400 request:0.006s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(400, {u'root_cause': [{u'reason': u"Field name [event.field1] cannot contain '.'", u'type': u'mapper_parsing_exception'}], u'type': u'mapper_parsing_exception', u'reason': u"Field name [event.field1] cannot contain '.'"})

[hjdr4]

Same problem here, cardinality rule does not work with nested field, and does not fire any error.

[pjl]

v0.1.3 adds support for nested cardinality_field.

[daniilyar]

Same problem here, Elastalert 0.14, frequency rule with query_key:

ERROR:root:Error writing alert info to Elasticsearch: TransportError(400, {u'root_cause': [{u'reason': u"Field name [k8.container,class_name_full,line_number] cannot contain '.'", u'type': u'mapper_parsing_exception'}], u'type': u'mapper_parsing_exception', u'reason': u"Field name [k8.container,class_name_full,line_number] cannot contain '.'"})
Traceback (most recent call last):
  File "elastalert/elastalert.py", line 1021, in writeback
    doc_type=doc_type, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 227, in create
    return self.index(index, doc_type, body, id=id, params=params, op_type='create')
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 257, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 301, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 72, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 102, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, {u'root_cause': [{u'reason': u"Field name [k8.container,class_name_full,line_number] cannot contain '.'", u'type': u'mapper_parsing_exception'}], u'type': u'mapper_parsing_exception', u'reason': u"Field name [k8.container,class_name_full,line_number] cannot contain '.'"})

[Qmando]

This should be fixed by https://github.com/Yelp/elastalert/pull/853/

[armoured-ape]

Similar issue from the other side. The alerter (hive) can referenxe {match[event_data]} but not {match[event_data.CommandLine]}

[bangejsans]

I am seeing the same issue for nested fields for hive alerts.

[Qmando]

The Hive alerter had this done in a very weird way, I'll try to get it fixed if I have time.

[Qmando]

Instead of {match[event_data.CommandLine]} you can use {match[event_data][CommandLine]}