Elastalert: Integration with elasticsearch v5

[rauizab]

Hi

I know that the new version v5 of elasticsearch has just being released, but do you know when it will be available the integration of elastalert with the new version? At the moment I am getting:

INFO:elastalert:Starting up
WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert/_search?size=1000 [status:400 request:0.013s]
WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert_status/_search?_source_include=endtime%2Crule_name&size=1 [status:400 request:0.002s]
ERROR:root:Error querying for last run: TransportError(400, {u'line': 1, u'root_cause': [{u'reason': u'Unknown key for a START_OBJECT in [filter].', u'type': u'parsing_exception', u'line': 1, u'col': 12}], u'type': u'parsing_exception', u'reason': u'Unknown key for a START_OBJECT in [filter].', u'col': 12})
WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert_error?op_type=create [status:400 request:0.003s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'})
Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1033, in writeback
doc_type=doc_type, body=body)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(args, params=params, *kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 227, in create
return self.index(index, doc_type, body, id=id, params=params, op_type='create')
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(args, params=params, *kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 257, in index
_make_path(index, doc_type, id), params=params, body=body)
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 72, in perform_request
self._raise_error(response.status_code, raw_data)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 102, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'})
WARNING:elasticsearch:GET http://elasticsearch:9200/payment-/_search?_source_include=eventtime%2C%2A%2CuserId&ignore_unavailable=true&size=10000 [status:400 request:0.003s]
ERROR:root:Error running query: TransportError(400, {u'line': 1, u'root_cause': [{u'reason': u'no [query] registered for [filtered]', u'type': u'parsing_exception', u'line': 1, u'col': 67}], u'type': u'parsing_exception', u'reason': u'no [query] registered for [filtered]', u'col': 67})
WARNING:elasticsearch:POST http://elasticsearch:9200/elastalert_status/elastalert_error?op_type=create [status:400 request:0.003s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'})
Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1033, in writeback
doc_type=doc_type, body=body)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(args, params=params, *kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 227, in create
return self.index(index, doc_type, body, id=id, params=params, op_type='create')
File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 257, in index
_make_path(index, doc_type, id), params=params, body=body)
File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 72, in perform_request
self._raise_error(response.status_code, raw_data)
File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 102, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, {u'root_cause': [{u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;', u'type': u'action_request_validation_exception'}], u'type': u'action_request_validation_exception', u'reason': u'Validation Failed: 1: an id must be provided if version type or value are set;'})
INFO:elastalert:Ran User1111MakePayment from 2016-11-02 08:20 UTC to 2016-11-02 13:20 UTC: 0 query hits, 0 matches, 0 alerts sent

Thanks for this great tool!!!!

[penekk]

ES 5 has been around for a while already, its the GA version that just came out, any ETA on the support would be lovely :+1:

[watollop]

+1

[lrolsen]

+1

[vijayarulmuthu]

+1

On Sat, Nov 5, 2016 at 5:14 AM -0700, "lrolsen" <[email protected][e-mail]> wrote:

+1

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/Yelp/elastalert/issues/790#issuecomment-258607858, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AH7j4V-D_DPsJ-KmIqww-sOMgZ5FaPG-ks5q7HMggaJpZM4KnOvN.

[sabueso]

@watollop say me to put +1 here

[pasqualguerrero]

+1

[tynor88]

+1

[bennneuh]

+1

[nick-george]

+1 Duplicate of #510

[AndrewPix]

+1

[Alexwii]

+1

[stumyp]

Is it known what is broken and how can we help ?

[bkeifer]

IRC pointed me to https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L159

[doublesea]

+1

[doublesea]

Is it known what is broken and how can we help ?

as i known, it is because "filter" is not available any more in ES 5.0.

There are many "filter" query in EA which will get the 400 HTTP response for the incorrect query clause.

[kojisaiki]

+1

[branchnetconsulting]

+1

[stumyp]

It looks like all needed is to rewrite this query:
https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L159
(thanks @bkeifer )

Documentation says that change should be simple: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/query-dsl-filtered-query.html

I'll try to find some time, my python is a bit rusty :)

[timwsuqld]

My reading of that doc suggests that the change should be:

diff --git a/elastalert/elastalert.py b/elastalert/elastalert.py
index 2cbd553..a13e739 100644
--- a/elastalert/elastalert.py
+++ b/elastalert/elastalert.py
@@ -156,7 +156,7 @@ class ElastAlerter():
         if starttime and endtime:
             es_filters['filter']['bool']['must'].insert(0, {'range': {timestamp_field: {'gt': starttime,
                                                                                         'lte': endtime}}})
-        query = {'query': {'filtered': es_filters}}
+        query = {'query': {'bool': es_filters}}
         if sort:
             query['sort'] = [{timestamp_field: {'order': 'desc' if desc else 'asc'}}]
         return query

I'll see if I can make it work in our ES5 environment

[stumyp]

@timwsuqld: plus if statement to detect ES version

[timwsuqld]

@stumyp Good point. Any idea if we already detect ES version anywhere?

[stumyp]

@timwsuqld : So far I saw only this: https://github.com/Yelp/elastalert/commit/63584350031599a59a90816c85fb2de7ed5a1bd0

Not really a version detection :)

[timwsuqld]

I'm also not sure if the elasticsearch library (https://pypi.python.org/pypi/elasticsearch/5.0.1) is backwards compatible. My understanding is that it should be, so we can use version 5 of the library with older ES clusters.

[stumyp]

From what they say on the link you gave: major version of library must match ES version.
Means, elastalert should split versions too or just warn users for incompatibility ?

[Qmando]

I believe this patch must also be applied, because id must be explicitly set to None.

diff --git a/elastalert/elastalert.py b/elastalert/elastalert.py
index 2cbd553..fd49223 100644
--- a/elastalert/elastalert.py
+++ b/elastalert/elastalert.py
@@ -841,7 +841,8 @@ class ElastAlerter():

         res = es.create(index='kibana-int',
                         doc_type='temp',
-                        body=db_body)
+                        body=db_body,
+                        id=None)

         # Return dashboard URL
         kibana_url = rule.get('kibana_url')
@@ -1015,7 +1016,7 @@ class ElastAlerter():
         if self.writeback_es:
             try:
                 res = self.writeback_es.create(index=self.writeback_index,
-                                               doc_type=doc_type, body=body)
+                                               doc_type=doc_type, body=body, id=None)
                 return res
             except ElasticsearchException as e:
                 logging.exception("Error writing alert info to Elasticsearch: %s" % (e))

[timwsuqld]

For what it's worth, I appear to have no problems using the elasticsearch 2.4.0 library, and the index creation worked without a problem, as do my filters with my patch. I'm sure there are incompatibilities between the 2.4.0 library and ES 5, but maybe the things that we need won't actually hit those incompatibilities?

[Qmando]

The filters work fine? The current format is

query:
  filtered:
    filter:
      bool:
        must: [filters from rule here]

The docs say that filtered has been deprecated. I'm not really an expert on the query DSL though.

Another breaking change: No more fields
https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L243
That should be stored_fields instead.

search_type=count is removed too
https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L318
Instead, you have to add size: 0

These are just what stood out when scanning the breaking changes page

[stumyp]

@Qmando : I believe it should look like

query:
  bool:
    must: [filters from rule here]

[timwsuqld]

@Qmando the filters work fine with my above patch, and the ES 2.4.0 library against a ES 5.0.0 server
The filter comes out something like:

query:
  bool:
    filter:
      bool:
        must: [filters from rule here]

It looks odd with a bool then filter then bool, it just happens we are using the bool filter.
@stumyp I'm not sure if your shorter query would work, it's not what I understood the docs to mean.

[timwsuqld]

I've started a branch (https://github.com/suqld/elastalert/tree/support_es5) that we can work on. I'll try and find ways to make it crash (based on the breaking changes), then commit fixes

[stumyp]

you're right @timwsuqld , my query would work, but a bit differently:

from documentation:

*filter*
The clause (query) must appear in matching documents. 
However unlike must the score of the query will be ignored.

[timwsuqld]

@Qmando Regarding the ID, it's not as easy as id=None :(
https://github.com/elastic/elasticsearch-py/issues/474#issuecomment-256903012

Looks like we need to change the call from create to index

[amirkool]

+1

[timwsuqld]

Given the number of changes, I'm wondering the best way to handle the ES5 changes. We could have a config option (or autodetect ES version) and then in all the places the query needs to be changed, have an if statement. Alternatively, we maintain 2 branches.

Suggestions?

[stumyp]

I'm not sure how much work (and time answering people in issues) will it be to maintain everything in one place with detection/configuration option.

If it is easier to keep separate branch/tag for ES5 - I'm fine with that.

[k2xl]

+1

[bdh1011]

+1

[Fuhunter]

+1 need this for my Bachelorthesis :P

[KenanAlptekin]

+1 would love this for our new ES5 stack.

[timwsuqld]

Please don't +1 this, use the thumbs up on the issue.
The branch at https://github.com/suqld/elastalert/tree/support_es5 is currently working for me in production against ES5. I'm thinking it would be good if @stumyp could create an es5 branch that I can submit a merge request against, so we can get my changes into the elastalert repo.

[Fuhunter]

@timwsuqld
Your branch seems to be working fine! Except mass amount of errors while installation process.
Btw skipped pip install -r requirements.txt in my docker container, but no problems occurred

[stumyp]

@timwsuqld I'm external collaborator, same as you, don't have any control over this repo.
I think @Qmando can do it.

[Qmando]

Hey guys. I took @timwsuqld's changes and added code to grab the elasticsearch version.

Please pull the support_es5 branch and test it for me!!

$ git fetch origin
$ git checkout origin/support_es5
$ pip install elasticsearch>3.0

https://github.com/Yelp/elastalert/pull/820/files

[timwsuqld]

@Qmando Thanks for doing that! I'll try and test that in the next few days. I knew it should be easy to work out the version, just hadn't had time to dig that deep!

[elmalto]

@Qmando with support_es5 I still get
ERROR:root:Error running query: TransportError(400, u'parsing_exception', u'no [query] registered for [query]')

[doublesea]

I encountered the same problem with elmalto, while I tried to resolved it.
It was because the rule.yaml file need to be modified ,too.
I changed it form

filter:
- query:
    query_string:
      query: "Extends:0x60"

to

filter:
  - query_string: 
      query: "Extends:0x60"

Then it works.

It seems that the sample need to modified in http://elastalert.readthedocs.io/en/latest/recipes/writing_filters.html#writingfilters

[i-sam]

+1

[Hronom]

Guys what the progress on this? We are extremely need this...

[Qmando]

@Hronom The branch is out. I would like more feedback as I don't have an ES5 test environment right now. There are some things that need to be done, like use_terms_query doesn't work.

[Hronom]

@Qmando So I can use branch support_es5?

[Qmando]

It's not 100% complete, but basic functionality should work. I'm looking for feedback, so I would love it if you tried to use it and reported any issues.

[khezen]

Hi,

• I installed elastalert on debian:jessie with the following script:

#!/bin/bash

# misc
apt-get update
apt-get install git wget build-essential python python-pip python-dev -y
wget https://bootstrap.pypa.io/ez_setup.py -O - | python
pip install --upgrade pip
pip install --upgrade six

#elastalert
git clone https://github.com/Yelp/elastalert.git /elastalert
cd /elastalert
git checkout support_es5
python setup.py install
pip install -r requirements.txt
cp -f config.yaml.example /.backup/elastalert.yml

# purge
apt-get remove git wget build-essential python-pip python-dev -y

• I am running es 5.0.2

• When I execute elastalert-test-rule example.yml

  • example.yml:

es_host: elasticsearch
es_port: 9200
es_username: elastic
es_password: changeme
name: example
index: metricbeat-*
type: any
filter:
- range:
    system.core.cpu.idle.pct:
        from: 0
        to: 0.1
alert:
- command
command: ["echo", "\"rules.example triggered\""]

• I am getting the following response:

elasticsearch:GET http://elasticsearch:9200/metricbeat-*/_search?ignore_unavailable=true&size=1 [status:400 request:0.004s]
Error running your filter:
RequestError(400, u'parsing_exception', {u'status': 400, u'error': {u'line': 1, u'root_cause': [{u'reason': u'no [query] registered for [filtered]', u'type': u'parsing_exception', u'line': 1, u'col': 68}], u'type': u'parsing_exception', u'reason': u'no [query] registered for [filtered]', u'col': 68}})

I thought this was because filtered queries do not exist anymore and have been replaced by the bool notation but it looks like you fixed it already.

It could come from here

Any insight would be helpful.

[josephharding]

@Qmando I've been using the support_es5 branch for the past few days without issue

[Hronom]

Guys previously I use in rule this:

filter:
- query:
    nested:
      path: "objects"
      filter:
      - and:
        - term:
            objects.id: "12323443456757687890"
        - range:
            objects.mass:
              from: 0.012
              to: 1.0

But with this I get error:

ERROR:root:Error running query: TransportError(400, u'parsing_exception', u'no [query] registered for [query]')

What I need to change to make it work?

[jabs76]

anyone make it work with wildcard filter?

[tfgm-bud]

I know this is closed but I wanted to report that the support_es5 branch, along with the changes suggested by @doublesea above to one of my alerts that had a "query", seems to be working with ES 5.1.1.

[i-sam]

Does it normally work with ES 5.1.1 ? still waiting PR closing and merging to master. Thanks for your work guys

[tfgm-bud]

@i-sam, I'm not seeing any issues with ES 5.1.1 using the branch (other than the query/query_string thing described above). That said, I only have half a dozen rules and they are pretty simple

[PeterDefendo]

Hi, I'm tying to use elastalert branch support_es5 against ES 5.0.2.
With the following config.yaml

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: example_rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: localhost

# The Elasticsearch port
es_port: 9200

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

# Email configuration
#smtp_host: "smtp.email.se"
#from_addr: "[email protected]"

# a list of email addresses to send alerts to
#email:
#- "[email protected]"

test.yaml

es_host: localhost
es_port: 9200
name: example
index: logs-*
type: any
filter:
- range:
    server_port:
        from: 0
        to: 80
alert:
- command
command: ["echo", "\"rules.example triggered\""]

But when running elastalert-test-rule example_rules/test.yaml we get the following error:

Successfully loaded example
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
Got 489338 hits from the last 1 day
Available terms in first hit:
        service_name
        @timestamp
        browser_os_name
        type
        received_at
        uri_stem
        bytes_received
        received_from
        cs_host
        protocol_status
        server_ip
        user_name
        method
        client_ip
        server_port
        tags
        browser_device
        time_taken
        win32_status
        browser_os
        geoip.region_code
        geoip.longitude
        geoip.region_name
        geoip.ip
        geoip.continent_code
        geoip.postal_code
        geoip.country_code3
        geoip.country_code2
        geoip.city_name
        geoip.dma_code
        geoip.country_name
        geoip.latitude
        geoip.timezone
        geoip.location
        bytes_sent
        browser_name
        protocol_substatus
        @version
        uri_query
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent. To send them, use --verbose.
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): es
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-test-rule", line 11, in <module>
    load_entry_point('elastalert==0.1.4', 'console_scripts', 'elastalert-test-rule')()
  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.4-py2.7.egg/elastalert/test_rule.py", line 324, in main
    test_instance.run_rule_test()
  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.4-py2.7.egg/elastalert/test_rule.py", line 319, in run_rule_test
    self.run_elastalert(rule_yaml, conf, args)
  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.4-py2.7.egg/elastalert/test_rule.py", line 232, in run_elastalert
    client = ElastAlerter(['--debug'])
  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.4-py2.7.egg/elastalert/elastalert.py", line 116, in __init__
    self.five = self.is_five()
  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.4-py2.7.egg/elastalert/elastalert.py", line 125, in is_five
    info = self.writeback_es.info()
  File "/home/defendo/.local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 71, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/defendo/.local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 222, in info
    return self.transport.perform_request('GET', '/', params=params)
  File "/home/defendo/.local/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/defendo/.local/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 80, in perform_request
    self.log_request_fail(method, url, response.request.path_url, body, time.time() - start, exception=e)
UnboundLocalError: local variable 'response' referenced before assignment

What could be the source of this issue?

[Qmando]

That bug is from the elasticsearch python library, possibly because of a version mismatch between the cluster and the library.
What version of elasticsearch-python do you have?
pip freeze | grep elasticsearch

[PeterDefendo]

pip freeze | grep elasticsearch
outputs elasticsearch==5.0.1 but we have ES 5.0.2 installed could that be the reason?

[PeterDefendo]

If I understand the output correctly it first connects to localhost as defined in both test.yaml and config.yaml and that works as intended, then it connects to es and fails as we have no computer named es...

Where does it get the hostname es from?

[Qmando]

That's a (very poor) default value. If config.yaml is in the same directory, it should open that up and use it. If not, you can add --config /path/to/config.yaml and it will pick it up. The whole elastalert-test-rule script needs to major work in general.

Your library version is fine, but there's still a bug in that library that is obfuscating the connection error.

[ThomasdOtreppe]

For reference: elastic/elasticsearch-py#476

[stumyp]

@Qmando , @timwsuqld : I think good way to test compatibility and see if any deprecated queries left is to run your elastalert instance on cluster with enabled deprecation logging:

curl -XPUT localhost:9200/_cluster/settings -d '{   "transient": { "logger": { "deprecation": "DEBUG" } } }'

This is what I have in my logs:

a lot of this:

[2016-12-28 16:01:55,259][DEBUG][deprecation.common       ] Deprecated field [ignore_unmapped] used, replaced by [unmapped_type]

and this:

[2016-12-28 16:01:56,658][DEBUG][deprecation.index.query  ] The [query] filter is deprecated, you can now use queries as filters directly.
[2016-12-28 16:01:56,658][DEBUG][deprecation.index.query  ] The [and] query is deprecated, please use a [bool] query instead with [must] clauses.
[2016-12-28 16:01:56,658][DEBUG][deprecation.index.query  ] The [filtered] query is deprecated, please use a [bool] query instead with a [must] clause for the query part and a [filter] clause for the filter part.

[PeterDefendo]

@Qmando Thanks adding --config /path/to/config.yaml resolved the issue.

[bHoskins07]

@tfgm-bud What elasticsearch python module version are you using? Also, can you post some of those simple queries. Thank You!

[tfgm-bud]

@bHoskins07 Not sure what you are asking with "What elasticsearch python module version are you using?". I'm using the github version and changing to the support_es5 branch:

git checkout origin/support_es5

And a simple query would be:

filter:
- term:
    env: "live"
- query_string:
    query: 'msgtype:activity AND ("transaction" adjusted unchanged cancelled)'

Not sure how much that really helps you though....

[Qmando]

@tfgm-bud To find the version of the elasticsearch module, type $ pip freeze | grep elasticsearch

[tfgm-bud]

@Qmando oh, I mis-read @bHoskins07 question and was thinking he was asking about elastalert.

elasticsearch==2.4.0

[PeterDefendo]

Im new to this but why does this work:

filter:
- query_string:
    query: "received_from: Prod AND (level: FATAL OR level: ERROR)"

But this doesn't:

filter:
- and:
    - term:
        received_from: "Prod"
    - or:
        - term:
            level: "FATAL"
        - term:
            level: "ERROR"

Aren't they the same thing?

The latter gives the following error {'message': "Error running query: TransportError(400, u'parsing_exception', u'[and] query malformed, no start_object after query name')"...

I'm guessing this is the same issue as @Hronom has.

[bHoskins07]

Thank You @tfgm-bud. I was testing 5.1.1 using elastalert-test-rule which was not working. When I switched to $ python -m elastalert.elastalert --rule it worked. Also, I am using python elasticsearch module 5.0.1

[tfgm-bud]

@bHoskins07 - I see what you mean. I upgraded to elasticsearch-5.0.1 with sudo pip install --upgrade elasticsearch and elastalert-test-rule fails:

$ elastalert-test-rule --config /etc/elastalert/config.yaml /etc/elastalert/rules/news.yaml 
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-test-rule", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3015, in <module>
    @_call_aside
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2999, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3028, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 657, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 670, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 849, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'elasticsearch<3.0.0' distribution was not found and is required by elastalert

I made the following change to try to fix it and it works for me:

user@elk:~/elastalert[support_es5]$ git diff
diff --git a/setup.py b/setup.py
index 1f433c1..ebf817a 100644
--- a/setup.py
+++ b/setup.py
@@ -23,7 +23,7 @@ setup(
     package_data={'elastalert': ['schema.yaml']},
     install_requires=[
         'argparse',
-        'elasticsearch<3.0.0',  # Elastalert is not yet compatible with ES5
+        'elasticsearch',
         'jira==0.32',  # jira.exceptions is missing from later versions
         'jsonschema',
         'mock',

@timwsuqld can you incorporate this into your branch? -- Bud

[deepthi0311]

@Qmando , I had the same issue while working with master branch. After checking out support_es5 branch and switching to that branch, it works perfectly fine. Can you please merge this branch to master branch and let me know after merging?

[aatchison]

Wahoo! I'm watching and waiting for ES5!

[dadadom]

I added an issue with the branch as a comment in the PR. Is that OK? Should I mention the problem here or should I create a separate issue for that branch?

[deepthi0311]

Hi,
Yes, thanks for adding it in the PR. No need to create a separate issue for that branch.

Thanks and regards,
Deepthi D

From: Dominik Sandjaja [mailto:[email protected]]
Sent: Monday, January 9, 2017 8:53 PM
To: Yelp/elastalert [e-mail]
Cc: Deepthi D [e-mail]; Comment [e-mail]
Subject: Re: [Yelp/elastalert] Integration with elasticsearch v5 (#790)

I added an issue with the branch as a comment in the PR. Is that OK? Should I mention the problem here or should I create a separate issue for that branch?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/Yelp/elastalert/issues/790#issuecomment-271311477, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AWWRWKEbKy_zcbgsbmW228UAHZqtCuVMks5rQlC4gaJpZM4KnOvN.

[ssinhaonline]

@Qmando thank you for the support_es5 branch. Works for me with changes to query as suggested by @doublesea

[rptete]

then, I would try elastalert on ES5?

[dreeco]

+1

[ssinhaonline]

@rptete Go right ahead.

[hollowimage]

+1. is there an eta for merging the support_es5 branch into mainstream?

[nsgeorgi]

Please notify when you merge this branch to the master one. Thanks

[Qmando]

This has been merged with master and released.

[yanncarlier]

Im running elasticsearch 5.2.2
wen I run elastalert-create-index i have this error:

root@logstash:/data/elastalert# elastalert-create-index
Traceback (most recent call last):
File "/usr/local/bin/elastalert-create-index", line 6, in
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3037, in
@_call_aside
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3021, in _call_aside
f(args, *kwargs)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3050, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 655, in _build_master
ws.require(__requires__)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 969, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 855, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jira>=1.0.10' distribution was not found and is required by elastalert
root@logstash:/data/elastalert#

[yanncarlier]

after
pip install -r requirements.txt
pip install functools32

its working, thanks