Elastalert: ElasticSearch 7.x support

[kago-dk]

I upgraded my Elasticsearch installation to 7.0.0 and since then elastalert is giving errors related to "[_source_include] -> did you mean any of [_source_includes, _source_excludes]", which I assume is due to "Remove deprecated url parameters _source_include and _source_exclude #35097 (issues: #22792, #33475)" (https://www.elastic.co/guide/en/elasticsearch/reference/7.0/release-notes-7.0.0.html)

Is there in plan to make Elastalert compatible with ElasticSearch 7.0.0 anytime soon?

[matsgoran]

I've currently got a pull request trying to resolve this issue https://github.com/Yelp/elastalert/pull/2194 . We are currently running this branch against our ES7.0-beta1 server. If the change is accepted into the main project is unknown

[pipari666]

Same problem here. Worked nicely before. I hope it will be merged soon, else i have to downgrade or change component for alerts.

best regards.

[moetemad]

Are there any updates on using elastalert on elastic 7.0?

[Yv-o-rL]

Hi!
Is there any type of workaround? I couldn't postpone the upgrade, but the alert is pretty important for me :(

[Qmando]

Try installing elastalert==0.2.0b2. I've just merged in a branch for ES7 support, but it's still under the beta releases as I've not extensively tested it.

[jbecker94]

I've tested the version 0.2.0b2 but unfortunatly. I'm still getting the error contains unrecognized parameter: [_source_include] -> did you mean any of [_source_includes, _source_excludes]?')"

I have been using the bitsensor/elastalert dockerfile to build an new image. Any other ideas?

[matsgoran]

@jbecker94 Can you provide more information? In order to track down a possible bug we need to know which version of Elasticsearch you're running against so we can replicate this. Just to make sure can you compare the elastalert.py file in the docker image against the same file in tag 0.2.0b2? The complete stack trace would also be of benefit to us since we get more information of the origins of this error.

[jbecker94]

@matsgoran I have done a little bit more troubleshooting and it is now working using the follwing commands according to bitsensor:

git clone https://github.com/bitsensor/elastalert.git && cd elastalert make build v=v0.2.0b2

the only thing which isn't working is the elastalert-test-rule command. I'm not sure if this is an issue with the docker-container or the version. I used the following commands to test my rule which works under 0.1.39 but not using 0.2.0b2.

sudo docker exec -it elastalert sh cd .. cd elastalert elastalert-test-rule rules/my-super-rule.yaml

This respond in the following stackstrace:

Traceback (most recent call last): File "/usr/bin/elastalert-test-rule", line 11, in <module> load_entry_point('elastalert==0.2.0b2', 'console_scripts', 'elastalert-test-rule')() File "/usr/lib/python2.7/site-packages/elastalert-0.2.0b2-py2.7.egg/elastalert/test_rule.py", line 441, in main test_instance.run_rule_test() File "/usr/lib/python2.7/site-packages/elastalert-0.2.0b2-py2.7.egg/elastalert/test_rule.py", line 408, in run_rule_test conf = load_conf(args, defaults, overwrites) File "/usr/lib/python2.7/site-packages/elastalert-0.2.0b2-py2.7.egg/elastalert/config.py", line 44, in load_conf conf = yaml_loader(filename) File "/usr/lib/python2.7/site-packages/PyStaticConfiguration-0.10.4-py2.7.egg/staticconf/loader.py", line 167, in yaml_loader with open(filename) as fh: TypeError: coercing to Unicode: need string or buffer, NoneType found
Elasticsearch-Version: 7.0.0

[matsgoran]

@jbecker94 from what i can figure the test-rule command expects a —config parameter pointing to the elastalert configuration, there is no default for this option as far as I can tell in elastalert. Therefore my first reaction would be that this option needs to be provided by bitsensor/elastalert. FYI I may be wrong in this assesment since I'm away from the computer atm

[jbecker94]

@matsgoran for my understanding if it runs under 0.1.39 it should also be running under 0.2.0b2. The config should be available since the exact same folders are mounted into both of the containers. I have used the exact same Dockerbuildfile as the original bitsensor-docker-image, so there should be no difference except the different elastalert-version.

Hopefully in a couple of days there will be an offical bitsensor-version which fixes this issue. Most likely I have done something wrong in the build process or can anybody confirm that the test-rule command doesn't work?

[matsgoran]

@jbecker94 Let's hope so:)The changes related to the ES7 compatibility in the beta did not touch any part of the test-rule bootstrap process

[Qmando]

[_source_include] -> did you mean any of [_source_includes..

Man, this is the most annoying kind of breaking change. Just grepping through the code I can see that these don't always appear together in a search/deprecated_search pairs. I'll go through the code and double check this gets used correctly. More context from THAT error would be useful.

Elastalert-test-rule is kind of a mess, the error handling there is not great, and there's all sorts of special snowflake config being done. The TypeError: coercing to Unicode: need string or buffer, NoneType found does seem clear cut like the config parameter was missing. Again, terrible error handling, sorry about that.

[RiteshKuchukulla]

I tried with 0.2.0b version of ElastAlert (I'm using Elastic stack 7.0 version)
I'm still getting this error -
"message": "Error running query: TransportError(400, u\u0027illegal_argument_exception\u0027, u\u0027request [/auditbeat-/_search] contains unrecognized parameter: [_source_include] -> did you mean any of [_source_includes, _source_excludes]?\u0027)",
"traceback": [
"Traceback (most recent call last):",
" File \"/etc/elastalert/elastalert/elastalert.py\", line 390, in get_hits",
" *extra_args",
" File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py\", line 76, in _wrapped",
" v = kwargs.pop(p)",
" File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py\", line 655, in search",
" to the total number of copies for the shard (number of replicas + 1)",
" File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py\", line 314, in perform_request",
" # add a delay before attempting the next retry",
" File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py\", line 90, in perform_request",
" self.log_request_fail(method, url, response.request.path_url, body, duration, response.status_code, raw_data)",
" File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py\", line 125, in _raise_error",
" error_message = additional_info.get(\u0027error\u0027, error_message)",
"RequestError: TransportError(400, u\u0027illegal_argument_exception\u0027, u\u0027request [/auditbeat-*/_search] contains unrecognized parameter: [_source_include] -> did you mean any of [_source_includes, _source_excludes]?\u0027)"

[matsgoran]

@RiteshKuchukulla Are you running 0.2.0b1 or 0.2.0b2? ES7 support was added in 0.2.0b2

[RiteshKuchukulla]

@matsgoran
I was running 0.2.0b1, which gave the above error.
Now I started to run 0.2.0b2, which doesn't give me this error, thank you

[fpompermaier]

When I try to build the beta version I get this error on Centos 7:

error: urllib3 1.25.3 is installed but urllib3<1.25,>=1.20 is required by set(['botocore'])

[moix]

hi here, I read different comments about beta version working or not in 7.x, Can anyone confirm if 0.2.0b2 works? I needed to upgrade to 7.1 and since then we dont have alerting so would like to give it a try soon.
I'll start with 0.2.0b2 but if you are using it already in 7.x please can confirm?
Thanks! great work

[shahid-dgs]

I Just installed latest version of ELK 7.1 but ElastAlert is not working i am getting. any one got any fixed for ES7.1

[Qmando]

@fpompermaier
Did you try updating botocore?

[Qmando]

@shahid-dgs
You need to share more info if you want help

[fpompermaier]

@Qmando what should I do? I just followed the build instructions

[bangejsans]

Anyone bold enough to see if this works with 7.2 yet?

[kago-dk]

@bangejsans No new issues seen using v0.2.0b2 and running for 12 hours.

[zozo6015]

Hello,

I have installed elastalert beta from pip, when I am trying to create index I am getting the following error message:

elastalert-create-index --help Traceback (most recent call last): File "/usr/local/bin/elastalert-create-index", line 7, in <module> from elastalert.create_index import main File "/usr/local/lib/python3.5/dist-packages/elastalert/create_index.py", line 13, in <module> from auth import Auth ImportError: cannot import name 'Auth'

Note I am running elasticsearch 7.2.0 with security enabled.

[gerbdla]

Getting the same errors as above. Is there a resolution yet to the source_includes error?

[Qmando]

Run python 2, use v0.2.0b2

[zozo6015]

Interesting. python2 is end of life in less than 6 months and we still have it as a dependency

[rickywu]

velocidi/elastalert this docker image works well for es 7.2

[gdoctor]

Has anyone run this on elasticsearch 7.10.x or 7.11.x?

[nsano-rururu]

@gdoctor

Has anyone run this on elasticsearch 7.10.x or 7.11.x?

It should work.

[NOULeENGINEER]

@nsano-rururu
it seems that the elastalert version, that've pulled is not compatible with my elasticsearch version 7.9.1. when i run "npm run start", I got this error.

[nsano-rururu]

@NOULeENGINEER

It's not Esper, so that kind of listening is frustrating.

elastalert-server?
docker?
manual install?
What is the OS?
What command was executed and what was installed before execution?

[nsano-rururu]

The latest docker image should be python3.8 environment?
https://hub.docker.com/r/praecoapp/elastalert-server

[nsano-rururu]

@NOULeENGINEER
This repository is not a good place to ask ElastAlertServer questions.

[nsano-rururu]

@NOULeENGINEER

It works with the latest 7.11.1, but what?

[NOULeENGINEER]

I'm going to try your latest version on the docker hub

@NOULeENGINEER

It's not Esper, so that kind of listening is frustrating.

elastalert-server?
docker?
manual install?
What is the OS?
What command was executed and what was installed before execution?

the version of the elasticsearch is 7.9.1
the OS is ubuntu 18.04.5
the installation is manual, i followed steps that i've found in a medium blog : Deploying a SOCaaS infrastructure
the latest command that i've ran is npm run start

[nsano-rururu]

@NOULeENGINEER

See the official website for the manual installation procedure. I wrote. Because I'm the maintainer.
https://github.com/johnsusek/praeco#manualdev-installation