Elastalert: Too many scroll contexts

[arber-od]

ELK 7.1.0 and Elastalert 0.2.0b

elastalert_error - {'message': "Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')", 'traceback': ['Traceback (most recent call last):', ' File "elastalert/elastalert.py", line 352, in get_hits', ' *extra_args', ' File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 84, in _wrapped', ' return func(args, params=params, **kwargs)', ' File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 818, in search', ' "GET", _make_path(index, "_search"), params=params, body=body', ' File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 353, in perform_request', ' timeout=timeout,', ' File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 143, in perform_request', ' self._raise_error(response.status_code, raw_data)', ' File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 168, in _raise_error', ' status_code, error_message, additional_info', "TransportError: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')"], 'data': {'query': {'sort': [{'@timestamp': {'order': 'asc'}}], 'query': {'bool': {'filter': {'bool': {'must': [{'range': {'@timestamp': {'gt': '2019-05-27T00:29:55.949668Z', 'lte': '2019-05-27T00:44:55.949668Z'}}}, {'query_string': {'query': 'program: ssl AND message: "Limit goes to maximum"'}}]}}}}}, 'rule': 'SSL Limit goes to maximum'}}

Need some help...

[axeljov]

I'm hitting the same issue using ELK 7.1.0 and Elastalert 0.2.0b . Increasing search.max_open_scroll_context In Elasticsearch's config to numbers like 5000 and 10000 made little to no difference. I suspect this might also have to do with the way that sigmac creates the rules for Elastalert backend because some of the rules look "malformed" and I wonder if that causes it to generate crazy amount of scroll contexts. If I load Elastalert with only a few basic rules that i know are good then I do not see the issue pop up (at least for short periods of time I have tested).

Here is an issue related to Sigma parsing to Elastalert format. I've been trying to get my malware hunting stack up and running for the past few days, would love to have this bug found soon :)

[agix]

Maybe related to https://github.com/Yelp/elastalert/issues/2249

This should solve the problem : https://github.com/Yelp/elastalert/pull/2271

[arber-od]

After https://github.com/Yelp/elastalert/pull/2271

root@elastalert:/opt/elastalert/elastalert# git pull
Updating 09a3eb6..3b871b4
Fast-forward
README.md | 1 +
docs/source/ruletypes.rst | 13 +++++++++++++
elastalert/config.py | 2 +-
elastalert/create_index.py | 2 +-
elastalert/elastalert.py | 8 +++++---
elastalert/schema.yaml | 6 ++++++
elastalert/test_rule.py | 4 ++--
elastalert/zabbix.py | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
requirements.txt | 1 +
9 files changed, 105 insertions(+), 7 deletions(-)
create mode 100644 elastalert/zabbix.py

root@elastalert:/opt/elastalert/elastalert# pip install -r requirements.txt

But got it

root@elastalert:/opt/elastalert/elastalert# systemctl status elastalert.service
● elastalert.service - Elastalert
Loaded: loaded (/etc/systemd/system/elastalert.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-06-04 07:58:39 UTC; 5s ago
Main PID: 7275 (node)
Tasks: 10 (limit: 4915)
CGroup: /system.slice/elastalert.service
└─7275 /root/.nvm/versions/node/v9.9.0/bin/node lib/index.js

Jun 04 07:58:40 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":50,"msg":"ProcessController: Traceback (most recent call last):\n File \"/usr/lib/python2.7/runpy.p
Jun 04 07:58:40 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":50,"msg":"ProcessController: Index create exited with code 1","time":"2019-06-04T07:58:40.782Z","v":
Jun 04 07:58:40 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":40,"msg":"ProcessController: ElastAlert will start but might not be able to save its data!","time":"
Jun 04 07:58:40 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":30,"msg":"ProcessController: Starting elastalert with arguments [none]","time":"2019-06-04T07:58:40.
Jun 04 07:58:40 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":30,"msg":"ProcessController: Started Elastalert (PID: 7287)","time":"2019-06-04T07:58:40.786Z","v":0
Jun 04 07:58:40 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":30,"msg":"Server: Server listening on port 3030","time":"2019-06-04T07:58:40.788Z","v":0}
Jun 04 07:58:40 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":30,"msg":"Server: Server started","time":"2019-06-04T07:58:40.788Z","v":0}
Jun 04 07:58:41 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":50,"msg":"ProcessController: Traceback (most recent call last):\n File \"/usr/lib/python2.7/runpy.p
Jun 04 07:58:41 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":50,"msg":"ProcessController: exec code in run_globals\n File \"/opt/elastalert/elastalert/elast
Jun 04 07:58:41 elastalert node[7275]: {"name":"elastalert-server","hostname":"elastalert","pid":7275,"level":50,"msg":"ProcessController: ElastAlert exited with code 1","time":"2019-06-04T07:58:41.327Z","v":0}

Did I miss something?

[opoplawski]

I'm hitting this as well with elastic 7.3.0

[levanvunam]

Do we have any fix on this? This cause my cluster unstable after a while. And need to restart elastalert everytime :(

[gurn33sh]

Is there any fix for this?

[rlreamy]

I am also experiencing this issue....would love to know if there is a fix.