Elastalert: ElasticSearch 7.7 - [bool] failed to parse field [must]

[kago-dk]

After I upgraded my ElasticSearch Stack to 7.7 is the Elastalert (Name: elastalert Version: 0.2.4) reporting an error in the log files.

WARNING:elasticsearch:GET https://server.abc.com:9200/metricbeat-*/_search?_source_includes=host.name%2C%40timestamp%2Csystem.filesystem.used.pct%2Csystem.filesystem.mount_point&ignore_unavailable=true&scroll=30s&size=10000 [status:400 request:0.000s]
ERROR:root:Error running query: RequestError(400, 'x_content_parse_exception', '[1:143] [bool] failed to parse field [must]')

It is a heads-up to anyone planning to upgrade to ElasticSearch 7.7.

Rule
name: "CRITICAL : Disk Usage Alert"
index: metricbeat-*
type: any
filter:

• query:
  
  
  
  • query_string:
    
    query: "_type:_doc"
  
  
• query:
  
  
  
  • query_string:
    
    query: "(system.filesystem.used.pct:[0.90 TO 1.0] AND system.filesystem.total: <2000000000001 AND NOT system.filesystem.type:cdrom) OR (system.filesystem.used.pct:[0.95 TO 1.0] AND system.filesystem.total: >2000000000000 AND NOT system.filesystem.type:cdrom)"
  
  

query_delay:
minutes: 1
query_key: ["host.name"]

[nickbabkin]

May it be because you're still using document type in the query which was deprecated?
https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

[bryantcj52]

I'm seeing the same error after upgrading to ES 7.7:
Error running query: RequestError(400, 'x_content_parse_exception', '[1:143] [bool] failed to parse field [must]')
Elastalert v0.2.1

name: "CRITICAL : Disk Usage Alert"
index: metricbeat-*

type: any

realert:
  hours: 24

filter: 
- query:         
  - query_string:                                      
      query: "system.filesystem.used.pct:[0.9 TO 1.0]"

query_delay:
  minutes: 1            
query_key: 
  - host.hostname
  - cloud.instance.id
  - system.filesystem.mount_point

data.query.query.bool.filter.bool.must | {   "range": {     "@timestamp": {       "gt": "2020-05-29T14:43:01.533802Z",       "lte": "2020-05-29T14:44:01.533802Z"     }   } }, {   "query_string": {     "query": "system.filesystem.used.pct:[0.9 TO 1.0]"   } }

All three of our alerts that use "type: any" are failing, the others are still working.

[bryantcj52]

Looking at the JSON of the data.query.query.bool.filter.bool.must field, it looks incorrect, note the array inside of the array which is not valid in ES query. Running the query in ES dev tools without the extra array works correctly. I'm guessing previous to 7.7, ES was either fixing it or ignoring it but I don't have an old version of ES to test with.

I fixed it by changing my rule's filters to not set query_string as an array which was making a double array and caused the sub array when inserting the timestamp.

A check to throw an error or automatically fix the extra array would be useful.

filter: 
- query:         
    query_string:                                      
      query: "system.filesystem.used.pct:[0.9 TO 1.0]"

instead of

filter: 
- query:         
  - query_string:                                      
      query: "system.filesystem.used.pct:[0.9 TO 1.0]"

{
  "_index": "elastalert_status_error",
  "_type": "elastalert_error",
  "_id": "blOglXIBzAfeg4t7oidA",
  "_score": 1,
  "_source": {
    "message": "Error running query: RequestError(400, 'x_content_parse_exception', '[1:143] [bool] failed to parse field [must]')",
    "traceback": [
      "Traceback (most recent call last):",
      "  File \"/opt/elastalert/elastalert/elastalert.py\", line 379, in get_hits",
      "    res = self.thread_data.current_es.search(",
      "  File \"/usr/lib/python3.8/site-packages/elasticsearch-7.0.0-py3.8.egg/elasticsearch/client/utils.py\", line 84, in _wrapped",
      "    return func(*args, params=params, **kwargs)",
      "  File \"/usr/lib/python3.8/site-packages/elasticsearch-7.0.0-py3.8.egg/elasticsearch/client/__init__.py\", line 810, in search",
      "    return self.transport.perform_request(",
      "  File \"/usr/lib/python3.8/site-packages/elasticsearch-7.0.0-py3.8.egg/elasticsearch/transport.py\", line 318, in perform_request",
      "    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)",
      "  File \"/usr/lib/python3.8/site-packages/elasticsearch-7.0.0-py3.8.egg/elasticsearch/connection/http_requests.py\", line 91, in perform_request",
      "    self._raise_error(response.status_code, raw_data)",
      "  File \"/usr/lib/python3.8/site-packages/elasticsearch-7.0.0-py3.8.egg/elasticsearch/connection/base.py\", line 131, in _raise_error",
      "    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)",
      "elasticsearch.exceptions.RequestError: RequestError(400, 'x_content_parse_exception', '[1:143] [bool] failed to parse field [must]')"
    ],
    "data": {
      "rule": "CRITICAL : Disk Usage Alert",
      "query": {
        "query": {
          "bool": {
            "filter": {
              "bool": {
                "must": [
                  {
                    "range": {
                      "@timestamp": {
                        "gt": "2020-06-08T20:28:31.984127Z",
                        "lte": "2020-06-08T20:29:31.984127Z"
                      }
                    }
                  },
                  [
                    {
                      "query_string": {
                        "query": "system.filesystem.used.pct:[0.9 TO 1.0]"
                      }
                    }
                  ]
                ]
              }
            }
          }
        },
        "sort": [
          {
            "@timestamp": {
              "order": "asc"
            }
          }
        ]
      }
    },
    "@timestamp": "2020-06-08T20:30:31.996510Z"
  },
  "fields": {
    "@timestamp": [
      "2020-06-08T20:30:31.996Z"
    ]
  }
}

[MakoWish]

I am experiencing the same issue since upgrading to ES 7.7 (currently on 7.8).

{
  "_index": "elastalert_status_error",
  "_type": "_doc",
  "_id": "Mwy_u3MByPwqxu_vWSIE",
  "_score": 1,
  "_source": {
    "message": "Error running query: RequestError(400, 'x_content_parse_exception', '[1:143] [bool] failed to parse field [must]')",
    "traceback": [
      "Traceback (most recent call last):",
      "  File \"/usr/local/lib/python3.6/dist-packages/elastalert-0.2.4-py3.6.egg/elastalert/elastalert.py\", line 385, in get_hits",
      "    **extra_args",
      "  File \"/usr/local/lib/python3.6/dist-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/client/utils.py\", line 84, in _wrapped",
      "    return func(*args, params=params, **kwargs)",
      "  File \"/usr/local/lib/python3.6/dist-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/client/__init__.py\", line 811, in search",
      "    \"GET\", _make_path(index, \"_search\"), params=params, body=body",
      "  File \"/usr/local/lib/python3.6/dist-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/transport.py\", line 318, in perform_request",
      "    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)",
      "  File \"/usr/local/lib/python3.6/dist-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/connection/http_requests.py\", line 91, in perform_request",
      "    self._raise_error(response.status_code, raw_data)",
      "  File \"/usr/local/lib/python3.6/dist-packages/elasticsearch-7.0.0-py3.6.egg/elasticsearch/connection/base.py\", line 131, in _raise_error",
      "    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)",
      "elasticsearch.exceptions.RequestError: RequestError(400, 'x_content_parse_exception', '[1:143] [bool] failed to parse field [must]')"
    ],
    "data": {
      "rule": "Critical Disk Usage",
      "query": {
        "query": {
          "bool": {
            "filter": {
              "bool": {
                "must": [
                  {
                    "range": {
                      "@timestamp": {
                        "gt": "2020-08-04T22:57:26.364778Z",
                        "lte": "2020-08-04T23:12:26.364778Z"
                      }
                    }
                  },
                  [
                    {
                      "query_string": {
                        "query": "((system.filesystem.used.pct:[0.90 TO 1.00] AND system.filesystem.total:[0 TO 107374182400]) OR (system.filesystem.used.pct:[0.95 TO 1.00] AND system.filesystem.total:[107374182401 TO 1073741824000]) OR (system.filesystem.used.pct:[0.98 TO 1.00] AND system.filesystem.total:[1073741824000 TO 8589934592000]) OR (system.filesystem.used.pct:[0.999 TO 1.00] AND system.filesystem.total:[8589934592001 TO *]) ) AND system.filesystem.type: fixed AND NOT system.filesystem.mount_point: /snap/* AND NOT system.filesystem.mount_point: /boot/efi"
                      }
                    }
                  ]
                ]
              }
            }
          }
        },
        "sort": [
          {
            "@timestamp": {
              "order": "asc"
            }
          }
        ]
      }
    },
    "@timestamp": "2020-08-04T23:12:26.367772Z"
  },
  "fields": {
    "@timestamp": [
      "2020-08-04T23:12:26.367Z"
    ]
  }
}

[eqxDev]

I am getting the same issue once upgrading to 7.7 any ideas how to fix this?
```{
"error":{
"root_cause":[
{
"type":"x_content_parse_exception",
"reason":"[1:106] [bool] failed to parse field [must]"
}
],
"type":"x_content_parse_exception",
"reason":"[1:106] [bool] failed to parse field [must]",
"caused_by":{
"type":"x_content_parse_exception",
"reason":"[1:106] [bool] failed to parse field [should]",
"caused_by":{
"type":"x_content_parse_exception",
"reason":"[1:106] [bool] failed to parse field [must]",
"caused_by":{
"type":"illegal_state_exception",
"reason":"expected value but got [START_ARRAY]"
}
}
}
},
"status":400
}

[bryantcj52]

@eqxDev @MakoWish

See my answer on how to fix your alert: https://github.com/Yelp/elastalert/issues/2807#issuecomment-640881868