Kibana: [Bug] /api/kibana/suggestions/values/{index} endpoint causes elasticsearch cpu spikes

Created on 27 Feb 2020 · 2Comments · Source: elastic/kibana

Kibana version: 7.6.0
Elasticsearch version: 7.6.0
Server OS version: Red Hat 7.7
Browser version: Chrome 78.0.3904.70
Browser OS version: Windows 7
Original install method: RPM from Yum repo

Ever since upgrading to 7.6.0 i've been noticing extreme cpu spikes on my cluster. They seem to happen whenever a new query is run in kibana, the same query can be run again and the spike doesn't occur again. My guess is because something is cached after the initial request.

It seems to be the /api/kibana/suggestions/values/{index} call that is the culprit.
The spike starts to occur as soon as the user starts typing in the search bar. Then each key press cancels the previous request and sends a new one, however it doesn't appear the queries are cancelled on the elasticsearch side and cpu climbs to 100%.

For instance, this following request causes the cpu to spike to around 40%

 curl --user cpmoore -H 'kbn-version: 7.6.0' -X POST -H 'Content-Type:application/json' 'https://kibana.elk.omf.cloud/api/kibana/suggestions/values/filebeat-prod-*' --data-binary '{"query":"datapower","field":"service.group","boolFilter":[]}'

The very last spike is this request, the others are my troubleshooting (trying to determine if this was the exact request causing the spikes). The eswk* hosts are the data nodes.

We have a relatively large cluster as you can tell from the below screenshot

I know there were some field filtering changes in 7.6.0, as @simianhacker has already rolled back one in a PR.
I was wondering if this could be caused by some changes to the suggestions endpoint.

Filters AppServices bug

Source