Kibana: Query bar input lag

[hanzo]

Kibana version: 5.5.0

Elasticsearch version: 5.5.0

Browser version: Chrome 61.0.3163.91 (Official Build) (64-bit)

Browser OS version: OS X 10.11.6

Description of the problem including expected versus actual behavior: After upgrading from Kibana 5.4.1 to 5.5.0, many of our users are seeing noticeable input lag when typing in the query bar on all Kibana pages.

Steps to reproduce:

1. Select the query bar on any Kibana page and start inputting text
2. Wait a second for the text to actually appear in the query bar

[hanzo]

Has anyone else on Kibana 5.5 noticed this? The lag is particularly noticeable on Chrome, slightly less noticeable on Safari. It makes it pretty frustrating to type out a long query when there's a delay for every character to appear.

[varunsharma27]

I'm not facing such issues on 5.4.1, 5.4.3 and 5.6.0. Can you test it out on 5.6.0 to rule out any environment/deployment issue? Maybe test on a clean installation of 5.5.0.

[Bargs]

I'm also not experiencing an issue in master

[trevan]

I noticed this with Firefox on a 6.0.0-apha3 snapshot.

[Bargs]

Could someone experiencing this capture a performance recording while typing into the query bar and upload it? https://developers.google.com/web/tools/chrome-devtools/evaluate-performance/reference#record

[hanzo]

@Bargs Here's a performance recording of me typing random text into the query bar. It takes a few seconds between when I finish typing and when the text appears in the query bar. Let me know if there's more info I can provide.

kibana_profile-20170925T135702.gz

[hanzo]

I just noticed that using the chrome web inspector to remove the input event handler on the query bar fixes the lag issue.

[Bargs]

Weird, it looks like multiple digest cycles are executing for every keypress. I booted up 5.5.0 to compare and my own profile looks normal, with a single digest per keypress and only about 6ms to run the input handler. Are you running any extensions in Chrome that might be interfering? Perhaps try testing in incognito mode with extensions disabled. Do you have any plugins installed in Kibana?

[hanzo]

For Kibana plugins, just Timelion. I tried in incognito mode with all Chrome extensions disabled, same issue.

Something to note is that we have a really large field count, around 16k. Most likely related?

[trevan]

I also have a really large field count as well.

[Bargs]

Something to note is that we have a really large field count, around 16k. Most likely related?

Shouldn't be related, we don't do any sort of lookups as you type. Aside from the history it's a pretty basic input.

@trevan you mentioned you could reproduce this in one of the alphas, do you still see the same thing in 6.0 beta2?

If either of you are up for testing a PR, could you try https://github.com/elastic/kibana/pull/14135? It's meant to fix another issue, but since it gets rid of some explicit calls to $digest in the typeahead code I think it's worth checking to see if it has an impact.

[trevan]

@Bargs, I haven't yet updated to 6.0 betas.

[Bargs]

Another report of this issue here: https://github.com/elastic/kibana/issues/14940

I'm going to see if I can reproduce this in the next couple days.

[Bargs]

Ok, so I reproduced the issue by indexing some documents with 15k fields. However, it's not just the query bar that lags. Pretty much every interaction is slow on any page that displays the field list. E.g. expanding documents or the quick count in Discover, or selecting a field in an agg config in Visualize.

@hanzo @trevan @jhejl2 is that consistent with your experience?

[trevan]

@Bargs, yes a lot of stuff is slow with lots of fields though I haven't seen the quick count be slow.

This issue in the query bar, though, is a new thing. It didn't exist a few versions ago.

I created #14974 to help with the visualization field list. There is still a weird slow down when you click on the input but it makes it easier to type and find the field.

[Bargs]

This issue in the query bar, though, is a new thing. It didn't exist a few versions ago.

Good to know. Based on two of these reports, it looks like something may have changed between 5.4 and 5.5. I'll see if I can track it down.

[jhejl2]

I can confirm that it's happening for the whole interaction.

[Bargs]

I spent some more time investigating this today and identified two areas of code that are mostly responsible for the slow down.

1. Angular's built in input directives trigger a full digest on every user interaction. I tried adding a debounce for model updates with ng-model-options which sped things up considerably. However, the debounce introduced the potential for the view and model values to get out of sync, which happened any time I typed quickly and hit enter. According to the ng-model-option docs the debounced function should execute immediately when an enclosing form is submitted, but this doesn't seem to be working for me. Maybe it's broken in our version of Angular. Unfortunately this ruins ng-model-options as a potential workaround.

2. The typeahead directive triggers a digest on every keypress. This isn't quite as bad as # 1 since we're using evalAsync instead of scope.apply. We can debounce this as well. When I did that in addition to the ng-model-options debounce pretty much all of the lag went away. Without the ng-model-options fix, this doesn't help much though.

So to sum things up, I've identified the sources of our digests but I haven't found an easy way to avoid them. Angular makes it difficult to avoid triggering digests on the root scope. A more forward looking solution might be to rewrite the query bar directive in React, which is what I think I'll investigate next.

[trevan]

@Bargs, both of those don't seem to be new. Could it be that something was added to the digest cycle to loop over every single field?

[Bargs]

@trevan you're right, neither of those were introduced between 5.4 and 5.5. I scanned the list of PRs between those releases and nothing popped out at me.

Here's how my thinking goes though: this slowness isn't just a problem in Discover. It's an issue on every page that displays the field list somewhere. So going through and trying to optimize every component in every app is going to be a much bigger effort than simply rewriting the query bar directive so it doesn't trigger digests. This solution would also fix the problem for good, since the query bar would be unaffected by any change that inadvertently makes the digest loop slow again.

[trevan]

@Bargs, good point

[mariowood]

We've recently upgraded to kibana and elasticsearch 6.3.0 and one user is seeing this issue. We have around 2000 fields in our index pattern, however as far as i know no other users are seeing this.

This has been seen on Chrome and Firefox, the user has a dell xps 15 with windows 10.

If input events are disabled in developer tools then the issue is not seen

[Bargs]

Similar issue reported in https://github.com/elastic/kibana/issues/21322. The problem may have gotten worse between 6.2 and 6.3. I'm going to start working on moving the query bar to React soon which should resolve this problem.

[r-tock]

Dumb question. Have you looked into moving the heavy work with a debounce and on requestIdleCallback instead of doing in synchronously inline. From reading the code, it doesn't look like you have to rewrite the whole thing into react in order to do that.

2c, since we use react, it is not a magic bullet, autocomplete needs to never block typing events.

[r-tock]

There are two issues happening
1) The typeahead implementation is blocking, for which it needs to be rewritten as a debounced.
2) The angular digest loop is running for the entire screen every time you type.

Just avoiding that I have been able to get back to 60fps buttery smooth typing.
If anyone wants to speed up typing, add the following to the queryBar input. If you have the binary version of kibana, just go into common.bundle.js, find the queryBar html inside the js and add the escaped version. And temporarily remove ng-keydown and ng-keypress from the typeahead directive.

<input
...
ng-model-options="{
    'updateOn': 'default blur',
    'debounce': {
      'default': 250,
      'blur': 0
    }
  }" />

[Bargs]

Thanks for sharing @r-tock. The changes you propose do definitely speed things up. There are a couple caveats I see that are worth mentioning though:

• A debounce in ng-model-options can lead to stale queries being submitted. I just fixed a bug related to this here.
• Removing the ng-keydown and ng-keypress from the typeahead directive will disable typeahead completely.

For folks who are desperate for a solution and who can live with these tradeoffs, it's worth a try as long as they don't mind modifying Kibana's source code.

I believe the main issue here is the digest cycle and I don't see a quick and easy way to avoid it without loss of other functionality. Re-writing the query bar and typeahead in React should allow us to avoid triggering digest cycles on input change, so I'm optimistic that it will fix this problem. In any case I need to rewrite these components for other unrelated reasons, so it can't hurt :)

If you've got other ideas for how to fix this quickly and easily though, I'm all ears. I'd love to be proven wrong :)

[r-tock]

Yeah, this is not a full solution, I barely spent an hour digging through the codebase, but a direction that you could go down to get big improvements.

If you rewrite into React it has the same rendering cycle issue, you just then try to tune the program to keep updates local. You are just trading one problem for another problem. then you spend more time tweaking the react render loop. (Speaking from experience, try out the type as you go search here https://www.exploretock.com/ powered by elasticsearch suggest btw)

The trick here is to make sure the inputs across kibana, do not trigger full digest cycle until you debounce or press enter. https://stackoverflow.com/questions/38381808/how-to-stop-digest-cycle-manually-in-angularjs

If you do this then you can bring back ng-keypress and ng-keydown.

[Bargs]

ng-keypress and ng-keydown trigger a full digest by design, they essentially wrap the callback in a $scope.$apply. We would need to eliminate our use of those directives to eliminate digests on input change. I imagine we could do that with some work. But like I said, I'm already rewriting this code in React for other reasons. So anything beyond a quick fix to the angular version is going to be wasted effort.

[jcmcken]

Any chance this will get some love soon?

[Bargs]

@jcmcken I'm working on a react implementation of the query bar as we speak

[chrisbenti]

It's a bit of a nuclear option, but you can run this as a bookmarklet:

var old_element = document.getElementsByClassName('kuiLocalSearchInput')[0];
var new_element = old_element.cloneNode(true);
old_element.parentNode.replaceChild(new_element, old_element);

Definitely a hack and removes a lot of the functionality but it makes the input work quickly

[Bargs]

Just an update, I've got a PR up for the React implementation of the query bar: https://github.com/elastic/kibana/pull/23704

Tested it with the data set I used to reproduce this input lag previously and the new implementation resolves the issue.

[graywolf-at-work]

Nuclear option (bookmarklets) updated for kibana 6:

1. Reload kibana with debug info

javascript:angular.reloadWithDebugInfo();

1. Make it fast (== useable)

javascript:let x = document.getElementsByClassName('kuiLocalSearchInput')[0]; let y = x.cloneNode(true); x.parentNode.replaceChild(y, x); y.addEventListener("keyup", function(e) { e.stopPropagation(); }); y.addEventListener("keydown", function(e) { if (e.keyCode != 13) { e.stopPropagation(); } else { angular.element('query-bar').scope().state.query.query = document.getElementsByClassName('kuiLocalSearchInput')[0].value; } }); y.addEventListener("keypress", function(e) { e.stopPropagation(); });