Kibana: Kibana 4 extremely slow displaying big text field

[rlueckl]

Hi,

this was already mentioned in https://discuss.elastic.co/t/kibana-4-x-very-slow-with-large-document/54249, but the issues I got as a reply didn't really cover the issue we're experiencing.

Basically if I have a document in Elasticsearch which has a large field (tested with a 4.5MB "lorem ipsum" text file) the rendering takes a lot of time. I can't reproduce the same issue with Kibana 3.

Kibana version: 4.1.5
Server OS version: Debian Wheezy (on a KVM)
Browser version: Firefox 47.0 (and Google Chrome 51.0)
Browser OS version: Linux Mint 17.3 (and Windows 7)
Original install method (e.g. download page, yum, from source, etc.): .deb file installed with "dpkg -i" from elastic.co/downloads

Description of the problem including expected versus actual behavior: rendering of the "Discover" pages takes very long if one (or more) of the documents in the results has a field with lots of text.

Steps to reproduce:

1. (optional) Create an empty index
2. Create a new document with 1 field, which has lots of text (4.5MB should be enough)
3. Open Kibana on the "Discover" page
4. Do a search for the document (either by "_id" or "*" if that's the only document in your index)
5. Open the found document (click on the small arrow left to the timestamp)

Errors in browser console (if relevant): "Warning: unresponsive script" pop-up from Firefox
Provide logs and/or server output (if relevant): -

For comparison: I created a brand-new empty index in Elasticsearch and added 1 document:

{
    "@timestamp": "2016-07-15T09:45:55.018+02:00",
    "@version": 1,
    "message": "Lorem ipsum dolor sit amet, ... [4.5MB of text] ..."
}

On setup "A": ES 2.3.4 and Kibana 4.5.1 -> rendering in browser takes about 20-25 Sec.
On setup "B": ES 1.7.5 and Kibana 3.1.3 -> rendering in browser takes about 1-2 Sec.

(You can generate the text for example by going to any "Lorem Ipsum" generator site, selecting a large number of paragraphs and the copy-pasting the same text in a file as many times as it takes to reach a few MB.)

[Bargs]

Any chance you could record a timeline in Chrome devtools? I'm curious where most of the time is being spent.

[Bargs]

Sorry, I just saw you already mentioned in the Discuss post that you can't grab a timeline because Chrome barfs. We'll have to do some debugging to see what's going on. Thanks for the clear steps on reproducing, that's very helpful.

[rlueckl]

I'll try to do more testing when I have some time. Maybe Chrome can handle a smaller document with a 2-3MB field to make a timeline (but still produce the "slowness").

[rlueckl]

Okay, so I took some more time to test.

2nd test: 2.4MB (same document as above, but only half the text)

On setup "A", Firefox: 10.45 Sec. (expected, half the data, half the rendering time)
On setup "B", Firefox: <1 Sec. (expected, half the data, half the rendering time)

Google Chrome 51.0 still doesn't render. After selecting the index and time range the "Searching ..." and the the "bubbles" are displayed indefinitely (waited for over 60 seconds). I also tried Chromium 51.0, same results as Google Chrome.

3rd test: 930KB

Similar results: Firefox gets faster (linear to the field size), Google Chrome still "hangs".

_UPDATE:_ a colleague tested with his ArchLinux+KDE+Chromium and Chromium did open the Discover page (with the 1x 930KB document) after 110 seconds! Opening the table (with the small arrow left of the timestamp) took only a few seconds after that.

We took a screenshot, but the timeline basically says that Chromium used 110 Sec. for "Scripting" (out of 120 Sec. total). The saved timeline is 60MB big and even compressed it's still 6MB. Is there a place where I could upload such a big file for you?

[Bargs]

I think you should be able to attach it to a github comment https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/

[rlueckl]

Thanks! (Sorry, I'm a Github n00b. :)) Obviously I had to blank out the hostname on the screenshot and changed it in the timeline to "testserver.loc.lan".

TimelineRawData-20160718T145753.json.gz

[rlueckl]

Hi guys,

do you have some news? Or do you need perhaps more information?

This is a blocker for us. We really would like to upgrade to ES 2.x, but as long as Kibana 4 has this bug we can't. :(

[Bargs]

Hi @rlueckl, sorry for the silence. This is still on my radar, but I haven't had time to look into it. Unfortunately I'm not sure how soon I'll be able to.

https://github.com/elastic/kibana/pull/7402 might help you once it gets merged, if you don't actually need to use the large fields in Discover. It adds a field filter functionality that allows you to avoid fetching large fields in Discover and on Dashboards.

[rlueckl]

Unfortunately the one large field is exactly what we need. It contains the log of long running requests which sometimes can have over a thousand statements.

[jean-lourenco]

I had the same problem. HTTP request/response headers/body can become really large, and storing all the data made Kibana unusable as every search took several minutes (even crashing the browser sometimes).

The solution was to trim down the data in the body of the request, just storing parts of it (the first and the last bytes). Of course, this was not an ideal solution, as some data is getting lost, but was the only way to get it working.

[rlueckl]

So, I had a little more time to play around and I tried ES5.0-alpha5 with Kibana 5.0-alpha5.

Same test as before, a brand new index with only 1 document:

{
  "name": "qwert",
  "number": "434",
  "something": "Lorem ipsum ... [360KB of text] ... quam enim."
}

Kibana 5 is almost as fast as Kibana 3, with no slowdowns. Works like a charm.
It even works with a 2,2MB document. Takes about 3 seconds to render.

Whatever you did to display (render) the results (table) in Kibana 5: could you backport it to Kibana 4? :)

[Bargs]

Ah, that's great to hear. Have you tried the exact same browser version in both Kibana 4 and 5 as well? I'm not aware of any major changes to the doc table between 4 and 5 so I'm not sure what would have made the difference on our end.

[rlueckl]

Unfortunately, I can't find a difference between FF 47 (tested previously) and FF 48.

I tried the 2.2MB document from above on Firefox 48.0 (Linux Mint 17.3)

• ES 1.7 and Kibana 3: 2-3 sec.
• ES 5 and Kibana 5: 2-3 sec.
• ES 2.3 and Kibana 4.5: ~10 sec. (same as before, see my comment from Jul. 18. -> "2nd test: 2.4MB ... On setup "A", Firefox: 10.45 Sec")

Google Chrome 53 still hangs while trying to load the results. Even with Kibana 5 (page unresponsive)! Kibana 3 works just like in Firefox and loads the document within 2-3 sec.

Conclusion: it has _not_ gotten better with the new Firefox version, but it's definitely faster when using Kibana 5.

[rlueckl]

Just a little update: I tested Kibana 4.6.1, but still no change. Opening the 2.2MB document takes 10 seconds, just like before with all other Kibana 4.x versions I've tried.

Updating to Firefox 49.0 didn't bring any changes either.

[rlueckl]

I wanted to ask if there was any progress regarding this issue?

[Bargs]

Sorry, there hasn't been. We've been pretty focused on 5.0. If this only affects 4.x it might not get prioritized very highly, but I'll update this ticket if we make any progress.

[asatsi]

Hi, we are also seeing the same issue. Its pretty annoying. We have huge XML payloads in the logs.
Configuration:
elasticsearch-2.2.1
kibana-4.4.2-linux-x64

[Bargs]

So I just tested this fix for a related issue in 5.x in 4.x and it made a huge difference. I created a single document with one giant field and the 4.7 branch crashed the browser without modifications. With the fix applied the page rendered in less than a second.

@epixa is this something we'd consider backporting to 4.x?

[epixa]

@Bargs I think it's worth considering, yeah.

[Bargs]

Fixed in https://github.com/elastic/kibana/pull/9150

[msporleder-work]

this is not fixed. Even a moderately large message pegs CPU and takes upwards of 90s to render at all. clicking "Table" and "JSON" really highlights the issue.

[Bargs]

@msporleder-work what version of Kibana are you using and how big of a message are we talking about?

[msporleder-work]

5.3.1, and about 2.5MB message(s). I should note that 2.5MB is a single line.

[Bargs]

Does each doc represent a single line? Or in other words, what's the total size of a single document in your index?

[msporleder-work]

I mean the "message" field from logstash. It is the bulk of the total document, which would probably be about 2.6MB if you added the rest of the fields. (hostnames, beats meta data, etc)

[Bargs]

Thanks for the extra info @msporleder-work, I was able to somewhat reproduce the issue. With a 2.3MB document Discover was quite sluggish on my machine, however it wasn't anywhere near 90 seconds. I only tested with a single large doc so far, so if you're referring to the initial page load that might be the difference.

Since this is an old ticket for 4.x I created a new one with more details: https://github.com/elastic/kibana/issues/11457

If you've got extra details, such as which browser version you're using, how many docs you're loading, or anything else that might account for the extra long (90s) load time you're seeing compared to what I saw, please add those details to the ticket!