Kibana: vendors.bundle.dll.js is not cached and often redownloaded several times during a kibana session

[jordansissel]

Kibana version:

7.2.0

Elasticsearch version:

Server OS version: Cloud ESS

Describe the bug:

The "Loading Kibana" throbber appears and stays for a long time during visits to Kibana. I tracked some of this down to a single file, vendors.bundle.dll.js, taking a very long time to download.

On Cloud ESS, this file is 4.67MB and takes 6-10 seconds to download. It does not appear to ever serve from cache.

Further, when navigating between Kibana apps (panels? The left-hand navigation), the "Loading Kibana" throbber will appear again sometimes. Network panel in Firefox shows it again is downloading this several-megabyte file.

Steps to reproduce:

1. Visit kibana. Repeat visits still take 10+ seconds to load the app.

2. Visit Kibana. Wait until it loads.

3. Click another area of Kibana such as Timelion, Management, Monitoring, etc.

Expected behavior:

• Kibana should be fast to load.
• Navigating between Kibana panels/apps should not cause a long reload of resources

Screenshots (if relevant):

Any additional context:

This has been going on for some time (since Kibana 6.x), but it's fresh in my mind today as I am fiddling with security roles which requires (or seems to?) me login/logout of Kibana to test new user roles (via SAML).

[elasticmachine]

Pinging @elastic/kibana-operations

[jSchwarzweller-ct]

Hi,

the same on 7.3 ... The file is 21 MB of size and 4,67 MB are transferred.
The same applies to canvas.bundle.js ... Size is 11 MB and 7.9 MB are transferred... (in case you use canvas dashboards).

This slows down the page loading extremely. If you have some advanced Canvas dashboard with picture the loading times are inacceptable.

Any workaround of other help available on this?

Thanks a lot

[rbwolf]

+1 for this. Consistently seeing latencies greater than 1 min.

[dennyx]

any solutions to help on this?

[tylersmalley]

Hey @jordansissel, I am looking into this and unable to reproduce locally or on Cloud. With subsequent refreshes, I am seeing a 304 for vendors.bundle.dll.js. Is there anything different with your setup which could be causing this? Can you try opening an incognito window to disable plugins (long shot)?

[tylersmalley]

Also, do you have a self-signed certificate or any certificate errors? I wonder if you're running into https://bugs.chromium.org/p/chromium/issues/detail?id=110649#c8

[jpittiglio]

I encountered the same problem in a particular environment, but like @tylersmalley was unable to replicate this locally or in the cloud.

Environment where this occurred was deployed on AWS, with Kibana behind an Application ELB (this same setup was replicated and tested in a personal VPC - Application ELB did not have any impact on caching). X-Pack is being used. Using Firefox on Windows 7 x64 with 4GB RAM, the vendors.bundle.dll.js file was always coming back with a 200 response. I noticed in the request headers the browser was not sending If-None-Match for this file, but it did for the others.

Luckily, I had access to another system running Windows 10 x64 with 8GB RAM. Weirdly, using this system presented no issues and produced the expected 304 response code.

Checking the about:config on both hasn't shown me anything that jumped out as significantly different between the two. On a hunch, I checked the about:cache and noticed on the Win7 box, my disk cache was near the maximum. I cleared the cache, and after navigating to Kibana and letting everything load as expected the first time, subsequent hits presented the expected 304 response code!

Unfortunately, this wasn't long-lived - after bouncing around within Kibana a few times, those 304s became 200s again. Clearing the cache would only resolve the issue temporarily.

Not sure this necessarily helps, but hoping additional details from others similarly impacted might identify a common correlation.

[jSchwarzweller-ct]

We also have the system running in a cloud env - Azure. Runs on Kubernetes with a Loadbalancer in front.
There are no issues with the certificate @tylersmalley ... Would also be unlogic, as then everything would be uncached...
@jpittiglio Clearing cache does not solve anything for me. Also disabling plugins in a private window... Always 200 and more then 4 MB of data transferred.
So your guess is that it is somehow related to some cloud LB?

[jpittiglio]

@rull0310 I don't believe it's LB related - as noted, the issue only occurred in Firefox on a Windows 7 client. I checked again, and IE actually had no issues (surprisingly) on that client. Using Firefox on Windows 10 did not have any issues. Unfortunately, I don't have much access into the desktop environment to dig too deeply - just wanted to throw out some initial observations in hopes there is some correlation.

Also, in re-reading the original post, I should note the environment uses SAML as the primary authentication mechanism, with fallback to basic.

[jSchwarzweller-ct]

We just saw this issue happening only in FF private mode. Normal mode it works as expected!
Maybe this is the easy explanation for this issue...

[Jefflu57]

+1 for this .

kibana 7.2
Chrome Ver 74.0.3729.131 (64bit)

Any solutions to resolve this problem?
Thanks a lot!

[danielaugustyn23]

We are on Kibana 7.3.1 and we're seeing the same issue. I've tested it on a Mac/Chrome. We've seen Kibana loading apps for ~40seconds or so.

[jbudz]

I'm taking a closer look trying to find common denominators. I don't have clear reproduction steps yet but I'll start a list

• win 7 + ff
• chrome 74
• full browser cache

• self signed certificate

• vendor.bundle.dll.js needs to be smaller. even with caching, my browser is taking half a second to parse on a fast machine

[jordansissel]

I can reproduce this on cloud. Here's a video of me using Cloud and clicking a few different areas in Kibana. Each time, the full vendors bundle is downloaded, it seems, and it takes quite some time.

https://www.youtube.com/watch?v=nziXRYOBJNI#t-8s

[jordansissel]

I tried to reproduce it a second time, but the 2nd time I did not use a private window. In this case, it loaded from cache (HTTP 304 on response), but it still takes significant time to process/execute. We may consider cached results vs long computation as separate issues, but for completeness:

1) I have a video above reproducing the redownload of kibana's vendors bundle.
2) It takes 4-6 seconds to switch apps in Kibana even _with_ caching. I think we can open a separate issue for this one though.

[jbudz]

Thanks - I can reproduce that in private browsing (but not in normal). Only that one file too.

[Luke7810]

Can your team split this BIG JS file to multi JS file, I also have a problem with this JS file, since this file is too big, our server truncating the response.

[agentmilindu]

Facing the same issue on Chrome 77 on Linux ( Ubuntu 18.04 ) with a self-hosted instance

[agentmilindu]

I figured out it was something to do with my VPN, I used a CDN to host the static files. I'm my case, I changed the code a bit to get the vendors.bundle.dll.js served via a CDN ( I hope a CDN like Cloudflare which cache all the asset file for traffic goes through it would also do, this was not my case, I put the file in an S3 bucket and served via CloudFront ).

@elastic/kibana-operations Do you have any plans for supporting serving static files via a CDN in the future?

[hailin84]

Hey, guys. Is it possible to fix it? It's really boring. Why doesn't cache those large files?

[agentmilindu]

DLL Plugin is usually used for development, to make the subsequent build faster while deloping, right? I'm not sure if using it for production is recommended. How about removing the DLL PLugin for prod builds?

[dails08]

In addition to not caching, on slow connections kibana fails to load at all. It starts downloading the dll.js file, gets to about ten seconds of loading or so, then throws an error about chunking size. This problem means even if it would cache it, it'll still never work because it can't complete loading the file the initial time.

I'm also running on cloud, GCP, on kubernetes behind a LB. In my case, I'm running it in a pod with a gatekeeper sidecar as a proxy. I'd guess that gatekeeper (keycloak) has something to do with it, but it passes the oauth handshake and starts downloading the file, it just doesn't complete it quickly enough. I don't know where the decision to kill the download happens, though: client side, kibana, or gatekeeper.

[btrump]

+1 for this .

Kibana 7.2
Client: Chrome Version 52.0.2743.116 m (64-bit) on Windows Server 2012, 24gb RAM

[icloudphil]

+1 following this thread.

initial load until see the login page, take about 12-20 sec, and from login to data has come back is around 16-20 sec, vendors.bundle.dll.js take the longest time to load as suggest above not cached.

Tested on
Chrome Version 77.0.3865.75 (Official Build) (64-bit) on Mac
Kibana 7.4

[Sagesh]

+1
We are facing similar issue in 7.4.0 version.

[nielarshi]

Did the loading time improve for anyone here? Which version of Kibana you are on?

[tylersmalley]

The caching issue brought up in this issue should be resolved in 7.6.0.

[ghost]

@tylersmalley I have tried 7.6.0, not solved, it split into 4 js files, vendors_0.bundle.dll.js , vendors_1.bundle.dll.js , vendors_2.bundle.dll.js, vendors_3.bundle.dll.js, but still too large.

[nielarshi]

@tylersmalley I tried 7.6.0 and checked the network tab for the download time of the chunks. I am not sure about others but I noticed the opposite. It took more time for downloading all parts than the single one as the parts aren't downloaded in parallel rather they are downloaded sequentially. Also, they aren't cached, so, every refresh takes the same time.

[mistic]

@nielarshi could you share the network tab info from 2 consecutive of same page on Kibana?

[pashashocky]

@mistic i can confirm that upgrading to 7.6.0 does not solve this issue. For me it takes roughly 3s to download each of thr 4 parts of the vendors bundle dll.

Refreshing the same page just downloads them all over again.

The other two js files that are taking a long time are the:

• kbn-ui-shared-deps (2.05s)
• commons.bundle (2.43s)

[mistic]

@pashashocky I need more info to go further with this. Currently, under my tests, I'm experience cache on those files both on windows and mac with the last Chrome browser.

I would like to know the conditions you are test this against: OS, Browser and its version, testing with dev tools opened?

Also I would like to have 2 screenshots, from the dev tools with the network request info for the dll bundles, for 2 consecutive loads of the same page.

[nielarshi]

@mistic
Below are the network tab screenshot for

7.1.1

7.6.0

The images show time near to a second or in milliseconds but I have seen it taking relatively way more time. Anyway, the time taken for both versions vary and doesn't show any improvement as such.

I am using Windows 10 Enterprise Version 1803 with Chrome Version 80.0.3987.106 (Official Build) (64-bit).

[tylersmalley]

Can you confirm that Disable cache is not checked?

[nielarshi]

Yes. It isn't checked.

[mistic]

@nielarshi I would like to have 2 consecutive screenshots of the network tab for the same Kibana version (7.6.0) and also if you can a third one with the request info for one of the vendors request. Thanks for the help

[tylersmalley]

Actually @nielarshi, instead of screenshots it would be better if you could record a HAR file of the entire page request and post a link to it here.

Also, is this in an incognito window? How much free memory do you have on your machine? Attempting to understand the circumstances that result in this not being cached.

[nielarshi]

Here you go:

https://drive.google.com/drive/folders/1LyXB2Q2kdgoUip6ND8UzsLW3Q5WTq56q

Yes, it's in incognito but findings are same even when not in incognito. I have around 30GB free space on the machine.

[tylersmalley]

@nielarshi, thanks for providing those.

I find it quite odd that the only files which are cached are woff2 font files. There are no Javascript assets cached. The problem this issue originally raised is some assets would not be cached due to their size and being limited to in-memory caching (due to incogneto).

I assume there is a proxy in-front of Kibana for this installation. Is it possible that the request headers are being modified, or possibly stripped (like the etag). If you access this Kibana instance directly, and not through a proxy, are they cached?

[pashashocky]

@tylersmalley, in my case there is a proxy and there is no possibility for me to currently test without it - however i can see that the tag in the response headers is the same every time that I refresh the page.

[cameronkerrnz]

I'm experiencing the same issue with 7.6.1. I suspect that there is something missing in Node's (configuration of) how it processes 304 responses. I see the same with other products frequently, such as Django or Tomcat. It seems, from a casual strace, that it seems to be reading in the entire file just to validate the etag matches.

So the web-server is still doing much the same amount of work to say "no change", although the amount transferred is decreased (round-trips and TCP slow-starts though), and the execution engine (Node) has some of its threads tied up serving uninteresting requests and dealing with I/O.

Plus there are a lot more resources coming from the same domain, so the browser will limit however many resources it will ask for in parallel, which is browser-specific (typically 8?)

If you put a reverse proxy with a bit of caching related logic in front of it (eg. nginx, haproxy or varnish) you would see a marked improvement. I would probably go even further and replace the cache-control headers on the responses for things like vendored resources, changing it from...

cache-control: must-revalidate

to

cache-control: pubic, max-age=600

enabling the browser to cache it and not even have to revalidate (at least for a short period). However, as noted in #6520, this could interfere with upgrades and plugin installations, which is why I've gone for 10 minutes.

Best thing would be for Kibana to refer to its static assets using a hash so they become immutable assets, then you can cache them indefininately. This would be similar to what WhiteNoise does for Django. Then they could simply have a generous, performant, cache-control policy. That would be a more significant change within Kibana itself though.

[coudenysj]

Any news on this?

I can confirm that only the .woff2 files return a 304, all the rest is downloaded every time (200 response code).

[nielarshi]

The whole idea of caching is based on headers. Few are set by application in response to tell browser to allow caching and for how long and few by browser in requests to tell server if they want new one.

If it helps, you could try checking if your browser is able to trust the certificate you have for Kibana host. After some updates in Chrome, the browser has started rejecting browser caching if it can't trust a certificate. I am not sure of the case when its on http. My guess is that browser would still reject it.

In my case, I created a temp CA and imported in the browser to trust the certificate. After that browser was not interfering with the headers.
Hope this helps.

[benatbermejo]

+1 Confirmed. I have issues when Chrome does not trust the certificate.

[honestverkh]

I have the same issues. As I am using nginx as reverse proxy before kibana in AWS ElasticSearch, I could change it in the nginx.conf to cache assets for 1 day.
Note the tilde before application/javascript, I need it because kibana has additional text (context-encoding) in content-type

# Expires map
map $sent_http_content_type $expires {
    ~text/css                   1d;
    ~application/javascript     1d;
    ~font/woff2                 1d;
}
...
server {
    listen 80 default_server;
    server_name $host;
    rewrite ^/$ https://$host/_plugin/kibana redirect;

    location ^~ /_plugin/kibana {
         # patch kibana not setting cache-control on assets 
        expires $expires;
        ...

[senorkrabs]

Also experiencing this issue. Firefox 68.6.1esr and ES 7.4. I am going to set caching on my NGINX proxy for now.

I am using a valid cert as well.

[calvinbui]

same for us using Elastic Cloud on v7.6.2. All devs complaining it takes 20 seconds to load any of our deployments, big or small.