Magento2: Javascript Bundling produces huge 13MB js files

[VegasJames]

Steps to reproduce

1. Install Magento via composer. v2.0.5
2. Use admin to enable Javascript Bundling. (Stores -> Configuration -> Advanced -> Developer -> JavaScript Settings) and set Merge JavaScript Files and Enable Javascript Bundling to Yes
3. Left "Minify JavaScript Files" to No so the file could be inspected, but even with it as Yes the bundle0.min.js files is 8MB.
4. Had to run bin/magento setup:di:compile
5. Then did bin/magento setup:static-content:deploy
6. Observe the size of the generated bundle0.js file.
   

   Expected result

Before bundling Chrome showed about 515KB of javascript being loaded, bundling the files in to one should result in a file of a similar size, not 25 times as big.

Actual result
ls -lh pub/static/frontend/Magento/blank/en_US/js/bundle/
total 13M
-rw-rw-rw-. 1 magento apache 13M May 14 07:23 bundle0.js

I've investigated the content of the file somewhat. After a lot of valid javascript we get to this bit:
//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJz (truncated)
This goes on and on with random characters for 1,414,372 bytes.
The rest of the file then appears to be valid javascript, so unfortunately the above base64 only explains about 10% of the bloat, so I'm not sure where the rest comes from, but certainly this seems wrong.

Additionally, this is more of a feature request, but can the bundling leave out jquery and jquery-ui?
These are the two largest files accounting for 700KB of javascript and almost 200KB of data transferred, and they also could be fetched from Google CDN instead:
https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js
https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.4/jquery-ui.min.js

[pboisvert]

Chuck--can you get the team to investigate given the potential for performance impact if reproducible?

[rkhoury82]

I also suffered the same problem on the same version installed via composer.

[viktor-zhuromskyy]

Same thing here. :-1:

[viktor-zhuromskyy]

Not even that! On 2.0.5 after all static content is compiled without any errors, there is a fatal error on whole Magento installation dues to errors in DOM bundling. With the JS Bundling OFF everything works normal.

Please see screenshots.

[choukalos]

I'll check with the team re bundle size. I believe we implemented a very simple bundling algorithm so it's possible we're pulling in some additional files in the bundle.

Is anyone else seeing bundle.js errors on 2.0.5? @devdesco-ceo did you create a separate Github issue for the bundle error on 2.0.5?

[vkorotun]

@VegasJames It seems like your theme doesn't inherit from Blank or Luma Magento native themes, so exclude list for JS files bundling is missed. You need to configre the exclude list for your custom theme. As a reference example you can have a look into configuration for Blank or Luma theme:

• /app/design/frontend/Magento/blank/etc/view.xml
• /app/design/frontend/Magento/luma/etc/view.xml

See "exclude" node content.

[choukalos]

@VegasJames Please close or let us know if that didn't address your issue.
@devdesco-ceo What themes do you have installed?

[viktor-zhuromskyy]

I have Porto theme installed and my theme is a child theme of Porto which is based on Blank.
my /etc/view.xml file has all proper exclude statements for the JS files.

[VegasJames]

incorrect, this is a stock Magento install. the bundle file inside both the
blank and Luma theme directories is 13MB

On Wednesday, May 18, 2016, Chuck Choukalos [email protected]
wrote:

@VegasJames https://github.com/VegasJames Please close or let us know
if that didn't address your issue.
@devdesco-ceo https://github.com/devdesco-ceo What themes do you have
installed?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/magento/magento2/issues/4506#issuecomment-220033699

[TommyKolkman]

I can second VegasJames's problem on 2.0.7. Also, my theme inherits correctly.

[atIOCrON]

@vkorotun we have the same issue verified with Magento 2.0.6 using the default Luma theme:

Without JS Bundling (453.8kb): http://tools.pingdom.com/fpt/#!/czoIzm/https://new.betterbatt.com.au/
With JS Bundling (2.1mb): http://tools.pingdom.com/fpt/#!/c8CxvG/https://new.betterbatt.com.au/

As you can see JS Bundling increases page size by 4x.

I also noticed that JS Bundling when enabled blocks the selection of role resources when adding a custom admin role.

[erikhansen]

From what I've read here and from my independent testing, it seems like the "Enable Javascript Bundling" is currently unusable and should not be recommended for production environments. The way bundling has been implemented for M2 negates much of the benefits gained by RequireJS.

I would recommend the following:

• To prevent developers from enabling bundling without understanding its ramifications, in the next patch release, remove the "Enable Javascript Bundling" option or at least add a disclaimer beneath the option alerting the admin that the feature is not yet recommended for production.
• Refactor the JS bundling functionality so that rather than bundling all of the JS files into a massive ~8MB file, the bundler should only bundle JS files that are known to be required by all pages. The logic behind how the bundler should know what files to include is up for discussion, but the <exclude> logic in view.xml could possibly be inverted to use an <include> logic so that only files/directories in this node would be bundled.

[hostep]

And let me just add one small addition: if your webserver is capable of running HTTP/2 (which will become more and more the case) it is really not recommended to enable bundling. Since HTTP/2 hasn't got performance problems with transferring a lot of smaller files.

[southerncomputer]

Expected '}'
File: bef36da44feee702d670095d789f94f7.min.js, Line: 136, Column: 49365

Getting this error on IE and Firefox plus admin section no longer functioning after enabling bundling on 2.0.6 !

This seems about as reliable as the old magento 1 compiler ! Having to load +140 .js files due to this feature not working is severely bad!

[erikhansen]

@southerncomputer Your browser will cache the JS files, so while the first page will load more slowly, subsequent pages will use the cached JS files. And since the JS is being loaded asynchronously via RequireJS, the page will render before the JS files are loaded.

That being said, it does seem a bit much to have ~118 JS files loaded for a vanilla M2 homepage—thus why I'd like to see a smarter bundling feature.

[southerncomputer]

https://github.com/magento/magento2/issues/3040 this is a duplicate ?

[erikhansen]

@southerncomputer No, #3040 is not a duplicate issue of this issue. Although the two issues are somewhat related in that they deal with JS minification/bundling.

However the issue you described in your comment above might be caused by the issue reported in #3040.

[southerncomputer]

https://github.com/magento/magento2/blob/6ea7d2d85cded3fa0fbcf4e7aa0dcd4edbf568a6/lib/web/mage/backend/validation.js

this file throws an error in the bundle regardless if you disable jshrink or not and have bundling enabled. I will try to manually minify this file to see where the file is causing issues in bundling.

The result is in IE11/FF it will throw an unmatched } error on the developer tools console, thus rendering the big bundle unusable for the most part!

[Zaylril]

I have to agree with @erikhansen in that in it's current state the bundling feature is simply not usable. Just think here for a second for a mobile user on 3G, even with the 13meg file gzipped it comes out at a whopping 3.3meg. Add this to the rest of the page and you are looking at 5meg+ for an initial page load. Not great.

Surely a much better solution would be to only bundle files which would be used on every page - or have a config where you could add files you wanted to be bundled together - whether this be automatic or not - meaning you would get some benefit from bundling files for a common.js file and then load the rest as and when they are needed.

Additionally I'm also not sure why the requireJS optimizer is not being implemented - where it combines related scripts together to build multiple bundles which are included as and when they are needed vastly reducing the number of individual files loaded, and greatly increasing performance.

Of course there is a balance to be had between file size and the number of files loaded - but as already mentioned when http2 is implemented this becomes a non-issue, however I can't really see this happening for everyone and all devices in the near future.

I think the only solution is really to create a smarter bundling feature - similar to that of the requireJS optimizer or implementing a build step where smarter bundling occurs. In this way you would reduce the number of requests down for each page and also reduce the file size as well - as was the initial intention of loaded each file separately. This really needs fixing though - especially on the checkout where there is something like 250 JS files loaded - not to mention the .html templates being loaded via AJAX too. Not the best for performance. The worst thing about this also is you can't really implement any kind of server cache on all those JS files to speed it up - as it totally relies on the browser.

[erikhansen]

@Zaylril Thanks for adding to the discussion.

@alankent I'd be curious to hear your thoughts on this issue.

[southerncomputer]

2.1.0-rc2 now creates 5 bundle0.js .. bundle5.js - not that is saving much space but breaks up the files to concurrently load?

[erikhansen]

@southerncomputer Thanks for reporting, but this still doesn't fix the underlying concern that someone visiting a Magento 2 homepage shouldn't be served ~8MB (uncompressed) of JS, most of which is not needed on the homepage.

[southerncomputer]

Well this is odd, After deploying to production mode, I switched to developer mode wiped pub/static/* and now it won't load the bundle0.js pair any more after deploying to production again, it appears to be pulling from pub/static/_cache/merged and loading the signed *.min.cs and *.min.js files. I've tried wiping var/di var/generation var/view_preprocessed etc. Then deploying using php bin/magento deploy:mode:set production..

The size between the two modes is greatly different as well. Bundle*.js was about 8meg uncompressed and the signed *.min.css and *.min.js are about 2meg uncompressed?

the bundle0.js .. bundle3.js are located in pub/static/adminhtml/Magento/backend/en_US/js/bundle/ and pub/static/frontend/Magento/luma/en_US/bundle/

How odd is this? I've restarted httpd and php-fpm between the deploy commands to rule out pathcache!

[dthampy]

An internal JIRA ticket -MAGETWO-55587 has been created for the original issue (reported on May 14)

[charleskj]

This issue resolved in 2.1 ?

[southerncomputer]

Nope! Still 3.5meg bundled javascript!

[TommyKolkman]

With all respect, but this is not an improvement or feature, but a bug, if you ask me. Right?

[leoquijano]

I agree with @TommyKolkman: this function is unusable as it is right now, so it looks more as a bug and not as a feature.

[pboisvert]

this is being tracked internally as a bug and should not have the improvement label

[erikhansen]

@pboisvert Glad to hear it. I would love to be able to use the JS bundling feature, but unless it's refactored to be smarter (as I suggested in a previous comment), it's basically unusable.

While the coming adoption of HTTP/2 will make it more viable to run M2 without bundling (due to multiplexing), I expect that if smart bundling is implemented in conjunction with HTTP/2, it will result in better performance.

[leoquijano]

I think this issue report should be reopened, right?

[Khaleel]

The entire front end using RequireJS has so much bloat and overhead - can this be sorted?

[pboisvert]

reopening just to confirm this is being tracked as a bug. We have an issue for this as noted above that is open.

[Khaleel]

Thanks @pboisvert

[Zaylril]

@erikhansen Just as a side note we have a few sites on CloudFlare which essentially serves all the JS files via HTTP2. It makes little difference to the performance of the checkout (250+ JS files) and other areas of the site. Performance improvement imo needs to be made to the component/widget and template rendering, especially the checkout, which in it's current state is pretty awful speed wise.

[hostep]

@Zaylril: that's exactly what we are seeing unfortunately. HTTP/2 makes little difference because the js files are being requested by other js files which in their turn also request other js files, and so on and so on (if I remember correctly, it has been a few months ago since I last looked at this).
So the js files aren't all being requested simultaneously, which makes using HTTP/2 not really helpful in this case. If all the js files would be referenced from within the html, we would see a bigger jump performance wise I think (when using HTTP/2).
So for the time being, we have enabled js bundling and that seems to be the best we can do for now.

[Khaleel]

If all the js files would be referenced from within the html, we would see a bigger jump performance wise I think (when using HTTP/2).

This is not entirely true. Magento 1 still does this - they include all the prototype, language, and all JS files right at the top of the application and this casues it to be slow. It needs configuration out of the box and a CDN only helps half way.

It is so disappointing about the Front End Development architecture. Magento has outsourced all these European developers and you can see that they have lack of experience (when I say experience I am talking working with 10+years on the web within ecommerce companies) and that the other developers have not fully thought things thru.

The collaboration between coders is not working I think.

[viktor-zhuromskyy]

It has been fixed in 2.1.1
Now it produces close to 5MB, and when delivered to the client as gzipped content - the file size is 1MB! That's super nice!

[erikhansen]

@devdesco-ceo Thanks for sharing. I've confirmed that turning on bundling in 2.1.1 results in a ~4.3MB file (~900KB compressed).

As a point of reference, if you visit a 2.1.1 homepage with bundling turned _off_, the page will return 120+ JS files that are 1.9MB uncompressed. So while bundling will double the overall file size, it is only a single file.

I'd be curious to see tests comparing the difference between bundling enabled vs disabled. Using Chrome's Network Throttling feature would allow for comparing the performance difference at network speeds.

[viktor-zhuromskyy]

@erikhansen
You are absolutely right about the sizes.
In my case with bundling turned on website load size is roughly 2MB, but without bundlin being on, but merging and minification ON, the overall size is about 1.6MB. So that's 400 MB savings on every new visit.
More than that, JS bundling is still broken, since I noticed it is breaking ./prototype/. and ./jquery/. paths in static view folder. So, I disabled it for good, especially with HTTP/2 enabled Nginx 160 files load vs 65 files to load is not an issue, hence these are piped via single socket, so to say...

[viktor-zhuromskyy]

To add to the discussion,
with proper implementation of PHP OpCache, CDN, caching, minifications and merging, my page load is between 3.1 and 6 seconds that gives a green signal on GTMetrix, depending on a page complexity.

[pynej]

Still major preformance issues on /checkout/ the rest of the site is fine with Varnish, bundling, and combining fills all enabled.

For our site the /checkout/ still takes 3.5s to generate and download the Html and its 490kb, and the combined script file is 3.4MB and takes 6s to download. This is a hell of a lot better then disabling bundling(then we get >100 requests and the page still takes >6 seconds to load, though it only downloads 2.1MB instead of 4MB.) The savings in size is lost in multiple requests though so its not any faster.

[mikel5]

We've discovered that bundling isn't usable because it requires localStorage. And localStorage isn't available when in Private browsing mode on iOS devices. And there are no try/catch checks around the localStorage calls.

[Khaleel]

@mikel5 Thank you for that insight - I was wondering why we get issues here and there - will look into this.

What I do temporary fix for now:

• All files on a CDN (CSS, JS, Images)
• All files minified manually, including the major JS libraries in vendor

Bundling is a science and in 12 years of ecommerce I have never seen an app, be it Gulp, Grunt or even Magento bundle JS files together correctly. It is a complicated procedure and unfortunate it has not be cracked well. Even CloudFlare does not do it well. I never bundle JS files unless I do them manually.

[pynej]

Never? https://github.com/Shazwazza/ClientDependency is a .Net package that does this and more with ease. Minification, compilation, caching, versioning. It's really not rocket science.

Jeremy Pyne

"Fear is the mind killer."

On Nov 14, 2016, at 11:48 PM, Khaleel ibn Anwar Mughal [email protected] wrote:

@mikel5 Thank you for that insight - I was wondering why we get issues here and there - will look into this.

What I do temporary fix for now:

All files on a CDN (CSS, JS, Images)
All files minified manually, including the major JS libraries in vendor
Bundling is a science and in 12 years of ecommerce I have never seen an app, be it Gulp, Grunt or even Magento bundle JS files together correctly. It is a complicated procedure and unfortunate it has not be cracked well. Even CloudFlare does not do it well. I never bundle JS files unless I do them manually.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[Khaleel]

Any updates on this? Knockout and the checkout is massive https://github.com/magento/magento2/issues/4868

[zchking]

Magento2.1.2 get 6M+ js file after merging, any update now？

[zchking]

does magento 2.1.3 solve this issue?

[viktor-zhuromskyy]

Did not check the 2.1.3 yet, but in 2.1.2 I had to manually inject requirejs-config.js into frontend and adminhtml in order to solve the issue.

[zchking]

thank you:)

[zchking]

@devdesco-ceo what's the js file size after you do that?

[zchking]

@okorshenko Does this commit fixes this issue, and how to patch it? thank you

[zchking]

any one still care this issue ?

[frustratedtech]

My issue with bundling is that, from my testing, the bundle is set to expire too quickly.
You basically have to download the file over and over again on page visit.
Eventually it caches and pulls from local cache.

[zchking]

yes, client cache is a way to make it load faster, but the js file should not so large.

[viktor-zhuromskyy]

My minified / gzipped bundle files total on average 700-750 KB, which is really good. On top of that amount of http requests dropped down to about 40%.

[sjauck]

This Ticket is part of Release 2.0.11CE/EE
http://devdocs.magento.com/guides/v2.0/release-notes/ReleaseNotes2.0.11CE.html
http://devdocs.magento.com/guides/v2.0/release-notes/ReleaseNotes2.0.11EE.html

[luism91]

@devdesco-ceo

Did not check the 2.1.3 yet, but in 2.1.2 I had to manually inject requirejs-config.js into frontend and adminhtml in order to solve the issue.

Do you mean in all the js files? I'm trying to bundle my javascript, i have the Porto Theme too.

[rkhoury82]

I can't see anything in 2.1.3 release notes but It appears that its fixed in 2.1.3 as well [including having to manually inject requirejs-config.js]?

[zchking]

think should drop the requirejs

[Khaleel]

I agree. Webpack. Gulp. Many better tools than RequireJS there is still so much bloat in Ajax calls, phtml, knockout and then requireJS - cannot do anything until requireJS has fired every ajax and knockout call.. then mixed in with localStorage - get back to American developers

[dunarri]

@viktor-zhuromskyy That is the size it should be, is that the result on v2.1.3?

[dunarri]

I'm the OP (changed name) and have retested the issue on the latest Magento v2.1.5.

With the merge and bundling options turned on (also note Magento has to be in production mode to use these settings) I now get 61 requests and 1.1MB transferred on a category page for about 5s load time.

The merged js file is still kinda large at 3.4MB uncompressed and this can be seen by
ls -lh pub/static/_cache/merged/

By comparison, testing the same page with merge and bundling turned off I get 200 requests, 131 of which are JS, and 1.0MB transferred. So it would appear they have addressed the issue.
Here's the kicker though, with merge and bundling turned off my page load time was only ~3s. So it doesn't achieve the desired result of faster page loads anyway. And I'm running HTTP/1.1, I'd imagine it gets even better with HTTP/2.

Also, as I stated originally the bundle includes the large JS libraries that could be offloaded to CDN (jQuery, jQuery-UI, Knockout, RequireJS).

In conclusion, don't use the bundling feature.
If you want to improve page load, get large JS libraries from a CDN and upgrade your web server to support HTTP/2.

[pynej]

The optimal setup is something like so:

HTTP/2 Nginx Proxy to Varnish Cache to Apache
With compression, minification, and etags/expiration manually set in apache as magenta doesn't so them all properly. At this point you can turn of file compilation but in reality the site really shouldn't be needing 131 separate JS @ 1MB files to build a simple page.

The only real issues is that due to how Magento does it's url rewriting rules I can't seam to get full site SSL working, and not just the admin/checkout pages.

[mombrea]

@pynej for full site ssl, try setting both the base url and base secure url to https

php bin/magento setup:store-config:set --base-url="https://example.com/"
php bin/magento setup:store-config:set --base-url-secure="https://example.com/"
php bin/magento cache:flush

[pynej]

Oh I'v tried that, The system gets really pissy and force rewrites URL's now matter how I do it when in production mode due to the HTTP Proxy layer. Oddly it seamed to work in standard mode.

[zetlen]

@dunarri and anyone else with these unacceptably large bundles: do you still have the inline sourceMappingUrl followed by lots of base64 at the end of your bundled file?

[benheart]

Hello guys, is there anyone solved this issue? I minified and merged the Javascript files and enable Javascript Bundling, but the bundled file is still very large(1M+), so home page load very slowly. I consider that can we use asynchronous js or move bundled js file to the foot of html body? But I don't know how to do that, if someone has good methods to solve this issue, please tell me, thanks very much!

[Khaleel]

@hwj727 One of the things we are doing:

• Minifiying all HTML
• Minifying all CSS
• Serving CSS on CDN
• Serving Images on CDN

I am going to try serving JS on the CDN too but there is a lot of JS which has PHP inside of it - such as Knockout files etc - so not sure about that. Magento bundling is poor

To place files at the bottom will only help a small amount as it is knockout dom rendering so it will make you site look funny - this also affects data-mage-init calls.. seriously.. Magento needs to get their shit together with this Front End.. especially the cart.. no documentation on serious front end development at all..

To move JS at the bottom edit the XML files in Magento Vendor files and theme XML files

[zetlen]

@hwj727 and @Khaleel: I'm sorry you're having so much trouble. Can I ask if either of you see the sourceMappingUrl section in your large JavaScript bundles?

[zchking]

2.2M after merge Magento 2.1.6, I can accept it:)

[Khaleel]

Even with bundling on Magento 2 are not using RequireJS optimiser so HTTP calls will still be over 40+ to 50+ for calling JS files (async or not async) the M2 default still has a heavy HTTP call..

[SchumacherFM]

Same here ... it's just a big PITA

[lingwooc]

I have 2.6mb merged .min.js which isn't too bad. It doesn't gzip though which rather spoils it. I have mod_deflate setup in the magento root .htaccess and it works for other files in _cache/merged. Any fixes?

Thinking about it, why aren't static files pre-gzipped when they are deployed rather than done on the fly each time?
edit: I'm using a cdn now which resolves the gzipping issue (as well as being generally better)

[magento-engcom-team]

@VegasJames thank you for your report.
The fix for this issue is already fixed in 2.0.11 release

[antonkril]

More efficient r.js-based bundling solution: https://github.com/antonkril/magento-rjs-config

[thaddeusmt]

FWIW, commenting on a closed bug... in 2.2.2 with JS bundling on (but not minification) the single JS file being output is 5.3MB right now. And our theme has bundle_size = 1MB set.

[antonkril]

Official bundling recommendation
https://devdocs.magento.com/guides/v2.3/performance-best-practices/advanced-js-bundling.html

[erikhansen]

@antonkril Thanks for sharing that detailed writeup about how to optimized Magento's bundling. I look forward to going through those steps for some of the Magento sites I'm working on. Due to the work involved with creating customized bundles, I expect the most merchants/developers will not implement them.

1. Do you anticipate a future core Magento release to ship with this type of configuration built-in?

2. Do you know of any Gist or Github pages that share bundle configurations for a vanilla Magento 2.2.x site?

[SKovbel]

2 ways

1. Duplicate each row with .min. in view.xml, exclude section

        <item type="file">Magento_Customer::js/zxcvbn.js</item>
        <item type="file">Magento_Customer::js/zxcvbn.min.js</item>

1. Fixing function isExcluded, ignore .min. suffix in comparasion
   if ($excludedFilePath === $filePath || $excludedFilePath === str_replace('.min.js', '.js', $filePath)) {

diff --git a/vendor/magento/module-deploy/Service/Bundle.php b/vendor/magento/module-deploy/Service/Bundle.php
index cf0d7a551..f661a69d9 100644
--- a/vendor/magento/module-deploy/Service/Bundle.php
+++ b/vendor/magento/module-deploy/Service/Bundle.php
@@ -193,7 +193,7 @@ class Bundle
         $excludedFiles = $this->bundleConfig->getExcludedFiles($area, $theme);
         foreach ($excludedFiles as $excludedFileId) {
             $excludedFilePath = $this->prepareExcludePath($excludedFileId);
-            if ($excludedFilePath === $filePath) {
+            if ($excludedFilePath === $filePath || $excludedFilePath === str_replace('.min.js', '.js', $filePath)) {
                 return true;
             }
         }

[webspeaks]

For me the culprit is JS bundling. I was seeing huge JS file (minified and bundled) even in admin side. Admin was too slow. Similarly frontend was slow due to huge JS file size.

Bundling enabled:
Admin minified and bundled js size: 17.2MB

Bundling disabled:
Admin minified and bundled js size: 100kb

That's a disastrous different for me. I am using Magento 2.3.0. So I turned off js bundling (even in production mode) and life is smooth now.

Disable js bundling:
php -f bin/magento config:set dev/js/enable_js_bundling 0

[magento-engcom-team]

Hi @VegasJames. Thank you for your report.
The issue has been fixed in magento/magento2#24506 by @hostep in 2.3-develop branch
Related commit(s):

• 41ef34feca73eea48f3dd1744bce529d36909c91
• cbe25f4960b59feed344eebee25e3dbe2f3efaaa

The fix will be available with the upcoming 2.3.4 release.