Shadow-cljs: Showing release build stats

[alex-dixon]

I came across this yesterday:
https://twitter.com/jamiebuilds/status/985316918453141504

I wondered if shadow-cljs had something similar that can show the advantage of CLJS with :advanced optimizations, and of course it does: https://clojureverse.org/t/help-wanted-release-bundle-size-visualization/871/9.

1. Is there any interest in outputting the results to the console as a table instead of in the browser?
2. Any interest in making this a shadow-cljs command? I followed the Clojureverse example to run it from shadow-cljs clj-repl. That works, but it's not very discoverable. Maybe once certain goals are met?
3. Are we confident the data output is accurate? I'm not able to quickly make sense of the numbers in the UI vs what I'm seeing in bundle-info.edn, in particular for cljs.core. It's probably just me but thought I'd bring it up anyway. Here they are:
   bundle-info.edn:

{:resource-id [:shadow.build.classpath/resource "cljs/core.cljs"], :resource-name "cljs/core.cljs", :module-id :base, :type :cljs, :output-name "cljs.core.js", :js-size 1266240, :source-size 330827}

UI:

[thheller]

Sure you can output this to the console. I initially set out to do a nice visualization for this but didn't get anywhere with that. I only did it in the browser because of that though.

The data should be pretty accurate. It is coming directly from the source map data.

:js-size 1266240
:source-size 330827

:js-size is the size of the generated JS, :source-size is the original code size (eg. cljs/core.cljs).

In the table in the browser I tried to "group" things by directory since that makes sense for node_modules stuff. No point in listing several hundred small files when all you care about is the total size of say node_modules/react,

Can only display the gzip size for the total bundle since gzip-ing the individual files would be vastly different summed up together.

[alex-dixon]

I think this is a separate issue but have you thought about surfacing a dependency graph? A question just came up on slack about why a dependency is being included. I'm confident shadow-cljs has that information but I'm not sure how it can be accessed from the CLI or REPL.

[thheller]

It is easily available yes but again I do not know how to visualize it properly. If you have something for this that expects data in a certain format I can show you how to get this data.

[stevebuik]

x-post from clojureverse...

I’d vote for a tree-map viz, works great for showing relative chunks in a fixed width/height display. can also do nice drilldown if done using d3. BTW: try react-faux-dom (npm dep) if you are using d3 with react, it fixes all the problems of react vs d3 fighting over the dom. I'd be happy to knock up a demo using the "custom targets" hook if that is feasible and useful?

e.g. https://bost.ocks.org/mike/treemap/

[thheller]

Other tools use tree-maps to visualize this information so it seems to be a good fit.

Not sure I understand the comment about "custom targets"?

[stevebuik]

I was thinking that https://shadow-cljs.github.io/docs/UsersGuide.html#_custom_targets would allow a fn to access the release-snapshot data. Is that possible?

[thheller]

Custom targets are meant to actual builds not utility stuff. You should look into clj-run instead. This lets you run Clojure code from the CLI, so you could create a shadow-cljs clj-run some.lib/visualize app where app is the build-id. This could call (shadow.cljs.devtools.api/release-snapshot :app {}) and then do some processing of the output after that step.

The output is always written to .shadow-cljs/release-snapshots/<build-id>/<tag>/bundle-info.edn. <tag> defaults to latest so for app you'd get .shadow-cljs/release-snapshots/app/latest/bundle-info.edn. The tag was meant as a way to enable keeping a "history" of sorts. So you could run (..//release-snapshot :app {:tag "1.0.0"}) and so on for every version and track how things change over time.

[alex-dixon]

@stevebuik Treemap would be great. Webpack bundle analyzer seems to use this exclusively: https://github.com/webpack-contrib/webpack-bundle-analyzer

@thheller I just tried this...https://github.com/alex-dixon/shadow-cljs/blob/d63d3bc1f620bfd724a9ff71340aa68be5c5d167/src/repl/shadow/cljs/ui_test.clj#L18

Do you remember if it works correctly? The path was off but once I changed it it spit out some JSON.

If it works, what are the disadvantages to loading this into the webpack bundle analyzer directly?

[thheller]

IIRC I did the webpack stuff for http://chrisbateman.github.io/webpack-visualizer/ not the webpack-bundle-analyzer. I think I got that to work yes.

webpack-bundle-analyzer seems to only be written as a webpack plugin which might make using it directly a bit hard. Not a fan of using this directly either since this has so many extra deps again. shadow-cljs already has way too many JS deps due to babel.

[thheller]

After the initial work from @alex-dixon I started adjusting a few things in an attempt to make the table more readable. My work project was otherwise basically unusable because it had too many files.

There is now an option to group the files. jars can be grouped as well as npm. For the demo build this looks like

| Resource                                   |   Optimized |  Total % |          JS |      Source |
|--------------------------------------------+-------------+----------+-------------+-------------|
| jar | org.clojure/clojurescript            |    281.9 KB |  37.68 % |      2.3 MB |    556.8 KB |
| npm | graphql                              |    226.8 KB |  30.31 % |    227.2 KB |    560.4 KB |
| npm | @firebase/auth                       |    139.6 KB |  18.66 % |    139.9 KB |    139.9 KB |
| jar | org.clojure/google-closure-library   |     26.4 KB |   3.52 % |      1.0 MB |      1.0 MB |
| npm | @firebase/util                       |     18.3 KB |   2.45 % |     18.4 KB |     65.9 KB |
| jar | org.clojure/tools.reader             |     17.0 KB |   2.28 % |    222.9 KB |     74.9 KB |
| jar | org.clojure/core.async               |     12.0 KB |   1.61 % |    405.8 KB |     56.2 KB |
|     | shadow/dom.cljs                      |      6.3 KB |   0.84 % |    100.4 KB |     17.4 KB |
| npm | tslib                                |      5.4 KB |   0.72 % |      5.4 KB |     11.2 KB |
| npm | @firebase/app                        |      4.4 KB |   0.59 % |      4.4 KB |     14.6 KB |
|     | demo/browser.cljs                    |      3.9 KB |   0.53 % |     20.2 KB |      2.2 KB |
| npm | iterall                              |      2.0 KB |   0.27 % |      2.0 KB |      4.7 KB |
|     | shadow/api.cljs                      |      1.6 KB |   0.22 % |     14.6 KB |      3.6 KB |
|     | shadow/js.js                         |      1.1 KB |   0.14 % |      3.0 KB |      3.0 KB |
|     | shadow/test/env.cljs                 |    317.0 B  |   0.04 % |      5.7 KB |      1.7 KB |
|     | demo/es6.js                          |    261.0 B  |   0.03 % |      1.2 KB |    382.0 B  |
|     | shadow/util.cljs                     |    195.0 B  |   0.03 % |      9.0 KB |      2.7 KB |
| npm | firebase                             |    165.0 B  |   0.02 % |    125.0 B  |    735.0 B  |
|     | demo/foo.js                          |    163.0 B  |   0.02 % |    134.0 B  |    213.0 B  |
|     | demo/browser_extra.cljs              |     92.0 B  |   0.01 % |    266.0 B  |    135.0 B  |
|     | demo/worker.cljs                     |     47.0 B  |   0.01 % |     83.0 B  |    118.0 B  |
|     | demo/never_load.cljs                 |     40.0 B  |   0.01 % |     77.0 B  |     87.0 B  |
|     | demo/always_load.cljs                |     39.0 B  |   0.01 % |     77.0 B  |     83.0 B  |
|     | demo/sm_before.cljs                  |     28.0 B  |   0.00 % |     65.0 B  |     55.0 B  |

Without grouping for jars

| Resource                                      |   Optimized |  Total % |          JS |      Source |
|-----------------------------------------------+-------------+----------+-------------+-------------|
| npm | graphql                                 |    226.8 KB |  30.31 % |    227.2 KB |    560.4 KB |
| jar | cljs/core.cljs                          |    166.8 KB |  22.29 % |      1.3 MB |    330.8 KB |
| npm | @firebase/auth                          |    139.6 KB |  18.66 % |    139.9 KB |    139.9 KB |
| jar | cljs/pprint.cljs                        |     96.3 KB |  12.87 % |    530.0 KB |    127.9 KB |
| npm | @firebase/util                          |     18.3 KB |   2.45 % |     18.4 KB |     65.9 KB |
| jar | cljs/spec/alpha.cljs                    |      8.8 KB |   1.17 % |    296.4 KB |     52.2 KB |
| jar | goog/base.js                            |      7.9 KB |   1.06 % |     97.3 KB |     97.3 KB |
|     | shadow/dom.cljs                         |      6.3 KB |   0.84 % |    100.4 KB |     17.4 KB |
| jar | cljs/tools/reader/edn.cljs              |      6.2 KB |   0.83 % |     36.2 KB |     15.4 KB |
| npm | tslib                                   |      5.4 KB |   0.72 % |      5.4 KB |     11.2 KB |
| jar | cljs/test.cljs                          |      5.1 KB |   0.68 % |     39.6 KB |     21.1 KB |
| jar | goog/math/integer.js                    |      4.7 KB |   0.63 % |     24.6 KB |     24.6 KB |
| npm | @firebase/app                           |      4.4 KB |   0.59 % |      4.4 KB |     14.6 KB |
| jar | cljs/core/async/impl/channels.cljs      |      4.3 KB |   0.57 % |     26.4 KB |      7.9 KB |
|     | demo/browser.cljs                       |      3.9 KB |   0.53 % |     20.2 KB |      2.2 KB |
| jar | cljs/tools/reader/impl/errors.cljs      |      3.5 KB |   0.46 % |     26.1 KB |      7.1 KB |
| jar | cljs/reader.cljs                        |      3.3 KB |   0.44 % |     21.0 KB |      7.2 KB |
| jar | cljs/core/async/impl/timers.cljs        |      2.4 KB |   0.33 % |     14.4 KB |      5.8 KB |
| jar | cljs/tools/reader/impl/commons.cljs     |      2.3 KB |   0.31 % |      9.5 KB |      4.4 KB |
| jar | cljs/tools/reader/reader_types.cljs     |      2.1 KB |   0.28 % |     38.5 KB |      8.9 KB |
| jar | goog/html/safehtml.js                   |      2.1 KB |   0.27 % |     37.8 KB |     37.8 KB |
| npm | iterall                                 |      2.0 KB |   0.27 % |      2.0 KB |      4.7 KB |
| jar | goog/string/string.js                   |      1.9 KB |   0.26 % |     53.2 KB |     53.2 KB |
| jar | cljs/core/async/impl/ioc_helpers.cljs   |      1.8 KB |   0.24 % |     28.8 KB |      4.6 KB |
| jar | clojure/string.cljs                     |      1.7 KB |   0.22 % |     15.2 KB |      8.2 KB |
|     | shadow/api.cljs                         |      1.6 KB |   0.22 % |     14.6 KB |      3.6 KB |
| jar | goog/dom/dom.js                         |      1.6 KB |   0.21 % |    111.2 KB |    111.2 KB |
| jar | goog/html/safestyle.js                  |      1.4 KB |   0.19 % |     20.2 KB |     20.2 KB |
| jar | cljs/tools/reader/impl/inspect.cljs     |      1.4 KB |   0.19 % |      8.4 KB |      2.7 KB |
| jar | cljs/core/async/impl/buffers.cljs       |      1.4 KB |   0.19 % |     16.8 KB |      4.1 KB |
| jar | goog/async/nexttick.js                  |      1.3 KB |   0.17 % |     10.3 KB |     10.3 KB |
| jar | cljs/core/async/impl/protocols.cljs     |      1.1 KB |   0.15 % |     13.3 KB |      1.8 KB |
| jar | cljs/tools/reader/impl/utils.cljs       |      1.1 KB |   0.15 % |     18.8 KB |      2.5 KB |
|     | shadow/js.js                            |      1.1 KB |   0.14 % |      3.0 KB |      3.0 KB |
| jar | cljs/core/async.cljs                    |    733.0 B  |   0.10 % |    304.2 KB |     31.2 KB |
| jar | goog/useragent/useragent.js             |    716.0 B  |   0.10 % |     16.6 KB |     16.6 KB |
| jar | goog/html/safeurl.js                    |    595.0 B  |   0.08 % |     16.8 KB |     16.8 KB |
| jar | goog/object/object.js                   |    514.0 B  |   0.07 % |     22.1 KB |     22.1 KB |
| jar | goog/array/array.js                     |    497.0 B  |   0.07 % |     59.4 KB |     59.4 KB |
| jar | cljs/tools/reader.cljs                  |    395.0 B  |   0.05 % |     85.5 KB |     33.9 KB |
| jar | goog/html/trustedresourceurl.js         |    354.0 B  |   0.05 % |     15.2 KB |     15.2 KB |
| jar | goog/style/style.js                     |    321.0 B  |   0.04 % |     75.9 KB |     75.9 KB |
| jar | goog/string/const.js                    |    319.0 B  |   0.04 % |      6.1 KB |      6.1 KB |
| jar | goog/string/stringbuffer.js             |    318.0 B  |   0.04 % |      2.7 KB |      2.7 KB |
|     | shadow/test/env.cljs                    |    317.0 B  |   0.04 % |      5.7 KB |      1.7 KB |
| jar | cljs/core/async/impl/dispatch.cljs      |    307.0 B  |   0.04 % |      2.0 KB |    824.0 B  |
|     | demo/es6.js                             |    261.0 B  |   0.03 % |      1.2 KB |    382.0 B  |
| jar | goog/asserts/asserts.js                 |    212.0 B  |   0.03 % |     13.6 KB |     13.6 KB |
| jar | goog/style/transition.js                |    198.0 B  |   0.03 % |      4.5 KB |      4.5 KB |
|     | shadow/util.cljs                        |    195.0 B  |   0.03 % |      9.0 KB |      2.7 KB |
| jar | goog/debug/error.js                     |    190.0 B  |   0.03 % |      1.8 KB |      1.8 KB |
| jar | goog/dom/safe.js                        |    189.0 B  |   0.03 % |     16.4 KB |     16.4 KB |
| jar | goog/labs/useragent/platform.js         |    177.0 B  |   0.02 % |      5.0 KB |      5.0 KB |
| npm | firebase                                |    165.0 B  |   0.02 % |    125.0 B  |    735.0 B  |
|     | demo/foo.js                             |    163.0 B  |   0.02 % |    134.0 B  |    213.0 B  |
| jar | goog/dom/tags.js                        |    149.0 B  |   0.02 % |      1.4 KB |      1.4 KB |
| jar | goog/labs/useragent/util.js             |    118.0 B  |   0.02 % |      4.3 KB |      4.3 KB |
| jar | goog/labs/useragent/engine.js           |    101.0 B  |   0.01 % |      4.7 KB |      4.7 KB |
|     | demo/browser_extra.cljs                 |     92.0 B  |   0.01 % |    266.0 B  |    135.0 B  |
| jar | goog/reflect/reflect.js                 |     88.0 B  |   0.01 % |      4.6 KB |      4.6 KB |
| jar | goog/dom/vendor.js                      |     84.0 B  |   0.01 % |      2.8 KB |      2.8 KB |
| jar | goog/dom/browserfeature.js              |     80.0 B  |   0.01 % |      2.4 KB |      2.4 KB |
| jar | goog/labs/useragent/browser.js          |     69.0 B  |   0.01 % |     10.6 KB |     10.6 KB |
| jar | goog/functions/functions.js             |     67.0 B  |   0.01 % |     14.8 KB |     14.8 KB |
|     | demo/worker.cljs                        |     47.0 B  |   0.01 % |     83.0 B  |    118.0 B  |
|     | demo/never_load.cljs                    |     40.0 B  |   0.01 % |     77.0 B  |     87.0 B  |
|     | demo/always_load.cljs                   |     39.0 B  |   0.01 % |     77.0 B  |     83.0 B  |
|     | demo/sm_before.cljs                     |     28.0 B  |   0.00 % |     65.0 B  |     55.0 B  |
| jar | goog/i18n/bidi.js                       |     12.0 B  |   0.00 % |     29.2 KB |     29.2 KB |
| jar | goog/dom/nodetype.js                    |      2.0 B  |   0.00 % |      1.4 KB |      1.4 KB |

Not posting the ungrouped npm stuff since it gets way out of hand. Removed the clojure.pprint/print-table since it was really bothering me that everything was right aligned.

I think this is pretty much the limit of what you can display in a readable fashion with just text.

[thheller]

I got tired of the limitations of the text and instead built the table in HTML via react-table. Everything is grouped by jar/package and can be expanded to see individual files.