Shadow-cljs: One-file bundler for node

[arichiardi]

One of the features that I think would be good to have for completeness in shadow-cljs is the ability to generate a unique Javascript file for nodejs targets.

This is present for :browser but missing for node. In one of my projects I am doing the following: https://github.com/paullucas/les-clj/blob/master/scripts/build#L14-L15 in order to obviate to that.

Not urgent :smile:

[arichiardi]

I have switched to doing this with webpack, which is quite easy. Still, I think it would be nice to have it in shadow because I can then use the watch.

Another solution would be to call webpack in a hook. I have tried and failed this however.

[thheller]

I finally gave in and started implementing this. I didn't realize how many cloud-hosted solutions expect you to ship .js files without any deps.

This is not released yet since I need to test more but a basic build already works in master.

  {:target :node-script
   :main demo.script/main
   :output-to "out/demo-script-bundle/script.js"
   :js-options
   {:js-provider :shadow
    :keep-native-requires true
    :keep-as-require #{"foo" "bar"}}}

:keep-native-requires tells the compiler to use js/require for the node built-in packages instead of replacing them with node-libs-browser like :browser builds usually do.

:keep-as-require is an optional set of packages that should also stay as require instead of being bundled.

Only tested with :node-script so far. Needs lots more testing.

[thheller]

I released 2.7.25 which has support for this.

I tested some basic packages like request and it seems to work fine. Can't really make any general claims about the viability of this yet. Could use some help testing. Should work in all node targets (eg. :node-script, :node-library, :node-test). I may add a :standalone true or so config options to make it simpler in the config. Will be updating docs once I tested this a bit more.

[arichiardi]

Thanks Thomas I will definitely test this out with our Azure functions

[arichiardi]

Oh yeah actually I should look into adding it there too

[thheller]

The :azure-app target is built on :node-library so it should be covered.

[arichiardi]

Testing this as we speak

[arichiardi]

@thheller putting here an error that I receive in my functions:

[2019-02-06 11:46:53.637 - INFO] :shadow.build.npm/js-invalid-requires - {:resource-name "node_modules/bunyan/lib/bunyan.js", :requires [{:line 79, :column 17} {:line 100, :column 13} {:line 106, :column 27}]}
[2019-02-06 11:46:54.500 - INFO] :shadow.build.npm/js-invalid-requires - {:resource-name "node_modules/require_optional/index.js", :requires [{:line 82, :column 18} {:line 90, :column 20}]}
Closure compilation failed with 8 errors
--- node_modules/mongodb/lib/operations/db_ops.js:77
Illegal variable reference before declaration: db
--- node_modules/mongodb/lib/operations/db_ops.js:86
Illegal variable reference before declaration: db
--- node_modules/mongodb/lib/operations/db_ops.js:86
Illegal variable reference before declaration: db
--- node_modules/mongodb/lib/operations/db_ops.js:86
Illegal variable reference before declaration: db
--- node_modules/mongodb/lib/operations/db_ops.js:641
Illegal variable reference before declaration: db
--- remaining errors ommitted ...

The target I use is:

{:target :azure-app
               :app-dir "azure/build"
               :fn-map {:trigger projection-fn.azure/http-handler}
               :js-options {:js-provider :shadow  ;; <- just added
                            :keep-native-requires true}
               :devtools {:preloads   [projection-fn.preload]
                          :after-load projection-fn.preload/after-load!}
               :release {:compiler-options {:optimizations :simple
                                            :variable-renaming :off
                                            :property-renaming :off
                                            :pretty-print true
                                            :source-map true}}}

[arichiardi]

 :js-options {:js-provider :shadow
              :keep-native-requires true
              :keep-as-require #{"bunyan" "mongodb" "require_optional"}}

I am receiving a new error after adding the above (just guessing here):

Closure compilation failed with 2 errors
--- node_modules/@elasticpath/evently-dsl/dist/execution/hydration.js:48
This code cannot be converted from ES6. Undecomposable expression: Please rewrite the yield or await as a separate statement.
See https://github.com/google/closure-compiler/wiki/FAQ#i-get-an-undecomposable-expression-error-for-my-yield-or-await-expression-what-do-i-do
--- node_modules/@elasticpath/evently-dsl/dist/execution/support/evaluation-context.js:45
This code cannot be converted from ES6. Undecomposable expression: Please rewrite the yield or await as a separate statement.
See https://github.com/google/closure-compiler/wiki/FAQ#i-get-an-undecomposable-expression-error-for-my-yield-or-await-expression-what-do-i-do

[thheller]

You can try tweaking the :js-options {:language-out :ecmascript6} (maybe :no-transpile) option so that it stops trying to transpile stuff that node supports.

Not sure what those requires in bunyan are supposed to do or how webpack deals with them.

https://github.com/trentm/node-bunyan/blob/fe31b83e42d9c7f784e83fdcc528a7c76e0dacae/lib/bunyan.js#L98

I'll take a closer look when I find some time.

[arichiardi]

Thank you Thomas, thanks for the hint

[thheller]

I looked into the mongodb errors a bit and from what I can tell this is an actual "bug" in the mongodb package.

There are several uses of the db variable but then it is actual declared as a const later on in the same scope. I didn't check the JS spec but this seems like an error to me.

https://github.com/mongodb/node-mongodb-native/blob/057139058953715478d9bbe62bd5823781c7d1be/lib/operations/db_ops.js#L91

https://github.com/mongodb/node-mongodb-native/blob/057139058953715478d9bbe62bd5823781c7d1be/lib/operations/db_ops.js#L100

I can slim this down to a reproducible example

function outer(a) {
  (function inner() {
    console.log(a);
    const a = 1;
  })();
}

outer(2)

which fails to compile with the same error and also doesn't run in the browser (Uncaught ReferenceError: a is not defined) or node. I'm guessing that this function is never actually called as it should break if it was.

[arichiardi]

OK awesome thanks for debugging I will open an issue to the mongodb repo.

About the other one, we are using TypeScript for building the library that triggers those errors. TypeScript should already compile to ES6, Can I tell the GCC not to transpile only that JS module?

This is also pretty interesting to read and I didn't know about it: https://github.com/google/closure-compiler/wiki/FAQ#i-get-an-undecomposable-expression-error-for-my-yield-or-await-expression-what-do-i-do

[Marclev78]

This seems legal in JavaScript, the if block in which the db variable is re-declared is a different scope.

For example, if you have:

function f(a) {
  if (true) {
    const a = "b";
    console.log("1: " + a);
  }

  console.log("2: " + a);
}

f(5);

Then the output is:

1: b
2: 5

It sure looks like a code smell, but it's not an error as such.

[thheller]

@Marclev78 You missed the use-before-declared case which does error out.

function f(a) {
  if (true) {
    console.log("1: " + a);
    const a = "b";
  }

  console.log("2: " + a);
}

f(5);
// Uncaught ReferenceError: a is not defined

[arichiardi]

Ok opened thanks @thheller and @Marclev78 -> https://jira.mongodb.org/browse/NODE-1874

[thheller]

@arichiardi I don't use mongodb and don't want to install a server and neither do I want to create a Jira account over there. The problem should appear when calling the addUser fn, not sure how you do that but if that works the problem may be in GCC. So instead of trying to describe the problem you can probably save some time by creating a reproducible example.

[arichiardi]

No problem, I did both server and Jira already, will keep you posted. Thanks for debugging it btw.

[thheller]

2.8.10 has support for :output-feature-set which should increase the number of supported packages since most transpilation can be skipped for node.

[arichiardi]

Got it! Newbie question - Can I avoid transpiring but still compact the code with renaming and :simple? Or one excludes the other?

[thheller]

Renaming is not affected. :simple is basically not affected by :output-feature-set. Some :advanced optimizations are not yet compatible with :es6 and up but given that the polyfills don't need to be included I think it balances out. For node in particular it is probably advisable to use the highest possible setting node supports. None of this should affect the generated CLJS code, only the imported npm code.

[thheller]

@arichiardi there is a new closure compiler release where something regarding the method decomposing logic changed. maybe your code works now?

Try [com.google.javascript/closure-compiler-unshaded "v20190301"]. I'll probably make a release later when I'm done testing but I have no code to reproduce your problem so you'll have to test that yourself.

[arichiardi]

Thanks for the heads-up! Will try and report back.

[arichiardi]

Ok so, this looks like seems a tool issue:

The required JS dependency "fs" is not available, it was required by "node_modules/@elasticpath/evently-dsl/dist/loader.js".

Isn't fs a built-in module?

[arichiardi]

Oh I did not have :keep-native-requires true, it works now!

EDIT: it compiles, but the created artifact is still split

│   ├── cljs
│   │   ├── shared.js
│   │   └── shared.js.map
│   ├── host.json
│   ├── Makefile
│   └── trigger
│       ├── function.json
│       └── index.js

[thheller]

All the azure specific stuff isn't implemented yet. That needs to be done separately. You can confirm that this runs without node_modules present?

[arichiardi]

Got an error at runtime:

[3/6/19 8:09:49 PM] Executed 'Functions.trigger' (Failed, Id=d35563f1-72cf-464f-b396-b571debda845)
[3/6/19 8:09:49 PM] System.Private.CoreLib: Exception while executing function: Functions.trigger. System.Private.CoreLib: Result: Failure
[3/6/19 8:09:49 PM] Exception: Worker was unable to load function trigger: 'SyntaxError: Invalid or unexpected token'
[3/6/19 8:09:49 PM] Stack: /home/arichiardi/tmp/test-shadow-fn/cljs/shared.js:48
[3/6/19 8:09:49 PM] ya["[object Uint16Array]"]=ya["[object Uint32Array]"]=!0;ya["[object Error]"]=ya["[object Function]"]=ya["[object WeakMap]"]=!1;var Uc={"\\":"\\","'":"'","\n":"n","\r":"r","
[3/6/19 8:09:49 PM]                                                                                                                                                                             ^
[3/6/19 8:09:49 PM] 
[3/6/19 8:09:49 PM] SyntaxError: Invalid or unexpected token
[3/6/19 8:09:49 PM]     at createScript (vm.js:80:10)
[3/6/19 8:09:49 PM]     at Object.runInThisContext (vm.js:139:10)
[3/6/19 8:09:49 PM]     at Module._compile (module.js:617:28)
[3/6/19 8:09:49 PM]     at Object.Module._extensions..js (module.js:664:10)
[3/6/19 8:09:49 PM]     at Module.load (module.js:566:32)
[3/6/19 8:09:49 PM]     at tryModuleLoad (module.js:506:12)
[3/6/19 8:09:49 PM]     at Function.Module._load (module.js:498:3)
[3/6/19 8:09:49 PM]     at Module.require (module.js:597:17)
[3/6/19 8:09:49 PM]     at require (internal/module.js:11:18)
[3/6/19 8:09:49 PM]     at Object.<anonymous> (/home/arichiardi/tmp/test-shadow-fn/trigger/index.js:1:80).

Not sure where this is coming from...maybe a library.

[arichiardi]

After conversation on Slack with Thomas, this issue was solved by adding

 :js-options {:closure-output-charset "ascii"}

[arichiardi]

Another error I am getting, it seems related to bunyan's internals:

[3/6/19 9:41:18 PM] Executed 'Functions.trigger' (Failed, Id=a3651426-f77a-4319-bb9d-88a1dddcf59f)
[3/6/19 9:41:18 PM] System.Private.CoreLib: Exception while executing function: Functions.trigger. System.Private.CoreLib: Result: Failure
[3/6/19 9:41:18 PM] Exception: TypeError: z.createDTraceProvider is not a function
[3/6/19 9:41:18 PM] Stack: TypeError: z.createDTraceProvider is not a function
[3/6/19 9:41:18 PM]     at new k (/home/arichiardi/tmp/test-shadow-fn/cljs/shared.js:2493:3)
[3/6/19 9:41:18 PM]     at Function.x.exports.createLogger (/home/arichiardi/tmp/test-shadow-fn/cljs/shared.js:2517:418)
...

[arichiardi]

So the code line in the library seems like this one:

https://github.com/trentm/node-bunyan/blob/fe31b83e42d9c7f784e83fdcc528a7c76e0dacae/lib/bunyan.js#L442

Nothing special to my untrained eye.

The only difference that I see is that I can see here is that the logger gets created in a library, which h Is therefore a Cljs transitive dependency. I don't know if this affects anything really.

Can it be an externs issue?

[galdolber]

Just tried this and worked great!
The one thing I had to do is add source-map-support as external dependency on aws lambda.
I think it's treating source-map-support as a native module.

[thheller]

source-map-support is only part of development builds? It is not used in release builds?

[galdolber]

No, it's not for dev only. source-map-support is being used in the release as well.

[thheller]

shadow-cljs doesn't use it in release? You mean you use it manually in your release build?

[galdolber]

I have source-map-support as a dependency on my package.json.
With the configuration below, it's bundling all my dependencies, except the native ones and source-map-support. If I deploy the script to lambda it fails on require('source-map-support'). If I deploy the bundled script with the node_modules containing only source-map-support it works.

          {:output-to "target/admin/admin.js"
           :target :node-library
           :exports {:handler admin.server/invoke}
           :js-options {:js-provider :shadow :keep-native-requires true}
           :compiler-options {:source-map true}
           :optimizations :advanced}

[thheller]

Who is doing the require('source-map-support')? It isn't shadow-cljs doing that so it doesn't know about the dependency and won't bundle it. package.json is not considered when building the package, only things that are actually required in the build statically via ns. It does not account for manual (js/require ...) calls in CLJS.

Just to be sure we are talking about the same thing:
You are publishing whatever shadow-cljs release build produced to aws, NOT shadow-cljs compile build or shadow-cljs watch build output?

[galdolber]

Thanks! I see what happened. I was importing it with ``require()``` and not from the ns form.

(def source-map-support (js/require "source-map-support"))

[lambrospetrou]

Hello @thheller, first of all, thanks for the amazing work with Shadow-CLJS. It has really become my favourite tool to use for CLJS.

The last thing I was missing was this feature of bundling all the dependencies into a single file so I am glad there is already some work going on.

I am trying to use this new experimental feature with my new package create-shadow-cljs-app in order to only bundle the compiled JS file instead of the whole dependency graph when people use the tool which leads to way faster execution (it avoids the whole npm install).

The tree of the project I tried the following is https://github.com/lambrospetrou/create-shadow-cljs-app/tree/3af8c24535936b2fe2b393048d0b2f28522d0da9

I used the following options, based on the previous comments on this issue:

{:target :node-script
  :output-to "dist/index.js"
  :main cscljs.main/-main

  :js-options {:js-provider :shadow
              :keep-native-requires true
              :output-feature-set :es6
              :closure-output-charset "ascii"}

  :release {:compiler-options {
    :optimizations :simple
    :variable-renaming :off
    :property-renaming :off
    :pretty-print true
    :source-map true}}}

The above threw the warnings below:

shadow-cljs - config: /home/lambros/dev/github/create-shadow-cljs-app/shadow-cljs.edn  cli version: 2.8.51  node: v12.8.1
[:app] Compiling ...
[2019-08-25 13:10:08.690 - INFO] :shadow.build.npm/js-invalid-requires - {:resource-name "node_modules/shelljs/shell.js", :requires [{:line 25, :column 2}]}
[2019-08-25 13:10:09.216 - INFO] :shadow.build.npm/js-invalid-requires - {:resource-name "node_modules/yargs-parser/index.js", :requires [{:line 536, :column 21}]}
[2019-08-25 13:10:09.427 - INFO] :shadow.build.npm/js-invalid-requires - {:resource-name "node_modules/yargs/lib/apply-extends.js", :requires [{:line 43, :column 82}]}
[:app] Build completed. (105 files, 0 compiled, 0 warnings, 21.42s)

Based on the code logging these warnings it seems that moment had the same problem but kept working, but it seems that many other libraries depend on the inner requires to work.

I chased the causes of the above warnings to:

• shelljs
• yargs
• yargs-parser - this one has a slightly off number line

Is there anything I can do to give you more information that can help in solving these inner requires?

How are they treated right now? Are they just ignored, or is there some logic that tries to resolve them?

BTW, the above options do lead to a successful compilation but there are issues when running the output JS, for example:

module$node_modules$shelljs$shell.echo is not a function

Thanks again, let me know if there is any way I can help with.

[thheller]

@lambrospetrou shadow-cljs does not support using require in a dynamic way like the shelljs etc packages do.

You can try alternative one file bundlers like https://github.com/zeit/ncc with a regular :node-script build but I'm not sure if those support it either.

[lambrospetrou]

@lambrospetrou shadow-cljs does not support using require in a dynamic way like the shelljs etc packages do.

You can try alternative one file bundlers like https://github.com/zeit/ncc with a regular :node-script build but I'm not sure if those support it either.

Yeah, I spent several hours trying almost every popular bundler, including @zeit/ncc, Parcel-bundler, Rollup, and finally Webpack.

I believe only a combination of Shadow-CLJS first and then everything going through webpack managed to resolve the dynamic requires but then had other issues with __dirname which I use in my code (whole other different issue).

I guess it's a problem that libraries will need to solve eventually, especially with the new ES6 modules so I understand not wanting to support this in Shadow-CLJS directly.

Thanks again though for the great work.

[thheller]

shadow-cljs uses a whole lot of dynamic require (CLJ-variant) at runtime too. It is a super useful thing to have. It is IMHO also totally fine to do this for purely node-targeted libraries for the price of not being statically-linkable.

In your build config you can also set :js-options {:keep-as-require #{"shelljs" "yargs" ...}} which will cause this libraries to remain unbundled while still including all the others. Might be an option to at least remove some of your dependencies?

[lambrospetrou]

It is IMHO also totally fine to do this for purely node-targeted libraries for the price of not being statically-linkable.

Yeah, I don't disagree. I actually had to use it in one of my own projects in the past as well to keep the code leaner, but as with everything it's a matter of trade-offs.

In your build config you can also set :js-options {:keep-as-require #{"shelljs" "yargs" ...}} which will cause this libraries to remain unbundled while still including all the others. Might be an option to at least remove some of your dependencies?

God, I read the guide like a million times looking for something like :keep-as-require... I guess I will try again to integrate webpack and see if I can get closer to what I want.

I noticed that also :keep-native-requires mentioned previously in the issue is also not documented in the guide. Is it because they are experimental? Is there any _experimental guide/wiki/README_ with all these options that are not yet ready so that I can try them out in my experiments?

[thheller]

All this is experimental so the only "docs" are in this repo. :keep-native-requires and :keep-as-require are the only options currently. I'm not sure I will any more features. Seems like a lot of effort and other tools like ncc already do the same stuff with zero config.

[lambrospetrou]

Yeah, I agree there is no need to re-implement things that other tools do. I played around with using Shadow-CLJS along with webpack, ncc, rollup and it can work.

I decided to go with bundling everything Shadow-CLJS can do and just leave shelljs out with :keep-as-require #{"shelljs"}.

It seems that shelljs is problematic even with advanced bundlers like webpack so there is no way at the moment to bundle it properly anyway.

I think with the 2 experimental parameters it's pretty nice experience so I would say make them official :)

Awesome work, thanks again!

[thheller]

Although this kinda works I'm not going to document or spend any more time on this. It is recommended to use ncc or similar tools instead.