Rescript-compiler: [Proposal] Publish /lib/js/* as an npm package

[jchavarri]

The seems to be no way at the moment to use BuckleScript generated JS code in apps or libraries without forcing consumers to include bs-platformin their dependencies, or authors to bundle their code.

A real world case can be found in VS Code extensions (node.js apps). Publishing a VS Code extension with bs-platform in package.json would mean hundreds of MBs are sent to Microsoft publishing servers. Actually, any regular node.js app will have a similar problem when being deployed.

I'm pretty sure this has been discussed before, but I'm interested in the _minimum work_ required to make this happen:

• Use lerna or a custom script to automate this new package publishing
• Add an option in bsconfig.json package-specs so the output JS requires from this new package optionally. Or alternatively, make the separate package the only option.
• Change the compiler so the new package name is used on each module that depends on /lib/js

Happy to help with this.

Related:

• https://github.com/BuckleScript/bucklescript/issues/2082: Generated code requires modules from bs-platform, forcing an extra production dependancy
• https://github.com/BuckleScript/bucklescript/issues/1187: Treat stdlib as a normal package
• https://github.com/BuckleScript/bucklescript/issues/1860: [Discussion] Start of the stdlib infra

[bobzhang]

I tend to make stdlib a separate package, if we make it a separate package, how do we make sure stdlib is always there for the convenience of users?

If we extract alone, I am not sure what's the best way to do it since the path may be incorrect, there are two work arounds:

1. use rollup
2. has a script to remove those unneeded binaries when publish to production

I am unclear what's the best way to do it

[jchavarri]

I would definitely vote for 1 (with rollup or any other tool), because there are myriads of edge cases between OSs, environments, etc. that npm is already solving, and these cases would need to be considered with option 2.

The minimal approach that I can think of would be something like this:

• Create a new package (let's say bs-stdlib), that publishes the contents of /lib/js to npm
• Add /lib/js to .npmignore so that folder doesn't get included in bs-platform package any longer
• Add bs-stdlib to bsb/templates package.json files (as dependency).
• Require bs-stdlib in the generated js files

There's the issue of users having one version for bs-platform and another one for bs-stdlib, but I guess each dev is responsible of keeping the projects updated. To keep things simple, the best would be probably to lock bs-stdlib versioning to be the same as bs-platform.

cc @chenglou @bassjacob @jaredly as you've discussed this topic on Discord at different times in the past.

[TheSpyder]

I’m greatly in favour of having a separate stdlib npm module. I’m hoping to start adding BuckleScript generated code into my es6 projects before the end of the year and having code that doesn’t depend on bs-platform is on my list of requirements before I can do that.

[bobzhang]

If you make stdlib a separate package, how do you make sure it’s always included when you start a new project?

[jaredly]

If we maintain a separate repo (say bs-core) that is a mirror of the stdlib w/ tags for versions, then when someone wants to deploy to a nodejs env, they replace "bs-platform": "1.2.3" with "bs-platform": "git+https://github.com/Bucklescript/bs-lite.git#1.2.3"

[TheSpyder]

Well it would go in the bsb init template, which is a good start.

There’s already a custom error message for when bs-platform isn’t found, couldn’t a new one be added for when bs-core isn’t found?

Or maybe bs-core is what we tell devs to install and it has a peer dependency on bs-platform 🤔

[jaredly]

Here's my proposed solution in action:
If you replace, in your package.json

  "bs-plaftorm": "1.8.3",

with

  "bs-platform": "git+https://github.com/jaredly/bs-core#1.8.3",

npm will install my hollowed-out version of bs-platform (that doesn't try to build the ocaml compiler) into node_modules/bs-platform, and all your bucklescript-produced javascript files should Just Work™
@TheSpyder want to try that out?

[TheSpyder]

My code compiles as es6-global, so no that doesn't work ;)

I think we're all on roughly the same page; we want to be able to compile to JS, check that JS in, and then have other developers work with the codebase without requiring them to install bs-platform.

What you've done is apparently similar to how facebook does it, according to a quick chat I had with @chenglou on discord they maintain their own custom copy of the stdlib for this purpose.

[jchavarri]

I've created a proposal more or less inline with what @jaredly is proposing in https://github.com/BuckleScript/bucklescript/pull/2171.

On prepublish the main bs-platform it would automatically build the whole JS lib (cd jscomp && make world) and then publish it as the bs-stdlib package. It checks both packages have the same version + update the templates to include bs-stdlib as dependency.

The downside is that if / when this was shipped, the existing projects would need to manually include bs-stdlib as dep.

[jaredly]

Bucklescript could easily check package.json to verify that the dep is there & fail with a helpful error message. Doesn't seem like too bad of an upgrade path

[bobzhang]

how about encouraging people to not use stdlib by default? the ocaml stdlib is not very attractive, we can basically copy a stdlib as it is and evolve separately

[TheSpyder]

That seems separate to this goal; you'd have as much problem referring to any replacement stdlib in compiled JS as you would with the default implementation.

Unless you mean leave the default as-is for compatibility and require people to opt in to the replacement stdlib to avoid depending on bs-platform?

[bobzhang]

Unless you mean leave the default as-is for compatibility and require people to opt in to the replacement stdlib to avoid depending on bs-platform?

Yes, I think it is not a bad idea. you have more flexibility to evolve the api.
It is non trivial work to factor out stdlib,

• stdlib Makefile is a mess (not portable across OSes, I used a hack to make it work on Win)
• stdlib Format is not real Js friendly, stdlib polluted namespace too much
• stdlib requires special runtime support (for those c polyfill)

With a new stdlib alternative, we can use bsb for that stdlib and introduce conditional compilation etc. The downside is that it may not work with existing ocaml libraries, but we still keep the old ocaml stdlib for compatibilitiy reasons

[OvermindDL1]

Could just bring in Core in full? I really really do not want any kind of standard library that is not available from pure OCaml as well so we don't need entire sections of platform testing macro's to test what to call. ^.^;

[TheSpyder]

I understand where you’re coming from, but code using BuckleScript is already full of calls to Js.* modules that won't work in pure OCaml. And Core, sadly, generates _huge_ output files. There was some chatter a while back about building on top of Containers, though.

There are a lot of nice things we could do in a custom stdlib, but if we need to keep the standard one anyway can we please separate that development from this issue? I don't want to wait for a custom stdlib to be able to use bucklescript in projects where depending on the entire bs-platform is a hassle 🙂

[OvermindDL1]

Bucklescript uses such Js.* calls yes, but anything intended to be used by user code should either be properly wrapped in a macro (testing the environment) or should be useable everywhere.

[ncthbrt]

This issue is quite important to me for code intended for usage on server-side Node.js. Our code doesn't run through any processors aside from the reason compiler and having to bake ~200MB of additional code into a container is quite heavy.

[toots]

It would be nice if y'all could prioritize this. Just spent a mad evening working out our deploy on firebase functions and it wasn't very pleasant. Thanks!

[toots]

PS: I don't want to run through code processors. Those aren't reliable for my usage.

[aaronshaf]

Got bit by this today as well. Would much appreciate it.

[TheSpyder]

Work seems to be in progress:
https://bucklescript.github.io/blog/2018/11/19/next-half
https://bucklescript.github.io/blog/2018/12/05/release-4-0-8

[MoOx]

Until we got an official solution for this, here is my trick (very simple/stupid)

• I have a monorepo (yarn+lerna), so I am creating a new one named @my-namespace/bs-platform https://github.com/phenomic/phenomic/commit/9fca303956767248b5406142e3eb93dec1288432
• this new package have bs-platform as a dev deps and just copy lib/(js|es6) locally on prepare npm step https://github.com/phenomic/phenomic/blob/9fca303956767248b5406142e3eb93dec1288432/packages/bs-platform/package.json#L14-L23
• this package contains a tweak that will be used later https://github.com/phenomic/phenomic/blob/9fca303956767248b5406142e3eb93dec1288432/packages/bs-platform/tweak
• for all packages that are using bs-platform/lib/ from compiled code https://github.com/phenomic/phenomic/commit/5fb575723e2ae8c4995d460348758f90cad1d160
  
  
  
  • I add @my-namespace/bs-platform as a dep
  
  
  • I use js-post-build with a dirty replacement
  
  

This way, my project can safely ship js code generated by bs without the entire bs-platform (simple example https://github.com/phenomic/phenomic/commit/9656d92de1a8a7c469e62a2931c70df0e8cca2a5).

(Note that I use suffix .bs.js + in-source https://github.com/phenomic/phenomic/blob/5fb575723e2ae8c4995d460348758f90cad1d160/packages/core/bsconfig.json#L13-L16 for convenience with other tools and the fact that codebase has handwritten js and generated js)

Edit: I adjusted the tweak to make it cross platform https://github.com/phenomic/phenomic/commit/c763a1434c711841ca086b9d3780dc384f9a9fc4

[fakenickels]

Would be amazing to have this so JS users could take advantage of Reason packages without needing to install bs-platform

[Austaras]

I believe @fakenickels 's reason along should be enough to accept this proposal as of 2020 people are tending to ship unbundled code directly which requires a separate bslib package

[chenglou]

Done