Scala-js: Consider native GCC for faster optimization with less memory
Running fastOptJS/fullOptJS on multiple sbt sub-projects is slow and consumes a huge mount of memory.
For my case below, it takes 10 minutes on clean;fullOptJS and it requires to increase heap size like -Xmx8G or above. Otherwise OOM.
- 30 Scala.js sub projects
- 6 Core and 24 GB
Recently, Google Closure Compiler offers native binary version from v20180716 which is distributed as npm module. It is supposed to reduce startup time, optimization time and memory usage.
It would be great for developer experience if Scala.js bundle the native GCC to reduce compilation time and memory usage.
(Or maybe GCC binary compiled by GraalVM, if possible?)
exoego
All 2 comments
Yeah ... sorry, no.
At the moment, Scala.js core depends on nothing but a JVM, sbt and Node.js (and the latter only for running, and sbt only if one uses sbt and not something else). That makes Scala.js reliably easy to "install" on any kind of system. Bundling binaries is not something I'm willing to do, because it will jeopardize that property, as well as making our own testing infrastructure more complicated (e.g., right now we don't test MacOS at all, because I can trust that, given the technologies we use, anything that runs on Linux also runs on MacOS). And since there are no binaries for Windows, we would really have to duplicate our entire CI for Windows as well (currently we get away with some simple smoke tests) -- and our CI is worth tens of hours路CPU.
Moreover, Scala.js does not give JavaScript source files to Closure. Instead, we programmatically build Closure's internal AST directly from our JS AST. This allows to skip all the pretty-printing and parsing, but also reliably transfers source positions from our compiler to Closure. Doing so via source (which we would have to do with a native Closure) is much more difficult, source maps notwithstanding.
If someone wants to experiment with that, feel free to do so "at home". After all, one only needs to implement another subclass of LinkerBackend. You might have to put your code inside the org.scalajs.linker.backend.whatever package to get access to some ~internals, but that should be good enough for experimentation. If something comes out of it, we can publicize the relevant hooks.
sjrd
on 22 Dec 2018
I'm going to close this issue as the core team is not going to work on this, and it should be done in a separate repository anyway.
sjrd
on 11 Jan 2019
Related issues
Atry
路
4Comments
laughedelic
路
3Comments
japgolly
路
7Comments
dispalt
路
7Comments
sjrd
路
3Comments
Most helpful comment
Yeah ... sorry, no.
At the moment, Scala.js core depends on nothing but a JVM, sbt and Node.js (and the latter only for running, and sbt only if one uses sbt and not something else). That makes Scala.js reliably easy to "install" on any kind of system. Bundling binaries is not something I'm willing to do, because it will jeopardize that property, as well as making our own testing infrastructure more complicated (e.g., right now we don't test MacOS at all, because I can trust that, given the technologies we use, anything that runs on Linux also runs on MacOS). And since there are no binaries for Windows, we would really have to duplicate our entire CI for Windows as well (currently we get away with some simple smoke tests) -- and our CI is worth tens of hours路CPU.
Moreover, Scala.js does not give JavaScript source files to Closure. Instead, we programmatically build Closure's internal AST directly from our JS AST. This allows to skip all the pretty-printing and parsing, but also reliably transfers source positions from our compiler to Closure. Doing so via source (which we would have to do with a native Closure) is much more difficult, source maps notwithstanding.
If someone wants to experiment with that, feel free to do so "at home". After all, one only needs to implement another subclass of
LinkerBackend. You might have to put your code inside theorg.scalajs.linker.backend.whateverpackage to get access to some ~internals, but that should be good enough for experimentation. If something comes out of it, we can publicize the relevant hooks.