Nw.js: Cannot compile es6 module source

Created on 23 Nov 2017 · 67Comments · Source: nwjs/nw.js

NWJS Version : v0.26.6
Operating System : macOS 10.13.1 (HighSierra)

Now we already can use es6 module features in v0.26.6 (Chrome 62). One example is:

`
-------------lib.js--------------

// lib.js
class TestModule {
foo() {
console.log('---foo----');
}
}

export {TestModule};

-----------testmodule.js----------------
// testmodule.js
import {TestModule} from "./lib.js";

var f = new TestModule();
f.foo();

-----------index.html----------------

<body>
    <script type="module" src="testmodule.js"></script>
    Hello.
</body>

----------package.json-----------
{
"name": "helloworld",
"main": "index.html",
"dependencies": {}
}
`

The above example can work properly. However, nwjc tool cannot compile them to binary code. The error message is:
`
$/Applications/nwjs/nwjc lib.js lib.bin

Failure compiling 'lib.js' (see above)
`

What's the correct procedure to use es6 module with nwjc? This is important for large scale application.

Thanks.

P2 bug triaged

Source

Mann90

👍1

Most helpful comment

For people who are having problems loading the es6 modules, I leave here as I do in my app so that it can help them.

Estructure:

mymodule_1.js | mymodule_1.bin

export function person( name = "Manu", age = 37, height = 180 ) {
    return {
        name: name,
        age: age,
        height : height ,
    }
}

mymodule_2.js | mymodule_2.bin

import { person as Person } from "./mymodule_1.js";

class ClassRoom {
    constructor() {
         this.students = [];
    }

    set_student( name, age, height ) {
        this.stundents.push( Person( name, age, height ));
    }

    num_students() {
        return this.stundents.length;
    }
}

const classroom = new ClassRoom();
export { classroom };

myscript.js | myscript.bin

import { classroom as Classroom } from "./mymodule_2.js";

class App {
    constructor() {
        Classroom.set_student( "Manu", 37, 180 );
        document.querySelector( "h1" ).innerHTML = "Num of students in classroom: " + Classroom.num_students();
    }
}

const app = new App();
export { app };

index.html

<html>
    <head></head>
    <body>
        <h1></h1>

        <!-- Importing modules -->
        <script>nw.Window.get().evalNWBinModule( null, './src/modulesES6/mymodule_1.bin', './mymodule_1.js' );</script>
    <script>nw.Window.get().evalNWBinModule( null, './src/modulesES6/mymodule_2.bin', './mymodule_2.js' );</script>

        <!-- Importing scripts -->
    <script>nw.Window.get().evalNWBinModule( null, './src/scripts/myscript.bin', './myscript.js' );</script>

        <script type="module">
            import { app } from "./myscripts.js";
        </script>
    </body>
</html>

Note that the import in * myscript.js *, I do not put the relative path to the module that I am importing, I put the base path of the app. Where would you have to write import { classroom as Classroom } from" ../ modulesES6 / mymodules_2.js ", write import { classroom as Classroom } from" ./mymodules_2.js ".

You just have to adapt it to the structure of your app and you should not have problems to make it work.

It may seem a bit cumbersome, but with a little skill, I have programmed a compiler that adjusts the import paths in the modules automatically when I compile the application, as well as changing me <script type="module" src=" ../ scripts / mysscript.js"></script> by <script nw.Window.get().evalNWBinModule(null, './src/scripts/myscript.bin', './myscript.js ');</script> in html files also automatically.

So with a single console command, the application is compiled, modifies imports in .js and scripts in html, packaged in exe and zip and uploaded to my repository on my server by FTP ready for users to download or when a user who already has it installed, receive automatic update and install.

It is undoubtedly the ability of nw.js to convert .js files into .bin files that prompted me to switch from electron to nw.js. When I tried nw.js and saw its capabilities I will never go back to electron. Not only because of the protection of the scripts, but also because of the operation of the application's contexts, the possibility of including Polymer that didn't work in electron and the non-SDK flavor that doesn't allow inspecting the app's windows.

Once you understand nw.js you can easily make node.js an automatic compiler with a little effort that even works better with electron builders.

I hope to help.

manujoz on 21 Feb 2020

❤4 👍1

All 67 comments

I can reproduce this issue on Linux with nwjs-sdk-v0.26.6.

Christywl on 24 Nov 2017

I just pushed a fix for nwjc to nw26 branch. Added a --nw-module command line switch which should be used when a module is being compiled.

We need a way to load compiled module binary into the application in the next step. Please propose.

rogerwang on 24 Nov 2017

We need a way to load compiled module binary into the application in the next step. Please propose.

You mean another new API similar to evalNWBin? A quick-and-dirty way is make a evalNWBinModule?

Mann90 on 24 Nov 2017

I'm not sure yet. testmodule.js wants to load lib.js, which is obviously not there when there is only binary.

rogerwang on 24 Nov 2017

Any proposal should take this into consideration: https://html.spec.whatwg.org/multipage/webappapis.html#integration-with-the-javascript-module-system

rogerwang on 24 Nov 2017

How long could it be taken to support module completely in V8 binary? Are there any technical difficulties? Thank you.

Mann90 on 5 Dec 2017

It should not be difficult to implement. But we're waiting for API
proposal, which would work well with work flow on the application
developer's side.

On Dec 5, 2017 10:47 PM, "Mann90" notifications@github.com wrote:

How long could it be taken to support module completely in V8 binary? Are
there any technical difficulties? Thank you.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/nwjs/nw.js/issues/6303#issuecomment-349325441, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAKGGU5MzizJxTjudRBrj8ag66I7nihNks5s9VdpgaJpZM4Qo-3j
.

rogerwang on 6 Dec 2017

Is there any progress on this?

SMotaal on 22 Dec 2017

@smotaal as said in the last comment we are waiting for proposal from application developers. So if you want it to be implemented soon, please take some time to submit a proposal first.

rogerwang on 22 Dec 2017

👍1

@rogerwang I was keeping a very close eye on the work the NodeJS team was doing to adopt V8's actual Modules using their own ModuleWrap (C++) objects together with their new internal/loader (JS) API. They made a very solid effort in keeping close to the standards to the point that they used file:// urls and are even working on porting Blobs and createObjectURL to make sure that they never expose the C++ implementation to the wild, ie no unchecked injection potential.

The bottomline is that it takes a hefty learning curve to understand how the new internal/loader works, especially when it was a moving target with --experimental-modules in versions 8 and 9 and not to mention the sometimes seemingly abrupt design choices which were driven by very lengthy and disjointed discussion threads.

IMHO I don't think that dealing with module loading should be an ad hoc effort if the goal is to get the community involved, it takes dedication and structure.

I am willing to be part of this effort, but let's not loose sight that like many application developers, I have a very limited understanding of the internals of NWJS, even when I spent weeks crawling over your various repositories and discussion threads, without knowing your workflows and with no clear context, it all seems like magic sometimes. I am only trying to rely an outsider's perspective and I am certain of your tremendous efforts behind the project and especially in involving the community in your efforts.

That said... The most important aspect behind Node's new loader is that we finally have specifications and native V8 implementations. In essence, you use a resolve(specifier, referrer) to map import paths to absolute urls and format (module/script) and an opt-in async dynamicInstantiate that can declare exports for a given url's namespace with getter callbacks for those declared names (called once during Module::instantiate).

So this is just a conversation starter, I am hoping we can keep this conversation going.

SMotaal on 22 Dec 2017

I want to share how my thinking has evolved about modules with the evolution of native implementations (compared to before):

Shall never down-transpile modules, should not even transpile (aside from esnext or non-es code with clear user opt-in). So all the existing function-wrapped module/bundling ecosystems are outdated in my books.
Bundling should recreate [sic] modules (if only as blobs) with url mapping at the loader level without any string manipulation.
Transpiler functions should not supersede the loader by overloading it, they can use it's hook methods and defer control to the loader. They should function deterministically and be agnostic to the environment (apart from contextual aspects for asset handling and resource optimization). They can chain into the resolve(specifier, referrer, defaultResolve) as "custom loaders" to perform the necessary functionality that would resolve a url and a format that is known to the loader. So they might need to verify a few aspects with the loader.
Context-specific async injectors or factories for specific "custom formats" should only be allowed to act on completely resolved urls with the only purpose of linkage (eg: document.createElement('link') ... {href, type} …, or fetch.then(…)) ensuring proper exception handling to fulfill all upstream import(…).then() chains.
This one was counter-intuitive at first, but a URL (including it's ?query) is only instantiated once, which is an acquired taste that grows on you when you realize how far this simple compromise can go in ensuring clear and predictable behaviour. So those getters I mentioned above are evaluated on instantiate and the returned values are locked in for the life span of the process / page.

SMotaal on 22 Dec 2017

Thanks @smotaal . Chromium engine had its module system implemented and it's working in NW. We could just reuse it for module binaries. It's just that a module would load its dependency by referring to it's JS filename (see testmodule.js importing lib.js in OP's example).
Regarding NW.js binary module, it might not be feasible to let developers change the source code to refer to the binary's filename (importing lib.bin instead of lib.js in testmodule.js), because that would require changing all the source code when the binary modules are to be built. I believe that would break developers' work flow.

I am thinking to extend the API Window.evalNWBin() which NW used to load compiled JS binary to something like :
evalNWBin(null, 'lib.bin', 'lib.js');

This creates a mapping between the paths of the module source and binary. After that, each time when Blink engine wants to load module lib.js, it will try loading lib.bin if the source code is missing.

I'm not sure if this would work the best way for JS application developers because my first language is C++. So please advise.

rogerwang on 23 Dec 2017

👍1

So I just want to clarify first that you are only referring to modules imported from inside a <script type="module" … and not in node's context (with --experimental-modules flag)...

The reason I ask is that so far in all my tests, I only had success using import … inside <script type="module" … but node contexts throw the same error that you get if you run node without explicitly opting for --experimental-modules.

quick side note… I am yet to uncover a way to pass --experimental-modules to node contexts other than to force execArgs to child_process but this creates yet another degree of separation between the contexts and may only be ideal for worker-like functionality.

So I concluded from my tests that nwjs still did not address this huge transition that allows native ES modules in nodejs which I believe is planned for primetime (without flags) in the first public release of v10.0 (stable).

Is that a correct assumption?

SMotaal on 23 Dec 2017

<script type="module"> was shipped in Chrome 61 and JavaScript module import() was just shipped in Chrome 63. Both should work fine in latest NW 0.27, which is based on Chrome 63.

Chrome stable often has newer V8 version than Node.js latest stable so it has more features. In NW (thus the Node.js code in NW), we are always using Chrome's V8 version. So you should be able to use all those features by default without any flags.

rogerwang on 24 Dec 2017

Thanks for the overview @rogerwang... Trust me, I've been keeping a very close eye on all the developments regarding native implementations for ES modules, especially in the V8 realm.

I apologize for the delay, but I used the last few days catching up on the internals of NW, rebuilding your latest nw28 mac builds and playing around with node's ESMLoader subsystem in separate and mixed contexts to be able to elaborate on the issues that I am trying to outline.

So now that I got the ESMLoader to work in node contexts, I must point out the following:

Node's and Chrome's internal code base has and continues to be (for obvious historically necessities) script-based using non-standard JS modules (which are actually simply javascript code encapsulated in functions).

The now "legacy" ways of dealing with fake JS modules in Chrome, Node, NW, Electron, everyone, are not interchangeable with ES modules because they all evaluate the code inside a function, and for all intents and purposes, import and export are top-level only.

Instead of the idea of a "loader", ES modules require a "resolver" that maps every imported "specifier" to a URL that when accessed will pass all the security conditions necessary (like CORS and MIME) outside of the scope of the module subsystem. And the wammy, just for kicks, every unqiue URL get's evaluated only once in a given execution context and only after all it's imports have been resolved.
Node and Chrome both use v8::Module (excuse my C) when a standard ES module is used but each use their own specifier-to-url mapping subsystem (which are "only partly conceptually" similar to what goes on in node-nw/lib/module) but are completely independent from their previously exisitng fake-JS-module subsystems.

Chrome (for obvious reasons) sticks very close to the loader specs you mentioned above and since the formalization of the "custom loader" aspects did not land in the first round and the spec is now in upstream limbo, they did not implement (or at least don't expose) a way for us developers to be able to map "at the module level" things like import myapp from 'app:/…' to specific urls. It is possible however to intercept requests and play around with those (but that complicates the simplicity of ES module specifiers).

Node on the other hand decided to hide this ugliness behind a subsystem of their own making and expose a new parallel but independent hook-based mechanism (like the one they used for "fake" Common JS modules) which for all it's challenges complies with at least one of the most fundamental aspect of ES modules, that a module is evaluated only once (in the process's sole context), and never evaluated until all it's dependencies are resolved. Their poorly named --loader is fundamentally a --resolver and/or --loader because the resolver aspect is far more important for native ES module consumption than the non-standard perks that you can bake into those custom loaders.
Although there is no doubt that both Chrome's and Node's ES module related extension subsystems will coexist in NW, but they do not yet cooperate. For valid ES modules to cooperate, they must resolve equivalently across all contexts types, but load equivocally based on the context, so that importing node's process yields a wrapped module around a specific node-context bound process module exports when imported in a browser context or any other context, and this is where things start getting tricky.

IMHO, as a JavaScript developer using an application development platform that can use web technologies "outside of the constraints imposed by the web", Node's ESM subsystem provides the necessary mechanisms to use ES modules efficiently and effectively.

The same resolution mechanism should be followed for non-web-only uses:

When <script type="module" src="{{specifier}}"> or <script type="module"> import x from "{{specifier}}" or <script type="module"> import("{{specifier}}").then(…) or [console] > import("{{specifier}}").then(…) where document.baseURL (or the baseUrl of the ESMLoader of the node-context of the background-page) is the referrer, then URL[content-type='text/javascript'] is the result of calling the custom loader's resolve(specifier, referrer), which is hooked into the ESMLoader as a custom resolve hook.

When import x from "{{specifier}}" or import("{{specifier}}").then(…) statements where import.meta.url is the referrer (ie inside a module), then URL[content-type='text/javascript'] is the result of calling the custom loader's `resolve(specifier, referrer)… etc.

Node's implementation uses special protocols for resolved URL, like "node:process" which resolves to the builtin "dynamically instantiated" node module that exposes as "constant snapshot" of the CommonJS exports yielded by require('process') and they are naturally "context-bound". While context's may be a little tricky to figure out, still NW can also use bin:{{filename[.bin]}} as a resolved URL since it can check before hand if it should use the bin version or the file://…{{filename[.js|.mjs|.m.js]}}.

So when I import process from "process" node's default resolver calls my custom resolver and then it is "process", so node's default resolver now maps that to the special protocol "node:process" and handles that for me. The same can be accomplished for relative paths and bin-files... etc.

Chrome's implementation requires [content-type] for ES Modules and it does not allow except the "web standards" protocols which makes sense for web content but not for NW applications, so I gather there needs to be some work there.

https://github.com/nwjs/chromium.src/blob/45be0261c5a04cc38dead115740a8a6f7e6958a7/third_party/WebKit/Source/core/loader/modulescript/DocumentModuleScriptFetcher.cpp#L39-L51

Now here is the punchline:

If we are still talking real ES modules, not fake ones or "rollups" as bundles, then this is not a simple thing to do, because evalESMFromBin() (if it does not hook into "legacy" eval() or function-wrapped module loading mechanisms) must be a special variant of the native implementation of the import().then() which would revive a snapshot of those previously resolved modules modules and all their dependencies as if they had just been resolved from disk, and do so while also checking if one of those resolutions should be substituted when the file is found at the respective resolved path (for optional overloading).

FYI: I did not even bother looking into Electron since it really caters to Atom and VSCode and both already locked themselves with huge fake-JS module code bases and their own loaders, so they are in their own happy land.

SMotaal on 27 Dec 2017

In NW applications the files are with chrome-extension:// protocol, where the URL response has an expected content-type so Chrome's implementation should work well.

rogerwang on 27 Dec 2017

btw, are you saying that Chrome doesn't support "real ES modules"? I think it supports well and my idea of extending evalNWBin is going to reuse/hook into its implementation.

rogerwang on 27 Dec 2017

I do apologize for the very long post and multiple edits.

SMotaal on 27 Dec 2017

btw, are you saying that Chrome doesn't support "real ES modules"? I think it supports well and my idea of extending evalNWBin is going to reuse/hook into its implementation.

Chrome uses real ES modules, without doubt, however, all their existing Javascript code (ie if you look in their own extension:// files in the inspector) still uses "legacy" function-wrapped modules because ES modules require a lot of rewiring and it is not a simple switch.

SMotaal on 27 Dec 2017

I see, but that's only for their UI of the devtools or other web-ui for settings, etc. It has nothing to do with the code from the web page or NW application.

rogerwang on 27 Dec 2017

In NW applications the files are with chrome-extension:// protocol, where the URL response has an expected content-type so Chrome's implementation should work well.

Yes, but here is a tricky thing, if we don't use a custom resolver on the node end then node will only load ESM modules from files with the .mjs (tested with a modified lib/bootstrap_node.js that enabled ESMLoader if process.__nw by default since there is no way to configure NW to pass --experimental-module which node looks for in process.binding('config')).

So when I managed to load ./module.mjs in node, and tried to load chrome-extension:…/module.mjs, Chrome complained that the URL is not a valid content-type as per spec, which I have no means of dealing with as a javascript developer (it's hard coded in the lines mentioned above).

SMotaal on 27 Dec 2017

I see, but that's only for their UI of the devtools or other web-ui for settings, etc. It has nothing to do with the code from the web page or NW application.

True, but I figure, they did not really deal with non-web use cases themselves, so they did not do much of the work that the Node folks had to do (and here I was with the rest of the community wondering why they were so slow).

SMotaal on 27 Dec 2017

So when I managed to load ./module.mjs in node, and tried to load chrome-extension:…/module.mjs, Chrome complained that the URL is not a valid content-type as per spec, which I have no means of dealing with as a javascript developer (it's hard coded in the lines mentioned above).

Are you hitting this bug? https://bugs.chromium.org/p/chromium/issues/detail?id=797712 Otherwise it should work.

rogerwang on 27 Dec 2017

The nice thing is that v8::Module is the lowest common denominator. However, I should point out that when node tried to load a module for a URL that was already loaded in chrome (hence the v8::Module already existed for this url) I think it might have caused because of ESMLoader subsystem works with certain expectations of internalized side-effects that were not wired correctly for that instance of the v8::Module.

SMotaal on 27 Dec 2017

Yes, but here is a tricky thing, if we don't use a custom resolver on the node end then node will only load ESM modules from files with the .mjs

Finally I see. You want to make Node's module support and Chrome's working together ... I think it should be a separate issue. What OP is requesting is about supporting this NW feature with the module feature of Chrome.

rogerwang on 27 Dec 2017

Are you hitting this bug? https://bugs.chromium.org/p/chromium/issues/detail?id=797712 Otherwise it should work.

I used a custom resolver in node to allow it to load .m.js as an ES Module (like it would for '.mjs') and created three files:

A .js using require('process') and require('console')
A .m.js and a .mjs which import … from 'process' and import … from 'console'.

Here are some logs:

When I switched node's ESMLoader on I got this:

06:55:17.988 bootstrap_node.js:542 [Deprecation] 'webkitURL' is deprecated. Please use 'URL' instead.
(anonymous) @ bootstrap_node.js:542

06:55:17.994 bootstrap_node.js:542 [Deprecation] 'window.webkitStorageInfo' is deprecated. Please use 'navigator.webkitTemporaryStorage' or 'navigator.webkitPersistentStorage' instead.
(anonymous) @ bootstrap_node.js:542

When i tried this:

(async () => {console.log(await import('./lib/hello.mjs'))})()

I got this:

10:12:38.548 Promise {<pending>}
10:12:38.547 hello.mjs:1 Failed to load module script: The server responded with a non-JavaScript MIME type of "". Strict MIME type checking is enforced for module scripts per HTML spec.
10:12:38.548 VM169:1 Uncaught (in promise) TypeError: Failed to fetch dynamically imported module: chrome-extension://ingklgoklajmcipngeekcmdnhpobfglg/lib/hello.mjs
async function (async)
(anonymous) @ VM169:1
(anonymous) @ VM169:1

(async () => {console.log(await import('./lib/hello.m.js'))})();

06:55:32.486 Promise {<pending>}
06:55:32.485 hello.m.js:1 GET node:process net::ERR_UNKNOWN_URL_SCHEME
06:55:32.485 hello.m.js:2 GET node:console net::ERR_UNKNOWN_URL_SCHEME
06:55:32.486 VM148:1 Uncaught (in promise) TypeError: Failed to fetch dynamically imported module: chrome-extension://ingklgoklajmcipngeekcmdnhpobfglg/lib/hello.m.js
async function (async)
(anonymous) @ VM148:1
(anonymous) @ VM148:1

So I created a custom Module.prototype.import in 'lib/module.js' which awaits ESMLoader.import() so it is a node-context-bound dynamic import wrapper and tried this:

(async () => {console.log(await module.import('./lib/hello.mjs'))})();

10:12:32.356 VM166:1 undefined
10:12:32.362 Promise {<resolved>: undefined}

But so far all this while not even attempting any export statements.

SMotaal on 27 Dec 2017

I also dump to the stdout resolutions made by the custom resolver, and here is what I get from my node-main:

[RESOLVE]: "file:///Users/daflair/Projects/[Other]/nw.js/src/_generated_background_page.html"    ->    "./lib/hello.m"
  --> "file:///Users/daflair/Projects/[Other]/nw.js/src/lib/hello.m.js" [ESM]
[RESOLVE]: "file:///Users/daflair/Projects/[Other]/nw.js/src/"    ->    "/Users/daflair/Projects/[Other]/nw.js/src/lib/hello.m.js"
  --> "file:///Users/daflair/Projects/[Other]/nw.js/src/lib/hello.m.js" [ESM]
[RESOLVE]: "file:///Users/daflair/Projects/[Other]/nw.js/src/lib/hello.m.js"    ->    "node:process"
  --> "process" [BUILTIN]
[RESOLVE]: "file:///Users/daflair/Projects/[Other]/nw.js/src/lib/hello.m.js"    ->    "node:console"
  --> "console" [BUILTIN]
hello world! undefined
{ imported: {} }

import console from 'console';
import * as hello from './lib/hello.mjs';

(async () => {
    try {
        const imported = await module.import('./lib/hello.m');
        console.log({ imported });
    } catch (exception) {
        console.error(exception);
    }
})();

SMotaal on 27 Dec 2017

Finally I see. You want to make Node's module support and Chrome's working together ... I think it should be a separate issue. What OP is requesting is about supporting this NW feature with the module feature of Chrome.

I understand so here:

var xhr = new XMLHttpRequest();
xhr.responseType = 'arraybuffer'; // make response as ArrayBuffer
xhr.open('GET', url, true);
xhr.send();
xhr.onload = () => {
  // xhr.response contains compiled JavaScript as ArrayBuffer
  nw.Window.get().evalNWBin(null, xhr.response);
}

You want to trigger the entry point which is instead an ES module, so if I understand the general idea correctly, will either be unpacking a snapshot of the runtime environment side-effects (ie interlinked v8::Modules) from the entries in the bin or emulating loading the JavaScript source code as if it was being loaded from the original source (which I think is the case currently with evalNWBin).

So my concern here is, in either cases, how will you go about dealing with import x from '…' in my protected source.

SMotaal on 27 Dec 2017

I think I explained this well in https://github.com/nwjs/nw.js/issues/6303#issuecomment-353698074

rogerwang on 27 Dec 2017

Yes, I recall that, perfect, so can you elaborate a little on how you coerce blink's resolution there

SMotaal on 27 Dec 2017

What I am trying to do here is connect the dots that you have already connected, which will solve resolving specifiers, because if I create a bin for ES modules, those modules need their specifiers resolved, and you already have the mechanism to do so in blink, but ultimately, those modules are not being consumed by the node process as true ES modules, so I am trying to bridge the two worlds in such a way that they would.

In my opinion, as it turns out, the egg happened before the chicken, so an integrated ES module logic is part of the solution of real ES modules as compiled javascript (ie javascript source code in a single-file representing multiple files "virtually" by coercing mappings in the engine not bundled javascript that yields IIFEs)

SMotaal on 27 Dec 2017

https://github.com/nwjs/chromium.src/blob/45be0261c5a04cc38dead115740a8a6f7e6958a7/third_party/WebKit/Source/core/workers/WorkerGlobalScope.cpp#L178-L197

SMotaal on 27 Dec 2017

Yes, I recall that, perfect, so can you elaborate a little on how you coerce blink's resolution there

Blink maintains the modules map here https://github.com/nwjs/chromium.src/blob/nw28/third_party/WebKit/Source/core/dom/ModuleMap.h

Will change it to add the support needed for the new evalNWBin(), which I'm going to add before 0.28.0 release and mark the new API experimental (subject to change in the future)

rogerwang on 28 Dec 2017

👍1

so an integrated ES module logic is part of the solution of real ES modules as compiled javascript

The compiled javascript OP and I talked about here in this issue is not the same concept with yours. We were talking about a specific NW feature to transform JS source to binary (similar with what happened to C/C++ source code), not ' javascript source code in a single-file representing multiple files' compilation used by many JS programmers.

rogerwang on 28 Dec 2017

👍1

Yes, sorry about that, I was missing the key aspect that a bin file === a js file not an archive/snapshot of multiple files — but rest assured, I was not talking about js-wrapped bundles.

Still, thank you so much, your insights are very valuable :)

My apologies to everyone in this thread!

SMotaal on 29 Dec 2017

This is fixed in nw28 branch and will be available in the next nightly build.

rogerwang on 4 Jan 2018

This is done for now by adding evalNWBinModule() as proposed. Please see the docs change for usage. The API is subject to change in future versions.

rogerwang on 4 Jan 2018

Available now at https://dl.nwjs.io/live-build/01-05-2018/nw28-9c2248c-b158474-caf1f0d-1b41f4f/v0.28.0-rc1/

rogerwang on 5 Jan 2018

Awesome!!

But there are two issues found:

If import * from './lib.js' is not there, the nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js'); doesn't load and run the script.

For example, if there is a top-level statement console.log('lib loaded.'); in the lib.bin (lib.js), then that statement doesn't run actually.

Why does nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js'); take the third argument? Could we simplify it to nw.Window.get().evalNWBinModule(null, 'lib.bin'');. (And we assume that foo.bin must be generated from foo.js.)

Or even further, we could unify evalNWBinModule and evalNWBin, let NW to check if the binary is a compiled by --nw-module flag or not. Is it possible?

Another option is to add a new boolean argument to evalNWBin to indicate if the lib.bin is a module or not.

Mann90 on 26 Jan 2018

If I understand correctly the documentation, forgive me to not have yet tested the correctness of my understanding, when using compiled modules, we have to explicitely preload all the binaries via the new api before importing them into the application code.

So we have to programmatically create a map of all needed modules, then import all of them via evalNWBinModule before even starting the application code:

<script>
/* 
  index.html 
*/
/* here we start with preloading all the binary modules 
for them to be available to import in code*/
nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js');
nw.Window.get().evalNWBinModule(null, 'dep.bin', 'dep.js');

import('./lib.js')
</script>

/* lib.js */
import('./dep.js');



md5-c05c762b132944f363e6bc36a1090aa8



```javascript
/* 
lib.esm 
*/
import('./dep.esm');

ghost on 30 Jan 2018

After testing the function with nwjc, it seems that nested imports are not supported... as nwjc --nw-module lib.js lib.bin rises an error when compiling a script containing a dynamic import:

console.log("lib.js started");

import('./dep.js');

That makes it impossible to use in the perspective of an application made of many modules requiring each others and which would have been compiled into binaries.

ghost on 30 Jan 2018

This makes me think that it would be nice to have a loader module in NW.js which would control the way files are loaded via a require or an import call.

This could be a generalization of the binary loader for javascript files, based on extensions.

When we require or import a .js or an .esm file, nwjs would search for a binary file like described in my previous comment according to different parameters defined within the API, for example via functions.

But we could potentially use that same API to create any kind of loaders in nwjs. This could be for example a loader for webassembly files as well, or a loader for files to be transpiled at runtime by exposing hooks into the require and import mechanisms available in node and chromium.

Loader and transform functions would be defined within the API like evalNWBinModule or evalWasmModule which would be called when a require or import call is made with particular file extensions.

nw.loader['js'] = {
   onFetch:findPath, // function to call in order to overwrite the location of the required/importee, returns the path
   onLoad:loadBinaryFile, // function to call with path of importee as argument, returns the source
   onExecute:nw.Window.get().evalNWBinModule //function to call with source of importee as argument
};

nw.loader['json'] = {
   onExecute(src){
      return JSON.parse(src);
  }
};

ghost on 30 Jan 2018

I've already been working on a parallel module loading system based on node's new loader system which hooks into v8's dynamic import and import meta callbacks effectively creating a parallel module system that can be customized indefinitely. While I don't have any working knowledge of evalNW* functions, I am almost certain that it would be possible to support. (FYI: this only applies to scopes where import is supported, i.e. does not work for things like service workers and cannot be used to alter the behaviour of calls to global importScript functions)

My efforts are completely solo at the moment though, if anyone is interested in collaborating to refactor this into an open source project I am very interested. I have managed to limit the required patching to only require building libnode.dylib and made sure that the amount of C++ changes made are limited to a handful SLOC's aside from pulling one or two files from pull requests from the node repo (expected to land in v10).

SMotaal on 30 Jan 2018

reopen to track supporting nested imports.

rogerwang on 31 Jan 2018

For now, I may not use it correctly, but I'm not succeeding to make evalNWBinModule work, even when there is no nested imports, whether it is with dynamic or static import

<!-- index.html -->
<script>
nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js');

import('./lib.js')
</script>

<!-- index.html -->
<script>
nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js');
</script>
<script type='module'>
import './lib.js'
</script>

/* test/lib.js -> compiled to lib.bin */
console.log('lib.js started as binary...');

ghost on 31 Jan 2018

@ smotaal, can't help on C++ personally, but that sounds very neat: the ability to hook into v8's dynamic import callback for loading any kind of customized resources into javascript (including binary snapshots) would be very useful in a project like NW.js.

But how would that relate to nested imports of compiled modules ? My knowledge about snapshots is limited, but as far as I understand them, we can not have a 'require' or 'import' into the top-level scope of the module because the files are not loaded into the heap when nwjc creates the snapshot for individual files.

And nwjc seems to complain about dynamic imports even when they are not in top-level scope (within a function definition for example).

So, the only way to support dynamic nested imports presently seems to have a function into the global scope which would serve as a wrapper for dynamic imports, and within the module the wrapper call would have to not be on the top-level scope:

<!-- index.html -->
<script>
nw.getModule = (path) => {import(path)}
</script>
<script type='module'>
import {useDep} from './lib.js'
</script>

/* test/lib.js -> compiled to lib.bin */
export let useDep = () =>{ nw.getModule('./dep.js')}

ghost on 31 Jan 2018

I just want to point out a difference between static and dynamic imports in V8's implementation:

The following is static:

<script type=module>
import './lib.js'  // expected to load .bin (no .js)
</script>

// lib.bin
import {a} from 'lib-a.js' // also expected to load .bin (no .js)

// lib-a.bin
import {b} from 'lib-b.js' // this one expected to load .js

However, you might not get the same if (the following is dynamic):

// lib.bin
import('lib-a.js').then(({a}) => {})

Here is where V8's isolate->SetHostImportModuleDynamicallyCallback plays part:

https://v8docs.nodesource.com/node-8.9/d5/dda/classv8_1_1_isolate.html#a2b1e8fe982f0b2a8c61d9413cf94c87c

And this is where the discrepancy might be coming from.

My approach basically gives node's loader full control from the get-go, it takes charge of all static imports (I even wired it to tags) and that loader does all nested static imports.

At the same time, I make it register itself as the isolate's DynamicImportCallback. So effectively it hijacks it and processes it using the same logic, ie able to remap it with the same single logic ensuring consistency.

One issue with node's loader is that it was designed for a single "main" context, so it essentially crashed when it tried to access a module across nw frames and that required either redesigning it (no thanks) or finding ways to limit the scope of module-mapping to the contexts for which they are intended (which is 99% javascript code at the moment).

@ab38, like you, I never found my footing in C++ (it's like advanced calculus, which is cool, but no thanks, not my thing) and I basically geek out on finding the path of least resistance and most hackability (ie performance-oriented javascript).

SMotaal on 31 Jan 2018

So it comes down to this:

The previous mind-set that worked for require-oriented modules is not suited for es modules, and though it may be possible to patch the system enough to get it function for certain scenarios, it is like trying to fit the squares through the round holes, they only need to be big enough at the corners to pass (if all you care about is to make them pass). Instead, I believe that the benefits of es modules are far to important to be ignored when they are not handled correctly, not through square holes.

Node's loader provides that, and the best thing is that it is already wired for "require" loading as a bonus.

I have not yet looked into how to add nw's own features into node's loader, my focus was to make it function as a loader that can work in chromium, and it does. It can import('./lib.m.js').then(…), import process from 'process', it can even import x from 'package/src/index.ts' and compile that on demand. It is slowly but carefully adding support for things like import.meta - it works as advertised in upstream PRs - tested. It already has support for node native modules, json, and if you have a require statement, it handles it using the existing module wiring, ie honours any custom wiring like nw.js.

SMotaal on 31 Jan 2018

What I'm wondering, actually, is the following: is it possible to make nested imports (either static either dynamic modules) to work with binary compiled modules via nwjc ?

So that a whole NW.js application made up of separate es modules can be compiled into individual binary module files which would work together when the main module is loaded into NW.js.

Ideally, as a developper, and as stated by @rogerwang, there should not be any change to make into the application code in order to support it.

So, one's best guess would be that NW.js somehow would hook into the import internal mechanism of either node or chromium. And you seem to suggest, @smotaal , that the new node's loader would be the more pertinent to use internally by NW.js to achieve that goal.

Now, how would that connect to binary modules requiring or importing each other ? Can nwjc compile es modules with top-level imports or dynamic imports into binaries which would then be imported at runtime by the application, each binary module being also able to import the others like non compiled es modules?

My point is that perhaps to protect the javascript source code in the context of separate modules is not possible with nwjc snapshots, but I may be wrong, not knowing the internals of how a snapshot is made. But to use a customized node's loader as you pointed out, @smotaal, could also allow to use alternative encryption tools to protect the source code without using the nwjc snapshots (if they do not work in the context of modules).

ghost on 31 Jan 2018

👍1

Again, as just an end-user of NW.js developing js applications, what would be ideal is that NW.js itself loads the compiled binary when an import call (dynamic or static) is made into the application code (if that is at all possible in the context of modules).
javascript nw.loader['js'].enable(); import('./main.js'); // nwjs takes care of loading the binary if present and all its dependencies.
And as a generalized utility, this could be also very useful to have an api to also define custom loaders depending on files extension.

In definitive the 'js' extension would be associated with a predefined loader used by NW.js in charge of loading the individual binaries and connecting them together at runtime. There could be a simple commandline switch activating the loader associating the import call of js files with modules precompiled with nwjc.

ghost on 31 Jan 2018

From my perspective, a wholistic loader system is essential to developer platforms like nw. In my own opinion, browser implementations are mostly preferred, except when it comes to module loading that is adequate for a desktop application due to security restrictions intended for the web, so in this case, node's loader is the way to go.

The problem is that hooks for the dynamic import and import meta callbacks are not context-scoped, they are isolate-scoped, so you cannot pick and choose, if nw decides to move forward by depending on them for one purpose (i.e. to catch evalNW's) then anyone needing to hook to them for a different purpose will either override nw's hooks or if nw is aggressive, may end up with a mess where some hooks work and others don't depending on their luck.

As a developer, I want to decide on the loader that would be hooked. I want to bring my own — did I mention I have an awesome one already working for me :) — my only problem is that I am not equipped (or honestly maybe due to my limited understanding of snapshots not confident enough to be motivated) to explore adding support for snapshots. If someone knows enough and wants to collaborate, and maybe in the process can helping me understand it better from their perspective, it might turn out to be easier to solve than I imagine.

But, please, don't restrict such powerful new additions if you end up using them internally without providing means for developers to properly utilize them if they so choose.

SMotaal on 31 Jan 2018

@ab38 You can most certainly add an encryption layer, just keep in mind that unlike snapshots which are consumed in a post-source state (in theory they are never reversed to source) encrypted modules must be decrypted by the loader, so they are simply hard to hack but not unhackable.

SMotaal on 31 Jan 2018

@smotaal , thanks for the interesting comments.

Another way to look at javascript source code protection could be to use a custom encryption layer associated with a custom loader, both residing into a single snapshot and chosen by the developer, and hooked via a nwjs api to the import call within the application code. The nwjs api would then create at runtime and in-memory individual file-based snapshots or one multi-files snapshot which would then be injected into the v8's runtime engine, and modified at runtime, all from within a single snapshot.

So the snapshot (the nwjs api part of it) would just be in charge to transform decrypted javascript code (coming from disk or elsewhere depending on the developer's logic) into a runtime in-memory snapshot.

Schematically, the single snapshot would take care of the following process:

-> connect to import hook via a nwjs api
-> load encrypted js via custom logic
-> decrypt js via custom logic
-> nwjs api creates a runtime snapshot for the file
-> nwjs api injects it into v8 at runtime

In other words, one snapshot to rule them all... :)

ghost on 31 Jan 2018

Proposal for javascript source code protection compatible with ecmascript modules in NW.js

Prerequisites

The Source Code Protection (SCP) logic should be compatible with ecmascript modules.
It should not interfer with the application code.
It should be optional and customizable
It should take advantage of V8's snapshot mechanism
It should be backward compatible with older NW.js versions

Rationale

In order to maximize source code protection (SCP), developers should be able to use their own encryption logic, because if snapshots can be reversed one day, all nwjs applications would be vulnerable at the same time.
In order to be compatible with ecmascript modules without interfering with the application code, the SCP logic would hook into the import calls of the V8 engine (both static and dynamic).
By hooking into the import call, it could be customizable for different types of files depending on their extensions.
The underlying mechanism would offer a loader utility mechanism to developers, which would serve beyond the mere SCP logic.

Description

In a startup snapshot, optional custom developer logic is created for dealing with the import calls of particular file extensions. The developer logic is in charge to optionally resolve, fetch, load and decrypt js source code. Then the js source code is transformed into a snapshot via the nwjs api, and executed into the V8 engine.

API

/* 
    js content of startup.bin (nwjc dev/startup.js dist/startup.bin) 
*/
function init(){
  nw.enableLoader( 'js', {
    onResolve(importee, importer){
                /*
          optional custom developer logic modifying the path (importee) of the import call
              importee: path of the import call, ie: import('importee')
              importer: path of the module importing the importee
              return importee by default
               */
    },
    onFetch(importee){
                /*
           optional custom developer logic for fetching the source code
               return source code or encrypted source code
                */
    },
    onLoad(source){
                 /*
            optional custom developer logic after fetching the source code
            custom decryption logic goes here (local or remote through authorization,
                    code-signing verification (including the startup.bin), etc.)
                return source code or decrypted source code
                 */
    },
    async onExecute(source){
         /* 
                    optional custom developer logic for executing the source code
            snapshot protection logic goes here via nwjs api 
                 */
          let snapshot = await nw.createSnapshot(source, true); //nw.createSnapshot(source, asModule)
          nw.evalNWBinModule(null, snapshot);
               /* or 
                  let snapshot = await nw.createSnapshot(source);
                  nw.evalNWBin(null, snapshot, true) //nw.evalNWBin(frame, snapshot, asModule)
               */
    }
  }); // enable 'js' loader to hook into import calls
}
init();

ghost on 1 Feb 2018

The issue that I pointed out in https://github.com/nwjs/nw.js/issues/6303#issuecomment-361903249 is caused by reloading the page with "Ctrl + R" which I use to do to test my application code, instead of restarting the nwjs process.

When one reloads the page, evalNWBinModule doesn't work anymore in my tests, one has to restart the nwjs process altogether. Otherwise it seems to work fine as long as there is no nested import into the binary module.

ghost on 1 Feb 2018

My two cents: It'd be better if we can simplify the APIs to be consistent, such as:

Merge evalNWBin and evalNWModule. Let evalNWBin take an optional boolean argument "asModule"
Remove --nw-module argument from "nwjc". Let "nwjc" just compile "foo.js" to "foo.bin", and then let evalNWBin to decide load it as a module or not.

Thus, it could be more consistent to the HTML semantics: One js file could be loaded as a module or not.

Related issues

Shouldn't the logo point north-west?

daverave1212 · 3Comments

How to load extensions automatically in an extension?

azer · 3Comments

Window 'resize' event fires when the window is dragged on hidpi displays

jportoles · 3Comments

webview.addContentScripts seems running in a different context

mmommo · 4Comments

registerGlobalHotKey and alert

xland · 3Comments