Binaryen: wasm2js / wasm2asm planning

[kripken]

This issue lays out a possible plan for our wasm2asm tool, based on previous discussions. The big picture:

• The goal is to convert an arbitrary wasm file to a JS file that behaves exactly like it. You just run wasm2js input.wasm > output.js and the JS contains a JS object that looks just like that wasm module.
• We should rename the tool "wasm2js" as "wasm2asm" (which was meant to imply asm.js) is confusing. For one thing, we can't do all we want in asm.js given all the features wasm has, and for another, we do want all the extra JS code that makes the output look exactly like the original wasm. Also, "asm" could be taken as native assembly code, which is what wasm is compiled to normally... ;)
• There may be limitations, like NaN canonicalization, which we can't do in JS in a fast way. May just document these.

The plan to stability:

• Get the wasm-bindgen test suite passing with wasm2js.
• Get the spec tests (or as many as we can, ignoring NaN canonicalization etc.) passing with wasm2js.
• Get the emscripten test suite passing with wasm2js.
• Run the binaryen fuzzer on wasm2js (with NaNs disabled).

Once all those are done, we should be in pretty good shape!

Thoughts?

[fitzgen]

Sounds great to me! Thanks for writing this up.

Only missing bit I can see, which is maybe an optional nice-to-have, would be differential fuzzing between another wasm implementation and the output of wasm2js.

[kripken]

Ah I should have been more clear, that's what the fuzzing at the end is meant to be: the binaryen fuzzer emits random programs and we'd run them in multiple VMs + through wasm2js, checking for any difference in the output.

[fitzgen]

Ah, ok! I think it is also valuable just to run random wasm programs through wasm2js as well, without the differential step :)

[kripken]

Oh, yes, definitely :)

[dschuff]

+1, sounds like a good plan.

[dschuff]

Another reason for "wasm2js" is that I'll bet all the engines that had good asmjs optimizations now also support wasm, so I guess the real target for this is older browsers that don't have good asmjs or wasm support.

[alexcrichton]

Sounds like a great plan to me as well! In wasm-bindgen I've got a was2es6js tool which currently supports a --wasm2asm flag which uses wasm2asm under the hood to replace a wasm file with an ES6 module. Some things I've learned from that which we'll want to do here:

• We'll want to initialize the contents of memory with the data segments of a wasm module. Currently I've just got it storing base64 strings and then decoding them on the fly to initialize the main memory chunk.
• At least what wasm-bindgen has been doing is emitting ES module output by default, but we'd probably also want a flag to not do that. For example wasm-bindgen has a --no-modules flag which is intended for directly loading in a browser (no packaging) and otherwise just disallows importing anything. I'm not sure the exact same would work for wasm2js but it may be worth thinking about the import story for wasm2js, e.g. how are we emitting JS with those imports? (maybe like a --amd flag? --commonjs?)
• We'll want to add a new pass which renames all imported functions in a wasm module. Right wasm2asm would choke if you have, for example, (import "foo" "bar") as well as (import "baz" "bar") (aka the same name from two different modules). It should be fine to handle this all transparently though by renaming everything to a unique name and then reassembling the mapping from the imported items.
• As a bit of a usability improvement I think we'll want to accept raw wasm input instead of only wast input, but that's a pretty minor addition!

FWIW I've found that the spec tests are quickly reaching diminishing returns. Most things are working pretty well and most of the remaining spec tests are exercising "interesting" aspects of the harness they're expected to be run under (multiple modules, etc). After https://github.com/WebAssembly/binaryen/pull/1558 I think it might be worthwhile to dive straight into the emscripten test suite or fuzzing without trying to get too many more spec tests working, although I could be wrong!

[alexcrichton]

Also FWIW with #1558 applied as well as https://github.com/WebAssembly/binaryen/issues/1562 implemented I believe wasm-bindgen's entire test suite is passing running through wasm2asm instead of through WebAssembly in node. Additionally I've now taken https://github.com/Hywan/gutenberg-parser-rs as-is and run it through wasm2asm and it works like a charm, so wasm2asm is definitely approaching "pretty ready"!

[kripken]

Agreed that the order doesn't need to be what was listed above. Once we have enough to run the fuzzer, no reason not to do that immediately, etc.

Regarding emitting a module or not, one thought I had was not to emit a wasm or ES6 module for just the wasm, but rather emit a complete polyfill for wasm itself. That is, define the WebAssembly object and polyfills of its compilation methods etc. This seems the easiest for integrating into code that loads and uses a module (like for emscripten output), but maybe not for other use cases?

[alexcrichton]

Hm perhaps yeah! Although I guess it'd be like a oneshot thing where it'd always just "compile" the same module?

I've found that the ES module boundary is at least a pretty easy one to polyfill at, but it definitely means that if you're using wasm2js then you'd have to have some sort of other bundler in play. Most of the time for production use cases that's already the case though?

[binji]

Most things are working pretty well and most of the remaining spec tests are exercising "interesting" aspects of the harness they're expected to be run under (multiple modules, etc)

None of the spec tests should be testing behavior of the harness, but some of them do require multiple modules to test proper linking behavior, validation, etc.

[alexcrichton]

@binji ah sorry yeah that's what I mean. In the sense that those things aren't too interesting in terms of the feature set of wasm2asm specifically :)

[binji]

I guess it depends on how you're planning to use wasm2asm/wasm2js. If it's traditional emscripten-style where you're converting a monolithic app to JS, then yeah, probably doesn't matter. But in the future I imagine we'll see a lot more wasm module components, so the proper linking behavior will be important.

[kripken]

@binji Agreed. So maybe wasm2js could compile into something that looks like a wasm Module, and we'd also have a separate JS file which is a polyfill for the WebAssembly object + APIs, and it would know how to use those objects, and eventually also how to use normal wasm objects and even link them.

[kripken]

When I say "module-like" I actually mean more "binary-like", in that it would be an input to a WebAssembly.compile* API.

[Hywan]

Additionally I've now taken Hywan/gutenberg-parser-rs as-is and run it through wasm2asm and it works like a charm, so wasm2asm is definitely approaching "pretty ready"!

❤️

[kripken]

Thinking about optimization in the PRs, it may still be useful to keep an "inner" asm.js module that validates. That would let us run an asm.js optimizer on it. That would mean leaving the other non-asm.js code on the outside. Not sure if it's practical, though.

If we can't do that, then we need to figure out something else for optimization. Closure compiler and others can do a good job on general code shrinking but may not be as good as an asm.js optimizer for what we emit here.