Parcel: WASM loader does not expose importObject

[jsheard]

WebAssembly modules are able to import foreign Javascript functions and memory objects that are passed in the second parameter of WebAssembly.instantiate(...). Unfortunately the current WASM loader is hard-coded to call instantiate with no import object.

https://github.com/parcel-bundler/parcel/blob/master/src/builtins/loaders/wasm-loader.js
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WebAssembly/instantiate

The simplest solution would be to have the WASM loader instead run WebAssembly.compile(...) and return the compiled WebAssembly.Module, so the user can call WebAssembly.instantiate(...) with whatever parameters they need. I think that's how Webpacks wasm loader works.

[devongovett]

Yep. I kinda like the way webpack is planning to approach this (I think): imports in WASM files get mapped to module imports the same way imports in JS do. The problem is that the current Rust wasm backend always generates only a single import, and it's always called "env". Ideally it would be possible to specify the name of this import so we could map it to a JS file. Maybe somehow as part of the extern syntax?

For example:

#[wasm_module = "./externs.js"]
extern {
  fn test1();
}

The env name appears to be hardcoded in binaryen which is what Rust uses to do the compilation to wasm. This issue seemed somewhat related: https://github.com/WebAssembly/binaryen/issues/943

cc @linclark @kripken @sokra @TheLarkInn

[linclark]

Yep. I kinda like the way webpack is planning to approach this (I think): imports in WASM files get mapped to module imports the same way imports in JS do.

You're correct, this is the right approach. WASM modules will eventually be ES modules, so imports/exports should work the same as they do in JS. I will likely be championing this over the next few months. Work will be tracked in https://github.com/WebAssembly/design/issues/1087

The problem is that the current Rust wasm backend always generates only a single import, and it's always called "env"

The webpack folks also brought this up. If no one responds with a plan to address this, I'll bring it up in our biweekly WASM toolchain meeting (which is next Tuesday) and see if I can figure out what the plan is.

[kazimuth]

In the shorter term, https://github.com/koute/stdweb/issues/93 would allow rust code using the stdweb library to do the following:

#[no_mangle]
pub fn call_js_function(x: i32) {
    js! {
        var js_function = require("./code.js").js_function;
        js_function( @{x} );
    }
}

The js! macro generates javascript code with rust shims to access it. The javascript code can then be fed to Parcel to resolve imports.

It won't be possible to implement this without access to instantiate, though.

[koute]

I've made an experimental (and really, really hacky) Parcel plugin here to compile Rust to WebAssembly using cargo-web. Compiling through it one can use the js! macro from stdweb to call into the JavaScript from Rust. (And in the next few days I plan to push a js_export! macro which will make it possible to export arbitrary functions from Rust, not just those using WASM-native types.) Unfortunately I had to be somewhat... creative to make that plugin work, since as far as I can see Parcel doesn't allow generating dynamic bundle loaders.

Even if WASM modules will eventually be treated as ES modules it would make sense to support customizing the imports until that future comes (or at least make it possible to do it from a plugin).

[mischnic]

This is also required for emscripten wasm support.

At first I tried to register a custom BundleLoader, but bundle loaders are registered by file extension.

bundler.addBundleLoader('emscripten',require.resolve('./EmscriptenLoader'));

Would it be possible to add an option to specify a bundle loader in an Asset or Packager?

[alexcrichton]

The problem is that the current Rust wasm backend always generates only a single import, and it's always called "env"

@devongovett, @linclark FWIW y'all probably already found this, but on the Rust side of things this is tracked at https://github.com/rust-lang-nursery/rust-wasm/issues/29 where we very much don't want to have anything using env but instead have dedicated syntax like @devongovett mentioned for naming the module.

[mvlabat]

@devongovett, can we expect any progress on this?
Speaking of Rust side, https://github.com/rustwasm/team/issues/29 has been solved

[0xcaff]

Any update on this? I'm not sure how to use rust + wasm-bindgen with parcel and typescript. wasm-bindgen relies on imports being passed to the WebAssembly module. Circumventing the bundler and loading the wasm bundle at runtime doesn't seem possible without writing a parcel plugin to do something about the static files.

[zakcutner]

It would be really nice to have support for importObject – a killer feature of WASM is the ability to call JS functions from WASM modules (as well as the other way round of course)!

I feel that you shouldn't have to compile WASM modules already knowing the paths of JS files they use. It would be nice to simply have them as standalone binary modules to which you can pass in imports they need to run (in keeping with the JS WebAssembly.instantiateStreaming() API).

Besides, this would also open up support for wasm_bindgen as well as other existing WASM compilers. @hitecherik and I thought perhaps a syntax like this would be preferable (and compatible with the current import syntax)...

const module = await import("module.wasm")(importObject);
module.foo();

[mysterycommand]

I created a plugin to try and do just this based on @catsigma's parcel-plugin-wasm.rs. In order to get it working, I used wasm-pack's --target no-modules and then modified that output based on the Parcel environment.

The trouble is, this approach uses that generated module as a "bundle loader" which are used by Parcel _per file type/extension_. This means that if I have two Rust entry points, one of those is going to be loaded with an imports object that doesn't match the generated wasm module's public API (unless they happen to expose the exact same API).

I created an issue about this against my plugin. I would love to come up with a way to treat wasm modules more like "regular" modules and be able to use more than one Rust library/entry in a project.

My current thinking is to have my plugin delete bundler.bundleLoaders.wasm; and handle the async loading/instantiation/imports object itself on a per Rust entry basis, but I'm not sure what the right dependency structure even looks like. The output of wasm-pack with --target bundler (the default) is a js file is like (just as an example):

import * as wasm from './example_bg';

/**
 * @returns {void}
 */
export function run() {
  wasm.run();
}

// ... a bunch of wasm bindgen stuff ...

I _think_ what I should be able to do is have my custom asset plugin emit a loader based on the --target provided to Parcel that imports the wasm-bindgen generated module above and uses that as the imports object during instantiation, like this:

import * as __exports from './example';
let wasm;

function init(module) {
  let result;
  const imports = { './example': __exports };
  // like what gets emitted by `wasm-pack --target web` 
  // except the __exports object is just the imported / bindgen-generated module
}

export default init;

I guess here's where I kind of stop being able to track what should happen. What do I so with the import * as wasm from './example_bg'; in the bindgen-generated module? In --target nodejs the bingen module starts out var wasm and ends with wasm = require('./example_bg'); where example_bg.js is what does const bytes = require('fs').readFileSync(path);, but then this module circularly requires the bindgen module with like: imports['./example'] = require('./example'); … how does that even work? Also, since it appears that the answer is "it works just fine", can we do something like that for the web/fetch/instantiateStreaming case?

I'm interested in doing the work, and have some time available to do it. Any help/guidance would be greatly appreciated.

[mysterycommand]

Hi, I'm back. This plugin that I worked up: parcel-plugin-wasm-pack now does the thing it should … I think. I have a couple ideas that might make it a bit cleaner, but for now it works with --target browser for sure. It has a couple issues but here's how it works:

1. Deletes the default bundler.bundleLoaders.wasm loader
2. Runs wasm-pack build --[release|dev] --target web on the directory that contains the Cargo.toml (--release or --dev based on options.production)
3. Adds the generated *_bg.wasm and *.js (the module containing the generated importsObject and initialization code) as dependencies to the bundle
   
   
   
   • one thing that I'm hoping to clean up is that I had to mangle the wasm-pack generated initializer to resolve to the __exports object which is what gets used in the "JS-land" of the bundle
   
   
4. Generates a module that depends on the path to the wasm, calls the initializer with that, and exports the promise
5. A packager that extends the JSPackager and only runs if the asset it's processing has the isWasm flag (that gets set by the WasmPackAsset) then generates it's own bundle loader module that looks like this:
   js const moduleTpl = (wasmName, wasmId, entryName) => `\ require('${wasmName}') .then(wasm => { // replace the entry in the cache with the loaded wasm module module.bundle.cache['${wasmId}'] = null; module.bundle.register('${wasmId}', wasm); }) .then(() => { require('${entryName}'); }); `;
   
   
   
   • this lets the entry depend on the wasm-pack generated JS module, but get the initialized wasm module interface … kind of a switcheroo, but it works and now that I see how to wire it up I think I can refactor toward something a little clearer