Emscripten: Loading extra files like the `.wasm`

[kripken]

Consider a library like ammo.js or box2d.js, where a site can have structure like this:
ammo.js demos/ demo1/ index.html demo2/ index.html
Both html files can have <script src="../../ammo.js"></script>. That way, we have one copy of ammo.js, used by multiple locations, and this just works. And likewise, if the code is run in a worker, the worker can do `importScripts("../../ammo.js");

If, on the other hand, there is an extra file like ammo.js.mem or ammo.wasm, that is loaded by ammo.js, then things are trickier. If ammo.js just loads the URL "ammo.wasm" then it won't find it - the current directory is demos/demoX/. There hasn't been a great solution to this, so to work around it we avoided such extra files, disabling the .mem file in particular - despite the downsides of doing so (larger code size, slower startup).

But with wasm this is unavoidable - we have to have another file.

Some options:

• Modify the URL inside ammo.js, either to an absolute path like "http://mysite.com/ammo.wasm" or the proper relative one "../../ammo.wasm". The relative path won't work if it can be called from different nesting depths, so really only the absolute one seems like an option here. But it seems bad to ask users to do this, and do it every time they move the file around...
• Auto-detect the proper relative path. This seems possible using document.currentScript.src, but this is not available in a web worker. Another issue is that the code may not be in a normal script tag if it is evaled (but that shouldn't be done, so maybe we can ignore it).

Maybe there's a better way to do this?

[curiousdannii]

Auto-detect the proper relative path. This seems possible using document.currentScript.src, but this is not available in a web worker. Another issue is that the code may not be in a normal script tag if it is evaled (but that shouldn't be done, so maybe we can ignore it).

Another messy situation is when the script is loaded with require() in Electron. I'm not sure how that's implemented, perhaps even through eval(), but it's less obviously bad practice.

Maybe there's a better way to do this?

For me, the ideal is to be able to pass in a path at my leisure. I've done this by making a dummy XHR for memoryInitializerRequest to delay loading, and then calling doBrowserLoad myself later. But now I need to do the same for emterpreter data and it's not going to be nearly so easy.

Edit: It doesn't look like it's really possible to delay loading the emterpreter data. Perhaps if I set Module.emterpreterFile to a dummy ArrayBuffer and then manually copied the code in the __ATPRERUN__ function which handles the relocations, but I would have to manually keep it up to date. So for now no external emterpreter file.

What I would love is a compile setting which made it not attempt to load anything, like Module.noInitialRun except all encompassing, and then if there was a single function to call with a path to load the mem, asm, emterpreter data.

If #5085 was implemented and the path could be passed in to the initial Module() call then that would probably work really well too - I would change my code to not add my API functions directly to Module but as a separate object, and only call Module() when I'm ready to.

[drola]

In theory it is possible to embed .wasm binary data to .js file by using something like base64 encoding. There is a size and conversion tradeoff though:

792K    wasm_bullet.wasm
320K    wasm_bullet.wasm.gz
1,1M    wasm_bullet.wasm.base64
496K    wasm_bullet.wasm.base64.gz

[dschuff]

Once browsers implement the response-based embedding API (
https://github.com/WebAssembly/design/blob/master/Web.md#additional-web-embedding-api)
we will really want to use that because it will allow streaming compilation
(i.e. overlapping wasm compilation with download) and implicit caching. So
trying to go back to a world where we stuff everything into one synchronous
script isn't what we want. Obviously there are issues around asynchrony and
URLs etc, but tackling them head-on will buy us a lot of nice things. (and
we can use the same approach for memory files, dynamic libraries, and even
user files such as game assets).

On Sun, Apr 2, 2017 at 12:18 AM Matjaž Drolc [e-mail]
wrote:

In theory it is possible to embed .wasm binary data to .js file by using
something like base64 encoding. There is a size and conversion tradeoff
though:

792K wasm_bullet.wasm
320K wasm_bullet.wasm.gz
1,1M wasm_bullet.wasm.base64
496K wasm_bullet.wasm.base64.gz

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kripken/emscripten/issues/5104#issuecomment-290969509,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABEiKIeVAleTLmgKzGx1upaCWshPErhQks5rr0u9gaJpZM4MwOZE
.

[kripken]

I noticed we don't support the locateFile API for the wasm binary. That lets you customize where e.g. the mem init file is. Seems like we should do that, which helps here, but is still not great as you need to create a locateFile method.

[kripken]

@mbebenita reminds me that setting Module.wasmBinaryFile to a custom path works, but we should still support locateFile for full flexibility.

[buu700]

+1 re: an option to embed everything inline — including the asm.js fallback where applicable. It'll be a lot easier to update my existing asm.js libraries to wasm if I don't have to implement bundling them into single js files on my own.

Base64 sounds like the best solution to me, and after Brotli compression it doesn't really have enough of an adverse impact to matter when testing with one of my libs (ntru.js):

82K ntru.wasm
16K ntru.wasm.br
110K    ntru.wasm.base64
25K ntru.wasm.base64.br

Edit: As to why embedding inline is a requirement in my use case, we use an in-browser code signing scheme that would preclude allowing the emscripten runtime to arbitrarily download and execute remote subresources; the next best alternative to base64 inlining (or equivalent) would be to continue using asm.js indefinitely. More generally speaking, I think it makes the most sense for emscripten's wasm implementation to remain fully backwards compatible with every use case that asm.js currently addresses except where breaking compatibility can't be avoided.

[kkimdev]

So is this considered resolved with SINGLE_FILE and locateFile options? Or is there anything more to do?

[curiousdannii]

Lots more to do. It's far from ideal IMO. The Emterpreter data must still be manually loaded for example.

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.