Emscripten: wasm-sourcemaps.py is slow and memory inefficient

[walkingeyerobot]

tools/wasm-sourcemap.py is very inefficient about its memory usage.

The output of dwarfdump can easily reach dozens of GBs of memory. Converting this output to a string and then running regular expressions on it is causing my machine to run out of memory.

Making use of something like elfutils (https://sourceware.org/elfutils/) would likely be better than doing the processing by hand.

[kripken]

cc @yurydelendik

[yurydelendik]

FWIW there is https://github.com/yurydelendik/dwarf-to-json project (published to https://www.npmjs.com/package/dwarf-to-json)

[kripken]

Thanks @yurydelendik!

Thinking on this, maybe the simplest thing in Emscripten is to treat that like we do Closure Compiler: we bundle the binary in the repo. In this case we can compile the rust code to a wasm binary and add the JS to load and run it, and then we can run it on node.js which is already a dependency of ours.

Thoughts?

[jgravelle-google]

In this case we can compile the rust code to a wasm binary and add the JS to load and run it, and then we can run it on node.js

This is the answer to everything.

[yurydelendik]

the rust code to a wasm binary and add the JS to load and run it

FWIW the npm package already has that.

Some of the wasm-sourcemaps.py functionality is missing (e.g. re-writing wasm file), but it will be easy to add in the JS code.

[kripken]

@yurydelendik good to know, thanks!

Do you perhaps have time to add it here? As you wrote the new code and the old code being replaced, it would be easiest for you I'd guess.

[kripken]

ping @yurydelendik , is this something you could help out with?

[kripken]

ping @yurydelendik - any updates here? We have a bunch of users that are waiting on it.

[yurydelendik]

any updates here

There is https://github.com/emscripten-core/emscripten/pull/9311 that migrates all logic, but I stuck on setting up the testing infra for emscripten locally.

[kripken]

Thanks @yurydelendik, looks like @sbc100 is reviewing that, maybe there was a miscommunication there.

[sbc100]

Perhaps #9580 is enough? Lets see how it fairs dealing with out internal OOM issues.

[dschuff]

I tested #9580 against our large internal app and it at least successfully generates the source map in a reasonable amount of time and memory. There's also a large external app that ran into this, I can ask to test it with.

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.