Emscripten: Add support for simultaneously using dynamic linking + pthreads.

[juj]

Current pthreads implementation has a limitation that the dynamic linking feature cannot be used with it. Investigate what is needed to enable this.

[kripken]

I would hope that this would just work. Did you see a problem when trying it?

[juj]

I did not try, this was motivated by your comment in the pull, will need a bit of an audit and testing.

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past 2 years. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

[jbkempf]

Any update on this?

[kripken]

I don't believe anyone is working on it. It would require some design thinking, as say dlopen on one thread would require adding to the Table on all other threads. However, non-dlopen dynamic linking may be a lot easier.

[jbkempf]

What about dlopening all from the "main-thread"?

[jbkempf]

However, non-dlopen dynamic linking may be a lot easier.

That would mean to have all the .so _(is it really .so?)_ modules loaded all the time, no?

[msabwat]

@sbc100 talked about a shared table instead, would it be feasible? the SIDE_MODULE would still expose its native table to JS (like a normal wasm module does). I don't see how emscripten reads the native table? is it here?

For now, "agressively" disabling the errors when we USE_PTHREADS+SIDE_MODULE lands into a wasm-ld assert:

/b/s/w/ir/cache/builder/emscripten-releases/llvm-project/lld/wasm/Writer.cpp:715: void lld::wasm::(anonymous namespace)::Writer::createInitMemoryFunction(): Assertion `WasmSym::initMemoryFlag' failed.

Makes me think there's another write. Is it like that? :

1. We extract the symbols from the binary. (with llvm-nm)
2. We write them into a JS Table object.
3. We give that table to wasm-ld to have the final .wasm binary.

Dynamic linking in our case would be to be able to feed the JS table from SIDE_MODULES and read from it from the MAIN_MODULE.

[sbc100]

Current there is no concept of a shared wasm table in the spec, so each thread necessarily has its own table. And the tables need to stay in sync because C/C++ code expects to be able the share function pointer between threads.

Right now we really on the startup code on each thread insantiating the wasm module and the local table such that local table on each thread ends up with the same content.

Each time you load a DLL new table entries get add, so these table entries (and the DLL itself) need to propagated to each thread). If it can be done deterministicly then as perhaps just loading the DLL on each thread would be enough to guarantee the same table contents?

[sbc100]

I'm curious to know that your use cases for dynamic linking are there? Are you sure you need it? If its all happening at startup for example then its almost always better to produce a single module (at least today).

[brion]

For vlc.js I believe the requirement is to avoid pre-downloading megabytes of code for supporting file formats and codecs that mostly won't be used on any given run.

Otherwise you either have a large initial download that's wasted by bundling all code together, or you have to make custom builds targeting just the formats that will be used on a given data set.

With dynamic linking you get the flexibility of supporting many formats without having to ship unused code over the wire.

[msabwat]

We are trying to port VLC, which is built around the idea that modules can be loaded at runtime. As long as this is not implemented, it will always be less performant with this target. I am very interested in trying to make it work as is.

[sbc100]

I see. Loading plugins dynamically is indeed a valid use case :) In fact I think its pretty much the only one that exists on the web today.

Presumably if you know the content type of the thing you wanted to play you could be a static build with just say, mpeg4+aac support? But then you wouldn't have a universal player. However I imagine it might be useful for a given site if they know the format of all their content. I might be worth the refactoring effort in your build system to be able to produce such tailored binaries?

[jbkempf]

Loading plugins dynamically is indeed a valid use case :) In fact I think its pretty much the only one that exists on the web today.

Yes. That will avoid using 30MB of binary (or more)

Presumably if you know the content type of the thing you wanted to play

De facto, you never know. There is no such thing as AAC, for example.
And you don't know what the machine is capable of, when accessing hw codecs.

For example, I have a build for DVD, but we want something universal, but more importantly, extensible, from the webpage.

So, I think we have a usecase here :D However, we don't need to dlopen from any thread. It could be done always from the same one.

[sbc100]

De facto, you never know. There is no such thing as AAC, for example.
And you don't know what the machine is capable of, when accessing hw codecs.

I was thinking if you were running a website the hosted videos one could imagine that they all might be in a known format. Maybe not a common use case, but perhaps this is true for someone like wikipedia?

I don't know why hw codecs have any bearing here since wasm by definition can't access such things.

That said, I do understand and sympathize with your use case now.

[jbkempf]

I was thinking if you were running a website the hosted videos one could imagine that they all might
be in a known format.

Usually, people think they do know. And they don't :D

I don't know why hw codecs have any bearing here since wasm by definition can't access such things.

WebCodec is being discussed right now. I do not see why wasm couldn't access to such things.

That said, I do understand and sympathize with your use case now.

Thanks :)

[hortonelectric]

+1

[rth]

Loading plugins dynamically is indeed a valid use case :) In fact I think its pretty much the only one that exists on the web today.

pyodide is another one. If there are multiple python modules available with C-extensions we don't want to load all of them at the same time, and threading is used quite a bit in Python. Though a blocker for threading is currently the lack of Firefox support by default in any case https://github.com/iodide-project/pyodide/issues/237