Emscripten: Moving towards a more traditional sysroot from emcc

[sbc100]

Today we have a kind of sysroot in the form of emscripten/system. It contains the following sub directories:

• lib
• bin
• include
• local/include

emcc uses include and local/include as default include paths, but it doesn't use lib as a library path. Indeed lib currently contains the source code to the libraries and the libraries end up ~/.emscripten_cache.

In the future I'd like to see this act more like a traditional sysroot with target libraries, includes and binaries all living there.

For emsdk users we could pre-populate this directory with ports and system libs.

[dschuff]

Continuing from #9286 regarding the right way to handle ports, which is related to this issue.

I theory even right now one would just be able to run the makefile of the library to produce a static library which you could then just drop in a /usr part of a sysroot or add with -L but that's not as easy right now as it could be since the lib directory isn't organized like a sysroot or documented AFAIK (as mentioned above).
So in principle we wouldn't need embuilder at all, but it is convenient to have it as a utility to just fetch and build a port. Having a stable ABI means that we wouldn't have to rebuild after every upgrade either (although we are still at a state of maturity where we'd probably want to recommend rebuilds anyway). Offering users control over whether build/rebuild-on-demand happens or whether to use an existing lib would be good too, so it probably makes sense to separate "use this port" from "build this port".

Maybe in the fullness of time, a situation where
1) we have a more traditional sysroot, (perhaps with a /usr for ports?)
2) the expected way to use a port is just to link the library
3) A tool like embuilder or similar exists which knows how to build certain ports (and perhaps could even customize them e.g. with different target features), and puts them in the same place that a user building a new or proprietary port would
3a) This mean we wouldn't need a flag to build ports on-demand. But we could continue to support -S USE_FOO for existing FOOs for either on-demand building or linking for compatibility if we wanted.

[sbc100]

Yes, I basically agree with all of that, although I think don't see is being able to share a single sysroot betweem emscripten versions any time soon. i.e. I think enforcing an ABI would be too much of a burden.

In terms of (3) I think we probably want to keep compatability but the basic difference would be between

Old way:

• emcc -s USE_ZLIB

New way (traditional way):

• ./embuilder build zlib
• emcc -lz

As with other transition we can continue to support the old way for a certain number of years (unless in STRICT mode).

[gabrielcuvillier]

On the same idea, there is #7633, where I suggested to update the cmake toolchain file to automatically include system/local as a sysroot for 3rd party libraries. This is very convenient to have this, as 3rd party libraries compiled for Emscripten (ie. not coming from Emscripten ports) can be put there and found with find_package.

[embeddedt]

it probably makes sense to separate "use this port" from "build this port".

+1 for this. I depend on that behavior for the MicroPython Emscripten port, which requires me to invoke clang manually in order to precompile some headers. In order to invoke clang the library headers already need to be in place so embuilder is a convenient way to ensure that.

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.

[kripken]

I think we still want this. Current progress is being made on https://github.com/WebAssembly/binaryen/issues/3043 which will simplify linking a lot and should make this much easier to achieve.