Emscripten: Is there a GLIB emscripten port available ?

[o-micron]

Hi, I had some code depending on glib and I went to build glib from source to find out it is a rabbit hole.
glib is using meson build system, no cmake or makefiles out there ...
6 hours later, it turns out the problem is building libffi for emscripten ...
Am I missing something ? Maybe there is a glib port available ?

[sbc100]

I'm not aware of any glib port I'm afraid. I imagine porting libffi might be tricky as IIRC it involves assembly code. Perhaps there is way to remove that dependency.

Feel free to add a port to https://github.com/emscripten-ports if you manage to get it working.

[kleisauke]

I had a similar problem while porting libvips to Emscripten (see https://github.com/libvips/libvips/issues/192 for details). GLib and libffi currently require patches to successfully build them with Emscripten. Could you try these?:
https://gist.github.com/kleisauke/acfa1c09522705efa5eb0541d2d00887

These patches were copied from my local Git repository and tested with libffi 3.3 and GLib 2.64.2. Most of the libffi work has already been done by @brion (thanks!), see: https://github.com/brion/libffi/commits/emscripten-work

I just made it compatible with the latest Emscripten (by swapping Module['usingWasm'] with Module['wasmMemory'] and converting the EM_ASM_ to EM_JS).

Anyways, I'll eventually try to incorporate these patches upstream.

[kripken]

That sounds great @kleisauke !

If there are issues getting them upstream, adding a port in emscripten-ports is also an option.

[o-micron]

thanks @kleisauke for your patches

I tried them on ubuntu 20.04 with :

• meson 0.54.1
• GLib 2.64.2
• libffi (the meson subproject master branch), not the brion's branch, because brion's branch doesn't contain a meson.build file which is a dependency for the main glib meson build script

Things look a bit better but I keep getting the error
ERROR: Problem encountered: Unsupported pair: system "emscripten", cpu family "wasm32"

I have applied your emscripten cross file --cross-file ./emscripten-crossfile.txt which is also what meson suggests for cross compiling as stated here starting a cross build

wasm32 is the right cpu family and not x86 of course ... but wasm32 seems unsupported
Meson is updated and so that shouldn't be the problem ?

I am exploring multiple issues at the same time I guess :blush:

[kleisauke]

@o-micron I've updated the gist with an example build script. Let me know if that works for you.

[o-micron]

thanks @kleisauke that definitely works ! :100:

Also for anyone coming from the future for the same issue, here are some packages needed as well

# required tools ...
sudo apt install autoconf libtool m4 automake

# install meson from pip3
pip3 install meson
export PATH=~/.local/bin/:$PATH

# make sure you have emcc and wasm-ld in path
cd emsdk
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh
export PATH=$(pwd)/upstream/bin/:$PATH

# download the gist https://gist.github.com/kleisauke/acfa1c09522705efa5eb0541d2d00887
# extract gist .zip and cd to that folder 
# now run ./build.sh and you should be good to go

[kripken]

Great!

If someone is interested to add this to emscripten-ports let me know - I can create a repo there and give you access. Would be good to do that since this has come up more than once.

[o-micron]

@kripken I would definitely help adding that to emscripten-ports but I would rather let @kleisauke do it and take the credit for his work

I have added above the simple steps I took to successfully build it on ubuntu

[o-micron]

@kripken Here is a script which does everything really ... could replace the whole port repository

Tested on Ubuntu 20.04

# required tools ...
sudo apt install autoconf libtool m4 automake

# install meson from pip3 because glib uses meson ..
pip3 install meson
export PATH=~/.local/bin/:$PATH

# Get the emsdk repo
git clone https://github.com/emscripten-core/emsdk.git

# Enter that directory
cd emsdk
# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file)
./emsdk activate latest

# Activate PATH and other environment variables in the current terminal
. ./emsdk_env.sh

# Make sure we have mainly wasm-ld and other tools in path as well
export PATH=$(pwd)/upstream/bin/:$PATH

cd ..

# Get @kleisauke patches
git clone https://gist.github.com/acfa1c09522705efa5eb0541d2d00887.git
cd acfa1c09522705efa5eb0541d2d00887

chmod +x ./build.sh
./build.sh

[kripken]

Nice @o-micron, yeah, if we don't need a separate repo then this is quite easy. That script could be converted into a python script and put under tools/ports/glib.py basically, and integrated into ports. I think that would be great to do!

[sbc100]

For a long time we (I) have been trying to come up with a place to put "contrib" ports like this. Simply adding everything and the kitchen sink to "tools/ports" doesn't scale very well. I don't have an answer right now.. only the question.

[kleisauke]

For a long time we (I) have been trying to come up with a place to put "contrib" ports like this. Simply adding everything and the kitchen sink to "tools/ports" doesn't scale very well.

The meson build system solves this problem with the Wrap dependency system.
https://mesonbuild.com/Wrap-dependency-system-manual.html
https://github.com/mesonbuild/meson/blob/master/mesonbuild/wrap/wrap.py

Perhaps we could do something similar with Emscripten ports? For example, port files could look like this:

Details

[port-file]
directory = glib-2.64.2

source_url = https://download.gnome.org/sources/glib/2.64/glib-2.64.2.tar.xz
source_filename = glib-2.64.2.tar.xz
source_hash = 9a2f21ed8f13b9303399de13a0252b7cbcede593d26971378ec6cb90e87f2277

patch_url = https://gist.github.com/kleisauke/acfa1c09522705efa5eb0541d2d00887/raw/37e3787a16cef2b8a1b577d469406ebbb80147d7/glib-emscripten.patch
patch_filename = glib-emscripten.patch
patch_hash = bb46e867ce457d4c34551968303ffde520b5f17a54df69653e11677b7df9cfc5

[port-git]
url = https://github.com/brion/libffi.git
revision = emscripten-work

^{(This is heavily inspired from Meson Wrap files)}

The advantage of this is that updating ports can then be done without much effort, since you don't have to embed an entire project into the source tree (i.e. tests/third_party) or to maintain separate forks (i.e. the emscripten-ports organization). Any Emscripten specific patches can then be easier integrated upstream.

This is only for the source part, I'm not sure how the actual build of these ports might look like (because it can be very specific to users' needs).

[sbc100]

The immediate problem I'd like to solve is how to add ports without adding:

1. New settings such as USE_ZLIB
2. Load on the CI system (its fine if we have a separate builder that build the universe but we don't want to block evey change on this).

There are lots good ports systems out there. Its not clear if we want to block this work on choosing one and switching to it, or sticking with our existing one. Part of me would like to do all of this out-of-tree using something like pacman (from arch) the freebsd ports system: https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-using.html. But as an incremental step I think we could just use tools/ports/contrib with out existing ports system. I think the plan was to have a generic -s USE_PORT=foo setting to avoid adding new settings for these. Also, embuilder would not build these by default. I think is convered by an existing bug but I can't seem to find it right now.

[sbc100]

At least one related bug: https://github.com/emscripten-core/emscripten/issues/10048

[kripken]

The immediate problem I'd like to solve is how to add ports without adding:

• New settings such as USE_ZLIB

I think we can get rid of that boilerplate by automatically adding a setting for each port. So adding a port would mean just adding a single file under tools/ports/ but no changes to src/settings.js for example. Does that address your concern, or even so would you be unhappy?

• Load on the CI system (its fine if we have a separate builder that build the universe but we don't want to block evey change on this).

I think we don't need to test every thing in ports/ on our CI. Newly added ports can be done with no new testing.

[sbc100]

Yes I pretty much agree with that. I do think that using a subdirectory would useful to indicate to two distinct and separate types of ports. I like tools/ports/contrib myself.

[sbc100]

Also the current ports system isn't quite plug and play enough, you currently also need to add to the list in __init__.py when you add one. We should make it purely file driven, but I've run into issues doing that in the past.

[sbc100]

For now, adding to __init__.py is not a blocker IMHO

[kleisauke]

For the record, I would like to check in glib and libffi as port, but my main concern with https://github.com/emscripten-ports is that it's difficult to maintain when a new upstream version is available, especially if the Emscripten parts are also integrated upstream. Downloading tarballs and applying custom patches (if necessary) seems more common and a lot easier.

I think is convered by an existing bug but I can't seem to find it right now.

Looks like you're describing https://github.com/emscripten-core/emscripten/issues/9353.

[sbc100]

emscripten-ports is basically just a way to store the result of applying patches to an upstream release. if you don't have any patches then you don't need to use it you can just use the upstream github release directly.

We can argue the benefits of using tarball + patche file vs using a git repo, but changing the way we do things would be much bigger change and if you just want to get glib working in emscripten than using emscripten-ports to hold your patches is the easiest way to go today.

If we do want to get into the weeds of git vs patch patch files, I think I would argue that maintaining a series of patches on top of an upstream set of code is pretty much exactly what git is designed to do. A system you like describe, such as quilt (https://linux.die.net/man/1/quilt) where the user must manually the parch series as a set of individual files, just seems like doing everything git does but with a lot more work and lot more error prone. I imagine there are some advantages to keeping the patches outside of git, but storing them as gists seems pretty odd, when you have git, which has things like history. I guess each gist has its own history? But then are you are basically maintaining each patch its own git repo? Anyways, as you can see, I don't think its clear cut argument.

[kleisauke]

emscripten-ports is basically just a way to store the result of applying patches to an upstream release. if you don't have any patches then you don't need to use it you can just use the upstream github release directly.

Is there a reason why some of the emscripten-ports repositories are simply forked without patches? Or is it just for the issue tracker? For example; zlib, Ogg, Vorbis, Asio (after commit https://github.com/chriskohlhoff/asio/commit/607f99ba80df71f1a96f0df71725059ee71e46ff), FreeType, boost (see https://github.com/emscripten-ports/boost/issues/2), bzip2 and libpng could point to upstream instead. That's why I thought Emscripten ports didn't prefer upstream tarballs / Git releases.

I imagine there are some advantages to keeping the patches outside of git, but storing them as gists seems pretty odd, when you have git, which has things like history. I guess each gist has its own history? But then are you are basically maintaining each patch its own git repo? Anyways, as you can see, I don't think its clear cut argument.

Indeed, storing them in gists is odd. It was just to resolve this issue as the libvips WASM port (which contains the same patches) is not released yet.

[sbc100]

If there is a historical reason I don't know it. I would have thought the point of creation of repository there is the point at which we need a patch. If no patch is needed no repository should be needed.

[o-micron]

I agree, I think it should be as simple as a single or couple of patching files.
I think consistency is very important here.
Also we can create some sort of template folder that anyone can clone and basically use it in his/her own repository, this way the problem is simply solved by allowing every repository owner to write a custom emscripten build and at the same time the user would see the exact same experience across all ports ... nobody knows how to build a project more than the one who actually created it, so we can delegate that ..

I like how cmake works, whenever I open a new project I search for CMakeLists.txt and simply create a build folder and write almost the same thing cmake .. and make

This type of consistency solves a lot of problems, it's like you can almost guess how to build the project and with 90% chance you will build it with no issues unless the author adds more complicated options and that's what makes cmake sometimes not so good ...

Emscripten could have same model but without the options and complexity, just a single command to build and everyone is happy .. that's not easy to do, but I think that's how it should be done

[wurc]

@kripken Here is a script which does everything really ... could replace the whole port repository

Tested on Ubuntu 20.04

# required tools ...
sudo apt install autoconf libtool m4 automake

# install meson from pip3 because glib uses meson ..
pip3 install meson
export PATH=~/.local/bin/:$PATH

# Get the emsdk repo
git clone https://github.com/emscripten-core/emsdk.git

# Enter that directory
cd emsdk
# Fetch the latest version of the emsdk (not needed the first time you clone)
git pull

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file)
./emsdk activate latest

# Activate PATH and other environment variables in the current terminal
. ./emsdk_env.sh

# Make sure we have mainly wasm-ld and other tools in path as well
export PATH=$(pwd)/upstream/bin/:$PATH

cd ..

# Get @kleisauke patches
git clone https://gist.github.com/acfa1c09522705efa5eb0541d2d00887.git
cd acfa1c09522705efa5eb0541d2d00887

chmod +x ./build.sh
./build.sh

@o-micron I did according to above steps, but below error pop up
configure: error: in /home/wrc/wasm_study/acfa1c09522705efa5eb0541d2d00887/deps/ffi': configure: error: C compiler cannot create executables Seeconfig.log' for more details
...and I checkes the config.log, below is the info
configure:3877: /home/wrc/emsdk/upstream/emscripten/emcc -O3 -L/home/wrc/wasm_study/acfa1c09522705efa5eb0541d2d00887/target/lib -O3 conftest.c >&5
cache:INFO: generating system asset: generated_struct_info.json... (this will be cached in "/home/wrc/emsdk/upstream/emscripten/cache/wasm/generated_struct_info.json" for subsequent builds)
requested a shared WebAssembly.Memory but the returned buffer is not a SharedArrayBuffer, indicating that while the browser has SharedArrayBuffer it does not have WebAssembly threads support - you may need to set a flag
/tmp/tmpt2kyses0.js:171
throw ex;
^

Error: bad memory
at Object. (/tmp/tmpt2kyses0.js:1533:13)
at Module._compile (internal/modules/cjs/loader.js:936:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:947:10)
at Module.load (internal/modules/cjs/loader.js:790:32)
at Function.Module._load (internal/modules/cjs/loader.js:703:12)
at Function.Module.runMain (internal/modules/cjs/loader.js:999:10)
at internal/main/run_main_module.js:17:11
FAIL: Running the generated program failed!

[kleisauke]

@wurc Could you try this?:

# The struct_info file must be built without modifications to EMMAKEN_CFLAGS
EMMAKEN_CFLAGS= embuilder.py build struct_info

(I just added this to the gist)

[wurc]

@kleisauke, it works :-)
Thank you ~

[wurc]

@kleisauke one more question,
when we "#include"
and use emcc to build, below error will pop up
"/home/wrc/wasm_study/glib_wasm/include/glib-2.0/glib/gtypes.h:32:10: fatal error: 'glibconfig.h' file not found

include

     ^~~~~~~~~~~~~~

1 error generated.
"
and I do check, there is really no glibconfig.h

[o-micron]

@wurc you should find glibconfig.h in your target/lib/glib-2.0/include .
Just add that folder to the include headers as well

[kripken]

I think there's no reason to create a forked directory unnecessarily. Perhaps we created some historically when we thought we would need patches, but ended up not needing them? PRs to simplify that would be welcome, and we can delete those repos.

[wurc]

@kripken when I use the generated glib, below error comes, do you know why this issue happened?
wasm-ld: error: /home/wrc/wasm_study/glib_wasm/lib/libgio-2.0.a(gthreadedresolver.c.o): undefined symbol: res_query
wasm-ld: error: /home/wrc/wasm_study/glib_wasm/lib/libgobject-2.0.a(gclosure.c.o): undefined symbol: ffi_call
wasm-ld: error: /home/wrc/wasm_study/glib_wasm/lib/libgobject-2.0.a(gclosure.c.o): undefined symbol: ffi_call

[kripken]

Those functions are not being linked in, it seems? To investigate it, find where they are defined, and then you can use llvm-nm to verify through the build process where the symbols are, and that they reach the final link etc.

[sbc100]

res_query is used to looking up hostnames in DNS. You should probably find a what to remove that call from gthreadedresolver.c to completely remove gthreadedresolver.c from the build since emscripten doesn't support networking by defaults.

ffi_call is most likely part of libffi which has not been ported to emscripten and porting it would most likely be a bunch of work. Can you find a way to disable the libffi support or disable the gclosure.c support completely?

[sbc100]

Sorry .. see the above discussion about libffi and had forgotten we had already discussed it and it looks like the a solution already.

[wurc]

Yes I just use gsocket related apis. It is impossible to use those apis?
@kleisauke can you help me with this issue?
g_inet_socket_address_new_from_string
g_socket_address_get_family
g_socket_set_blocking
g_socket_bind
...

[kleisauke]

@wurc You could try to adapt this patch:
https://gist.github.com/kleisauke/acfa1c09522705efa5eb0541d2d00887#file-glib-emscripten-patch-L183-L203

I created that patch because res_query is not available in WASM and would otherwise cause a build error:

Checking if "res_query()" links: NO 
Checking if "res_query() in -lresolv" links: NO 
Checking if "res_query() in -lbind" links: NO 

gio/meson.build:60:6: ERROR: Problem encountered: Could not find res_query()

Perhaps it's only needed to just disable res_query? It might be possible that one of the other functions listed here are still available:
https://gitlab.gnome.org/GNOME/glib/-/blob/11e9c5a9ce96f7e8ffbdd588f1c8f8df1e51761d/gio/meson.build#L40-155

@kripken That's great, thanks for clarifying! I'll take a look to prefer the upstream repositories if no patches are needed. This may also facilitate port maintenance and speed up delivering improvements from new versions to end users.

[wurc]

@kleisauke it means currently it is impossible to use gsocket apis which depends on network libs, right? Actually we use the glib socket apis to implement muti-cast function, and we want to transfer it to wasm module

[sbc100]

The socket I/O APIs in emscripten are based on websockets, which means you need to proxy all your socket data back through a websocket to your server and then out from there. I doubt very much that things like mutli-cast will work, but perhaps its possible. Sadly, sockets and web are not currently a good match and you will have to go through these extra hoops.

I would advise looking onto how the websocket bridge works and running some experiments to see if you use case can be made to work on the web.

[wurc]

Hi I use below method to use muti-cast, but when calling start_muticast_client in C++, it can't works
while the same code can be executed in node env
EM_JS(void, start_muticast_client, (), {
var dgram = require('dgram');
var socket = dgram.createSocket({ type: 'udp4', reuseAddr: true });
var multicastAddress = '224.0.0.251';
var multicastPort = 5353;
socket.on('listening', function(){
var address = socket.address();
console.log('UDP Client listening on ' + address.address + ":" + address.port);
socket.addMembership(multicastAddress);
socket.setMulticastLoopback(false);
});
socket.on('error', (err) => {
console.log('socket error:\n${err.stack}');
socket.close();
});
socket.on("message", function ( message, remote ) {
console.log("Message received from ", remote.address, " : ", message.toString());
});
socket.bind(multicastPort);
});

[kleisauke]

A first release for the above mentioned WebAssembly bindings for libvips has now been published.
https://libvips.github.io/libvips/2020/09/01/libvips-for-webassembly.html

I'll try to integrate the necessary patches of this project upstream (annotated with Upstream-Status: Pending in the commit message). The "In progress" column at https://github.com/kleisauke/wasm-vips/projects/1 shows the PRs that have already been submitted/accepted.

[justin-hackin]

I'm currently working to port some code from https://gitlab.com/inkscape/lib2geom to wasm and glib is the last dependency I need to tackle. Thanks a lot for working to sort this out folks.

I tried carefully and repeatedly to get a glib build using the instructions on Ubuntu 20.04 but failed. Here's the most relevant part of the log from the build log:

 Compiler stderr:
 emcc: error: STANDALONE_WASM does not support pthreads yet

Checking for size of "wchar_t" : -1

meson.build:1232:2: ERROR: Problem encountered: Compiler provides no native 16-bit integer type

My meson version was ahead of a tested version mentioned so I also tried downgrading from 0.56.0 to 0.54.1 and got the same results.

full console log
full meson log

_Note: I renamed the hash folder to glib-container._

EDIT: I found this issue which seems related to this one I'm having.

[justin-hackin]

@kleisauke do you think the glib and ffi patches are the only ones we need to fix the glib build as instructed above?

[kleisauke]

@justin-hackin I've just updated the gist with a tested Dockerfile and necessary patches for glib and libffi. Let me know if that works for you.

[justin-hackin]

@kleisauke that's kind of you to share, thanks a lot, it works! You made my day :)