Zstd: We need a WebAssembly binding

[naskarsubhajit]

We need a WebAssembly binding.
I see there is already a javascript binding using emscripten.
We need same to get a webassembly (.wasm) . Any lead on this will be really helpful.

[Cyan4973]

Some time ago, I made a test compiling zstd for wasm target.
It's not too difficult thanks to well documented development tools around clang( emscripten notably).
I was surprised by the performance, which was pretty good, within a factor 2 of native speed. So it shows great promise, even on mobile.

Unfortunately, there is a world of difference between testing wasm compilation and testing it locally, and maintaining a library that can be actually useful to web developers. Such test just serves as indication of feasibility and set performance expectations.

As far as I understand emscripten (translation: very little), it's a tool able to produce both javascript and wasm targets. It's merely a compilation option.

@yoshihito's zstd-codec is likely pretty close to the objective. At least, it already offers a Javascript API, and one version is even compiled for wasm.
So it seems it's merely a matter of compiling libzstd itself with wasm, and bundling it with the binding interface ?

[mvanleeu]

Would be interesting if @yoshihito would publish the Makefile / command line to build the emscripten variant of zstd, to boost your's and mine understanding of the toolchain :-)

[felixhandte]

This is what I came up with when experimenting with building a zstd decompressor for js. Obviously, you will have to modify the exported functions if you want to compress, etc.:

ZSTD_LIB_DIR ?= zstd-lib

libzstd.js libzstd.wasm: $(wildcard $(ZSTD_LIB_DIR)/**/*.c) $(wildcard $(ZSTD_LIB_DIR)/**/*.h) Makefile
    emcc \
        -O2 \
        -o libzstd.js \
        $(ZSTD_LIB_DIR)/common/*.c $(ZSTD_LIB_DIR)/compress/*.c $(ZSTD_LIB_DIR)/decompress/*.c \
        -I$(ZSTD_LIB_DIR)/ -I$(ZSTD_LIB_DIR)/common/ -I$(ZSTD_LIB_DIR)/compress/ -I$(ZSTD_LIB_DIR)/decompress/ \
        -DDEBUGLEVEL=0 -DZSTD_NOBENCH -DZSTD_NODICT -DNDEBUG \
        -s EXPORTED_FUNCTIONS='["_ZSTD_decompress", "_ZSTD_decompress_usingDDict", "_ZSTD_createDDict", "_ZSTD_freeDDict", "_ZSTD_createDCtx", "_ZSTD_freeDCtx", "_ZSTD_DCtx_reset", "_ZSTD_isError", "_ZSTD_getErrorName"]' \
        -s EXTRA_EXPORTED_RUNTIME_METHODS='["ccall", "cwrap", "writeArrayToMemory", "Pointer_stringify"]'

[arpu]

are the any news on this?

[cwoffenden]

I just added an example in the dev branch building for Emscripten using an amalgamate file. See contrib/single_file_decoder. There’s a simple script for combining the sources, then it’s just a case of adding the resulting file to your Emscripten project (optionally with the zstd.h header). The examples directory shows how.

The decoder adds about 25kB to the Wasm file.

[Cyan4973]

I guess @cwoffenden wasm decoder is a good answer to this topic.
Closing it.

[donmccurdy]

It took me a fair bit of trial and error to get from the single-file decoder to something I can use in a JavaScript library. In case it's useful to someone else, my steps and final result are described in https://github.com/mrdoob/three.js/pull/19932.