Emscripten: Compile Eigen::Threadpool to WASM failed

[Wenzhao-Xiang]

Hi, I'm trying to compile Tensorflow Lite ops kernel to WASM, used for machine learning on Web, and some issues happened when compile Multithread-Conv. This op is based on Eigen::Threadpool, which is based on std::thread. But when compile it, the tool show me some warning as the following:

warning: unexpected number of arguments 5 in call to '__atomic_compare_exchange_8', should be 6
warning: unexpected return type i1 in call to '__atomic_compare_exchange_8', should be i64
warning: unexpected number of arguments 5 in call to '__atomic_compare_exchange_8', should be 6
warning: unexpected return type i1 in call to '__atomic_compare_exchange_8', should be i64
warning: unexpected number of arguments 5 in call to '__atomic_compare_exchange_8', should be 6
warning: unexpected return type i1 in call to '__atomic_compare_exchange_8', should be i64
warning: unexpected number of arguments 5 in call to '__atomic_compare_exchange_8', should be 6
warning: unexpected return type i1 in call to '__atomic_compare_exchange_8', should be i64
warning: unexpected number of arguments 5 in call to '__atomic_compare_exchange_8', should be 6
warning: unexpected return type i1 in call to '__atomic_compare_exchange_8', should be i64
warning: unexpected number of arguments 5 in call to '__atomic_compare_exchange_8', should be 6
warning: unexpected return type i1 in call to '__atomic_compare_exchange_8', should be i64

warning: asm2wasm adding operands in call from $__ZN14EigenForTFLite15ThreadPoolTemplINS_20StlThreadEnvironmentEE11WaitForWorkEPNS_10EventCount6WaiterEPNS1_4TaskE to$___atomic_compare_exchange_8 (this is likely due to undefined behavior in C, like defining a function one way and calling it in another, which is important to fix)
warning: asm2wasm adding operands in call from $__ZN14EigenForTFLite10EventCount6NotifyEb to $___atomic_compare_exchange_8 (this is likely due to undefined behavior in C, like defining a function one way and calling it in another, which is important to fix)
warning: asm2wasm adding operands in call from $__ZN14EigenForTFLite10EventCount6NotifyEb to $___atomic_compare_exchange_8 (this is likely due to undefined behavior in C, like defining a function one way and calling it in another, which is important to fix)
warning: asm2wasm adding operands in call from $__ZN14EigenForTFLite10EventCount10CommitWaitEPNS0_6WaiterE to $___atomic_compare_exchange_8 (this is likely due to undefined behavior in C, like defining a function one way and calling it in another, which is important to fix)
warning: asm2wasm adding operands in call from $__ZN14EigenForTFLite15ThreadPoolTemplINS_20StlThreadEnvironmentEE11WaitForWorkEPNS_10EventCount6WaiterEPNS1_4TaskE to$___atomic_compare_exchange_8 (this is likely due to undefined behavior in C, like defining a function one way and calling it in another, which is important to fix)
warning: asm2wasm adding operands in call from $__ZN14EigenForTFLite15ThreadPoolTemplINS_20StlThreadEnvironmentEE11WaitForWorkEPNS_10EventCount6WaiterEPNS1_4TaskE to$___atomic_compare_exchange_8 (this is likely due to undefined behavior in C, like defining a function one way and calling it in another, which is important to fix)

The compilation is successful. But when calculating, this op doesn't work as expect. All outputs are zero, just like the worker didn't do anything. Maybe the warning leads to this failure, but I don't know how to fix it.
Is there any limits for WASM pthread support?

Here is the Multithread-Conv source code.
And here is the Eigen::Threadpool definition.

And the flags are
-s WASM=1 -s NO_FILESYSTEM=1 -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=8 -s TOTAL_MEMORY=268435456 -s SINGLE_FILE=1 -s MODULARIZE=1 -s NO_EXIT_RUNTIME=1 -s EXPORT_NAME='nn_ops' --memory-init-file 0 --bind

[kripken]

That's probably a bug in fastcomp or asm2wasm. We hope to deprecate them soon, so it would be good to test with the replacement, where emcc uses the LLVM upstream wasm backend. However, threads are not fully finished there yet (fences are still being debated).

[Wenzhao-Xiang]

@kripken
Hi, Thanks for the advice!
I have tried to compile my project with LLVM upstream wasm backend. However, when set the flag USE_PTHREAD=1, the compiler told me

wasm-ld: warning: function signature mismatch: _ZNSt3__215basic_streambufIcNS_11char_traitsIcEEE7seekposENS_4fposI11__mbstate_tEEj
>>> defined as () -> void in /home/gwy/.emscripten_cache/wasm-obj/libc++_noexcept.a(iostream.cpp.o)
>>> defined as (i32, i32, i32, i32) -> void in /home/gwy/.emscripten_cache/wasm-obj/libc++_noexcept.a(ios.cpp.o)

wasm-ld: warning: function signature mismatch: _ZNSt3__215basic_streambufIcNS_11char_traitsIcEEE7seekoffExNS_8ios_base7seekdirEj
>>> defined as () -> void in /home/gwy/.emscripten_cache/wasm-obj/libc++_noexcept.a(iostream.cpp.o)
>>> defined as (i32, i32, i64, i32, i32) -> void in /home/gwy/.emscripten_cache/wasm-obj/libc++_noexcept.a(ios.cpp.o)

wasm-ld: warning: function signature mismatch: _ZNSt3__215basic_streambufIwNS_11char_traitsIwEEE7seekposENS_4fposI11__mbstate_tEEj
>>> defined as () -> void in /home/gwy/.emscripten_cache/wasm-obj/libc++_noexcept.a(iostream.cpp.o)
>>> defined as (i32, i32, i32, i32) -> void in /home/gwy/.emscripten_cache/wasm-obj/libc++_noexcept.a(ios.cpp.o)

wasm-ld: warning: function signature mismatch: _ZNSt3__215basic_streambufIwNS_11char_traitsIwEEE7seekoffExNS_8ios_base7seekdirEj
>>> defined as () -> void in /home/gwy/.emscripten_cache/wasm-obj/libc++_noexcept.a(iostream.cpp.o)
>>> defined as (i32, i32, i64, i32, i32) -> void in /home/gwy/.emscripten_cache/wasm-obj/libc++_noexcept.a(ios.cpp.o)

wasm-ld: error: 'atomics' feature is disallowed, so --shared-memory must not be used

Do I have a wrong use of upstream wasm backend?

Here is my flags:

emcc -std=c++11 bind/src/binding.cpp -o nn_ops.js -O3 -s WASM=1 -s NO_FILESYSTEM=1 
-s TOTAL_MEMORY=268435456 -s SINGLE_FILE=1 -s MODULARIZE=1 -s USE_PTHREADS=1
 -s PTHREAD_POOL_SIZE=8 --memory-init-file 0 --bind 
-I ./ -I external/ -I external/tensorflow/ -I external/tensorflow/tensorflow/contrib/makefile/downloads/absl/ 
-I external/tensorflow/tensorflow/contrib/makefile/downloads/eigen/ 
-I external/tensorflow/tensorflow/contrib/makefile/downloads/gemmlowp/ 
-I external/tensorflow/tensorflow/lite/kernels/ -I external/tensorflow/third_party/ 
-s ASSERTIONS=1

[sbc100]

@tlively

[tlively]

Add -pthread to make the error about the atomics feature go away. It's a
bug with emscripten that USE_PTHREADS=1 does not do this as well when the
LLVM backend is used. This won't help with the function signature
mismatches, though.

On Wed, May 1, 2019 at 11:51 AM Sam Clegg [e-mail] wrote:

@tlively https://github.com/tlively

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/emscripten-core/emscripten/issues/8503#issuecomment-488376093,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABWKW6Z6N6BXLXVM2H3AZDTPTHRCTANCNFSM4HIT6LLQ
.

[Wenzhao-Xiang]

Hi, @tlively
I tried it. But the same errors came again.

The followings are the flags and errors
flags:

emcc -std=c++11 bind/src/binding.cpp -o nn_ops.js -O3 -s WASM=1 -pthread -s NO_FILESYSTEM=1 \
-s TOTAL_MEMORY=268435456 -s SINGLE_FILE=1 -s MODULARIZE=1 -s USE_PTHREADS=1 \
-s PTHREAD_POOL_SIZE=8 --memory-init-file 0 --bind \
-I ./ -I external/ -I external/tensorflow/ -I external/tensorflow/tensorflow/contrib/makefile/downloads/absl/ \
-I external/tensorflow/tensorflow/contrib/makefile/downloads/eigen/ \
-I external/tensorflow/tensorflow/contrib/makefile/downloads/gemmlowp/ -I external/tensorflow/tensorflow/lite/kernels/ \
-I external/tensorflow/third_party/ \
-s ASSERTIONS=1

or

emcc -std=c++11 bind/src/binding.cpp -o nn_ops.js -pthread -O3 -s WASM=1 -s NO_FILESYSTEM=1 \
-s TOTAL_MEMORY=268435456 -s SINGLE_FILE=1 -s MODULARIZE=1 --memory-init-file 0 --bind \
-I ./ -I external/ -I external/tensorflow/ -I external/tensorflow/tensorflow/contrib/makefile/downloads/absl/ \
-I external/tensorflow/tensorflow/contrib/makefile/downloads/eigen/ \
-I external/tensorflow/tensorflow/contrib/makefile/downloads/gemmlowp/ -I external/tensorflow/tensorflow/lite/kernels/ \
-I external/tensorflow/third_party/ \
-s ASSERTIONS=1

errors:

wasm-ld: error: 'atomics' feature is used, so --shared-memory must be used
wasm-ld: error: Target feature 'atomics' is disallowed. Use --no-check-features to suppress.
wasm-ld: error: Target feature 'atomics' is disallowed. Use --no-check-features to suppress.
shared:ERROR: '/home/gwy/emsdk_upstream/emsdk/upstream/5396/bin/wasm-ld -o /tmp/emscripten_temp_vqwf4kg6/nn_ops.wasm --allow-undefined --import-memory --import-table --lto-O0 /tmp/emscripten_temp_vqwf4kg6/binding_0.o /tmp/emscripten_temp_vqwf4kg6/bind_1.o /home/gwy/.emscripten_cache/wasm-obj/libc++_noexcept.a /home/gwy/.emscripten_cache/wasm-obj/libc++abi.a /home/gwy/.emscripten_cache/wasm-obj/libc.a /home/gwy/.emscripten_cache/wasm-obj/libcompiler_rt.a /home/gwy/.emscripten_cache/wasm-obj/libc-wasm.a /home/gwy/.emscripten_cache/wasm-obj/libdlmalloc.a /home/gwy/.emscripten_cache/wasm-obj/libpthreads_stub.a /home/gwy/.emscripten_cache/wasm-obj/libcompiler_rt_wasm.a /home/gwy/.emscripten_cache/wasm-obj/libc_rt_wasm.a -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --export __wasm_call_ctors --export __data_end --export main --export malloc --export free --export setThrew --export __cxa_can_catch --export __cxa_is_pointer_type -z stack-size=5242880 --initial-memory=268435456 --no-entry --max-memory=268435456 --global-base=1024' failed (1)

[tlively]

Ok, there are a few things happening here due to our incomplete implementation of pthreads+llvm backend in emscripten. Here's how you can hack around them.

wasm-ld: error: 'atomics' feature is used, so --shared-memory must be used

What's happening here is that Emscripten should be passing --shared-memory to the linker because multithreaded applications require a wasm shared memory, but it's not. You can pass this flag manually to the linker with -Wl,--shared-memory.

wasm-ld: error: Target feature 'atomics' is disallowed. Use --no-check-features to suppress.

You're getting this error because emscripten is trying to link your code against a libc (or other library) that contained atomic operations in its source but was compiled for MVP, which caused those atomic operations to be lowered to non-atomic operations. Linking your multithreaded code with these libraries is very dangerous because those libraries will not be thread safe, even if they were supposed to be.

The proper way to handle this is to recompile the libraries you are using with pthreads enabled, but unfortunately Emscripten does not know how to compile its libc with pthreads yet. As the error message says, you can pass --no-check-features to the linker to hack your way past this issue, but that would result in an unexpectedly thread-unsafe final binary.

tl;dr: Pass -Wl,--shared-memory,--no-check-features to make these errors go away, but that's not exactly safe.

[tlively]

I opened #8534, #8535, and #8536 to track these issues

[hujiajie]

warning: unexpected number of arguments 5 in call to '__atomic_compare_exchange_8', should be 6
warning: unexpected return type i1 in call to '__atomic_compare_exchange_8', should be i64

According to this, it seems that there's an error in the signature of __atomic_compare_exchange_8 in system/lib/pthread/library_pthread_asmjs.c and src/library_pthread_stub.js.

[truedim]

Tried to use the latest LLVM upstream in my multithreaded CMake project, got a similar problem, here my wasm-ld command:

wasm-ld: error: 'atomics' feature is disallowed by SDL_spinlock.c.o, so --shared-memory must not be used
......
Use --no-check-features to suppress.
..........
wasm-ld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)

shared:ERROR: 'C:/Users/user/Documents/emsdk/upstream/bin\wasm-ld.exe -o C:/temp\emscripten_temp_o4hvyal5\myproj.w asm --allow-undefined --import-memory --import-table --lto-O3 .....here my cpp.o files....... C:\Users\user\.emscripten_cache\wasm-obj\libSDL2-mt.a C:\Users\user\.emscripten_cache\wasm-obj\libc-mt.a C:\Users\user\.emscripte n_cache\wasm-obj\libcompiler_rt.a C:\Users\user\.emscripten_cache\wasm-obj\libc-wasm.a C:\Users\user\.emscripten_cache\wasm-obj\lib c++-mt-noexcept.a C:\Users\user\.emscripten_cache\wasm-obj\libc++abi-mt.a C:\Users\user\.emscripten_cache\wasm-obj\libc-extras.a C: \Users\user\.emscripten_cache\wasm-obj\libgl-mt.a C:\Users\user\.emscripten_cache\wasm-obj\libdlmalloc-mt.a C:\Users\user\.emscripte n_cache\wasm-obj\libpthreads.a C:\Users\user\.emscripten_cache\wasm-obj\libcompiler_rt_wasm.a C:\Users\user\.emscripten_cache\wasm- obj\libc_rt_wasm.a --shared-memory -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-ls r --export __wasm_call_ctors --export __data_end --export main --export file_open --export malloc --export free --export setThrew --export __errno_location --export emscripten_get_global_libc --export __pthread_tsd_run_dtors --export _register_pthread_ptr -- export pthread_self --export memalign --export _ZSt18uncaught_exceptionv --export _get_environ --export emscripten_GetProcAddress --export strstr -z stack-size=5242880 --initial-memory=16777216 --no-entry --max-memory=536870912 --global-base=1024' failed (1)

There are no problems with Fastcomp ...

[tlively]

This error means that SDL was compiled with threads disabled, so all its atomics have been turned into nonatomic operations and it would not be threadsafe to allow it to link with your properly threaded code. (Fun fact, this isn’t threadsafe with fastcomp either, even though fastcomp doesn’t complain.)

To fix this you need to recompile SDL with threads enabled. I’m not actually sure what that entails. Perhaps @kripken knows.

[truedim]

@tlively Thanks for the information!

Fun fact, this isn’t threadsafe with fastcomp either, even though fastcomp doesn’t complain.

Now I understand why I had problems with SDL_PushEvent and threads.

To fix this you need to recompile SDL with threads enabled. I’m not actually sure what that entails.

As I understand SDL is compiled automatically when I start to build my project and it uses the same flags (pthreads, shared memory, etc) :

cache:INFO: generating port: libSDL2-mt.bc... (this will be cached in "C:\Users\user\.emscripten_cache\asmjs\libSDL2-mt.bc" for subsequent builds)

Are there any instructions how can I recompile it with proper flags?

[kripken]

Looks like we didn't build SDL2 properly with threads - fix for that bug is in #8960.

[truedim]

Looks like we didn't build SDL2 properly with threads - fix for that bug is in #8960.

Thank you, that's resolved the problem with SDL. I've just installed 1.38.38, added your fix to sdl2.py and got another problem with wasm-ld:
Assertion failed: FunctionIndex != INVALID_INDEX, file C:\b\s\w\ir\cache\builder\emscripten-releases\llvm-project\lld\wasm\Symbols.cpp, line 152
Compiler flags: -fno-rtti -ffast-math -std=c++17 -s USE_SDL=2 -s USE_PTHREADS=1 -s WASM_MEM_MAX=512MB -s ALLOW_MEMORY_GROWTH=1 -s WASM=1 -mnontrapping-fptoint -Wno-logical-op-parentheses -DNDEBUG -O2

[kripken]

Looks like a wasm-ld bug. I don't see anything similar here or on the LLVM bug tracker. @sbc100 does that sound familiar? @truedim do you have a small testcase?

[truedim]

@truedim do you have a small testcase?

@kripken No I don't have. I tried ImGui+SDL+pthreads demo project - there are no problems. My project besides my own code additionally use Eigen and many header-only boost libraries (multiprecision, spirit, etc).

[truedim]

@kripken I've finally localized the problem: https://github.com/emscripten-core/emscripten/issues/8981

[sbc100]

Can this now be closed?