Binaryen: "-s WASM=1 -s USE_PTHREADS=1 "raises questions about Shared content.

[Lachrista]

When I compile a xx.c file using -s WASM= 1-s USE_PTHREADS=1.
eg.
emcc test.c -s WASM=1 -s USE_PTHREADS=1 -o test.html
The following files will be generated:
pthread-main.js test.html test.html.mem test.js test.wasm
Now,run the test.html on Firefox and Google,
Firefox will error:
TypeError: invalid array type for the operation
at initMainThreadBlock
Google will error:
Uncaught TypeError: [object Uint32Array] is not an integer shared typed array.
at Atomics.store ()
at Object.initMainThreadBlock (test.js:1824)
at test.js:5557
But about firefox and Google.I opened the relevant experimental configuration.
chrome://flags/ ,I've already set up
(1)shared-array-buffer
(2)All about webassembly

I looked at test.js, and the error was here.

The mistake made me very embarrassed because it could not go on.
Can anyone answer that for me?

[hlolli]

+1 getting the same in node.js 9.3.0

So I updated to node 9.7.1 and this is my version

emcc -v
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 1.37.35
clang version 5.0.0  (emscripten 1.37.35 : 1.37.35)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/emsdk-portable/clang/tag-e1.37.35/build_tag-e1.37.35_64/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/7
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/7
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
INFO:root:(Emscripten: Running sanity checks)

shared buffer array is surely there

> typeof SharedArrayBuffer
'function'

[hlolli]

some observation

function updateGlobalBufferViews() {
  if (!(buffer instanceof SharedArrayBuffer)) {
    buffer = new SharedArrayBuffer(buffer.byteLength );
  }
  Module['HEAP8'] = HEAP8 = new Int8Array(buffer);
  Module['HEAP16'] = HEAP16 = new Int16Array(buffer);
  Module['HEAP32'] = HEAP32 = new Int32Array(buffer);
  Module['HEAPU8'] = HEAPU8 = new Uint8Array(buffer);
  Module['HEAPU16'] = HEAPU16 = new Uint16Array(buffer);
  Module['HEAPU32'] = HEAPU32 = new Uint32Array(buffer);
  Module['HEAPF32'] = HEAPF32 = new Float32Array(buffer);
  Module['HEAPF64'] = HEAPF64 = new Float64Array(buffer);
}

The global buffer clearly isn't SharedArrayBuffer, buf running this code cuases error:

failed to asynchronously prepare wasm: CompileError: WasmCompile: Compiling wasm function #4451:_sbrk failed: invalid block arity > 1 @+2447972
CompileError: WasmCompile: Compiling wasm function #4451:_sbrk failed: invalid block arity > 1 @+2447972

prints two times

[timkerchmar]

Same issue observed when attempting to compile a bare bones OpenGL app: https://github.com/timkerchmar/tsintegration/tree/emscripten/painter/Emscripten

This version of build.sh also fails on tls initialization.

em++ ../Painter.cpp ../../lib/TSIntegration.cpp ../../lib/TSThread.cpp ../../lib/emscripten/main.cpp -I../../lib -s FULL_ES3=1 -s USE_PTHREADS=1 -s WASM=1 --emrun --separate-asm -o foo.html

[retrogradeorbit]

I am also getting this problem. I have not put any pthread code in the codebase, I have just switched on the -s USE_PTHREADS=1 flag. The code compiles, however, when launching the test javascript loader in node, the first line requires() the emscripten produced javascript, which then blows up with:

/pthreadtest/pthreadtest.js:71
   throw ex;
   ^

TypeError: [object Uint32Array] is not an integer shared typed array.
    at Atomics.store (<anonymous>)
    at Object.initMainThreadBlock (pthreadtest/pthreadtest.js:1018:11)
    at Object.<anonymous> (pthreadtest/pthreadtest.js:5056:38)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
Makefile:57: recipe for target 'test' failed

My node version seems to support these inbuilts:

$ node -v
v8.10.0

$ node
> new SharedArrayBuffer(10);
SharedArrayBuffer { byteLength: 10 }
> Atomics
{}
> Atomics.store
[Function: store]

the generated pthreadtest.js line 1018 is:

  Atomics.store(HEAPU32, PThread.mainThreadBlock + 116 >> 2, tlsMemory);

[binji]

WebAssembly threading support is still in development, so it is not enabled by default in any browser. There has been some work done, see this page for more info.

[athei]

Same problem with Firefox 60 and javascript.options.shared_memory enabled. So pthreads + webassembly is plain broken right now? The quoted page infers that it should work to a certain degree, though.

[kripken]

I see things working in Firefox Nightly (62), after enabling the pref. For example, a basic OpenGL app like this works:

./emcc tests/hello_world_gles.c -s USE_PTHREADS=1 -o a.html

[athei]

I tried with Firefox Nightly today. I can confirm that it does work. However, the WebAssembly.instantiateStreaming()support seems to be broken:

wasm streaming compile failed: TypeError: Response has unsupported MIME type index.html:1249:13
falling back to ArrayBuffer instantiation index.html:1249:13

I suspect it is just something that is broken in Nightly, because Firefox should support it.

[kripken]

@athei - that's actually just a weird thing with the wasm spec. They require the proper MIME type for streaming support, and simple webservers often don't set it right. That's the case for the simple python webserver, for example. Which did you test on?

[kripken]

(But, when the MIME type is wrong, we fail to stream but still compile it without streaming, so it's not a fatal error, just means things are less efficient.)

[athei]

Yes its not fatal. I tested with this one: https://github.com/brson/basic-http-server

[kripken]

Hmm, might be good to file an issue there (or in the upstream MIME-handling library) about adding the wasm mimetype.

[athei]

I am not sure which create is responsible for mime selection. There is hyper, http and mime. But hyper depends on http but not mime. And http does not depend on mime, too. I guess I will just file with hyper and see how it goes.

[athei]

It seems that the mentioned webserver does not include any Content-Type whatsoever. Not even for html. As a consequence I switched to one with a larger popularity: https://www.npmjs.com/package/http-server

It seems to get it right. It does supply to correct mime which I verified with Wireshark: simc.pcapng.gz. But I still get the warning about wasm streaming compile failed: TypeError: Response has unsupported MIME type.

Using 63.0a1 (2018-06-27) (64-bit). Bug in emscripten or Firefox?

[kripken]

I'd check if it works in other browsers - if it works in all but Firefox, sounds like a Firefox bug; if it works in none of them, sounds like an emscripten bug.

[athei]

I checked with a hello world program with current Safari 11.1.1. The warning does not occur there. The same is true for Chrome 67.

It seems to be a Firefox Bug. Happens in Nightly and in Stable.

[kripken]

I can't confirm the bug on Firefox Nightly. Here's what I did:

./emcc tests/hello_world.c -O2 -o a.html

Create .py with

import BaseHTTPServer, SimpleHTTPServer
SimpleHTTPServer.SimpleHTTPRequestHandler.extensions_map['.wasm'] = 'application/wasm'
httpd = BaseHTTPServer.HTTPServer(('localhost', 8888), SimpleHTTPServer.SimpleHTTPRequestHandler)
httpd.serve_forever()

Running python a.py, I browse to http://localhost:8888/a.html. There is no MIME error shown in the console. But if I remove the MIME line from the .py then the error appears as expected.

[athei]

I can confirm that the console MIME error does not appear when using your python file. I did a quick skim through a trace and it seems that both servers using the correct MIME type application/wasm. There must be something else that the javascript server is doing wrongly. I traced both transmissions but for me they look basically the same. I do not get why it does not work with the javascript server.

javascript.pcapng.gz
python.pcapng.gz

[kripken]

Hmm sorry, I don't know enough about tracing network stuff.

Are there simple instructions for running it with your server where you see the problem?

[athei]

No worries I looked into it and looked deeper into the server response. The javascript server returned:
application/wasm; charset=utf-8 instead of application/wasm like it should. The latter is what your python server returns. I will report it upstream.

[kripken]

Great, thanks!

[Priysha-Aggarwal]

@kripken
I tried to compile a simple threaded C program using emscripten by command :
"emcc mutex.c -s USE_PTHREADS=1 -s WASM=1 -o mutex.html --memory-init-file 1" , I got the same error as posted by @retrogradeorbit on 31 May.
Also I tried running the generated html file on firefox nightly(63) but the JS console displays:
"TypeError: invalid array type for the operation" w.r.t. PThread.mainThreadBlock.
As mentioned by you, I tried installing Nightly version(62) from this link but the download button is hyperlinked to nightly version 63, can you please share the suitable link for the same.

[kripken]

This link should work: https://nightly.mozilla.org/

[Priysha-Aggarwal]

Thanks for the link.
Although this link again directed me to the download link for Firefox Nightly 63 browser, but i was able to get Nightly 62 successfully from this link.
Although, even now running the html file from compiling a simple "helloworld" c code, using pthreads, gives me the same error :
TypeError: invalid array type for the operation
Command used : emcc program.c -s USE_PTHREADS=1 -s WASM=1 -o program.html
I am working on a linux 16.04 system with x86 architecture.
Any suggestions on this issue would be very helpful.

[kripken]

Is that with javascript.options.shared_memory enabled? (Although, when disabled, the error message you get should be different, so it's probably not that.)

And is that on latest emscripten, 1.38.7? (I verified that works now.)

The line the error is thrown on may also have useful information - although I'm guessing it's creating the HEAP views or something like that?

Anyhow, if it's none of those, it may be a Firefox bug on your machine - could be platform specific which would explain why I don't see it.

Btw, does it work on Chrome canary? If so, I'd file a bug on Firefox.

[Priysha-Aggarwal]

Yes, i have enabled javascript.options.shared_memory in the browser, without that it gives "ReferenceError: Atomics is not defined".
I am currently working with emscripten 1.38.6.
As i mentioned in my first post, i get the error "TypeError: invalid array type for the operation" w.r.t. PThread.mainThreadBlock.
Also, as mentioned on this link, "Chrome Canary is currently not available on the Linux platform. " So, i can't try running the HTML file on it.
Edit : Tried with emscripten 1.38.7 too, leads me to the same error.
What OS have you tested this on? @kripken

[kripken]

I'm on Linux (Ubuntu).

As i mentioned in my first post, i get the error "TypeError: invalid array type for the operation" w.r.t. PThread.mainThreadBlock.

That's a variable, I think - which line that operates on it throws for you? Is it the same as the other poster, something like Atomics.store(HEAPU32, PThread.mainThreadBlock + 116 >> 2, tlsMemory); ? No idea what could cause an error there, though.

Maybe put up a build that doesn't work you, and I'll test that link over here? If it fails for me too I can debug it.

[Priysha-Aggarwal]

@kripken , thanks a lot for your prompt replies.
I am listing the build steps that I followed on an Intel i7 machine running Ubuntu 16.04.3. Please suggest if I am doing anything wrong with the build itself:
1) Built emscripten SDK exactly as mentioned in this link I have all the software prerequisites as mentioned there.
2) As mentioned here, cloned the incoming branch of emscripten using below command:
./emsdk install sdk-incoming-64bit (this is mentioned here )
3) Downloaded the Firefox nightly version 62 and 63 and set javascript.options.shared_memory boolean variable to “true”.
4) Compiled a pthread_create hello world example as following:
emcc program.c -s USE_PTHREADS=1 -s WASM=1 -o program.html --memory-init-file 1
5) Launched program.html generated above in both the Firefox nightly versions that always shows :
Preallocating 1 workers for a pthread spawn pool.
Preallocating 1 workers for a pthread spawn pool.
with a constant "running" sign for too long.
6) While running node program.js gives me :
/Users/priysha/emsdk/emscripten/incoming/mutex.js:106
throw ex;
^

TypeError: [object Uint32Array] is not an integer shared typed array.
at Atomics.store ()
at Object.initMainThreadBlock (/Users/priysha/emsdk/emscripten/incoming/mutex.js:1821:17)
at Object. (/Users/priysha/emsdk/emscripten/incoming/mutex.js:5651:38)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
at startup (internal/bootstrap/node.js:236:19)

Thanks in advance.

[kripken]

Those steps look ok to me. At this point I am out of ideas - I'd suggest putting up a build that I can test (as I said, if I see your build fail on my machine, I can then debug it), or you can debug it directly on your machine (I'd add some debug printouts to see what the objects are on that line - then debug backwards to see where they are created).

[Priysha-Aggarwal]

@kripken
I have uploaded my project here.
Note that i have build the project with debug information i.e. using -g flag with emcc and uploaded all the used and generated files on the above link.
Kindly try building this and suggest.
Thanks!

[kripken]

When I run your build it works as expected for me (although it seems to be built without -s PTHREAD_POOL_SIZE=8 or such - when I build with that, it also runs properly. I don't remember offhand why that pool stuff is needed, but all the examples in the test suite seem to use it).

So given that your build fails for you with that error, this does seem like a Firefox bug. If you still see it on current nightly, I'd suggest filing an issue there, https://bugzilla.mozilla.org/

[Priysha-Aggarwal]

@kripken
I tried compiling using -s PTHREAD_POOL_SIZE=8 flag too but no success.
I will file the issue as suggested by you.
Thank you for your time and effort.

[vivkumar]

Hi @kripken , went through all your above suggestions to @Priysha-Aggarwal . I am still not clear why I am unable to just use "node hello.js" command to run a plain hello world c code (doesn't have any pthread API/header inside it) when compiled using the command line "-s USE_PTHREADS=1". I tried this on Ubuntu 16.04 with the installations steps as mentioned in 5 posts above. Even using PTHREAD_POOL_SIZE also doesn't help. The exception thrown is exactly same as reported by @Priysha-Aggarwal 5 posts above. Is it that the only way to run an executable created with "-s USE_PTHREADS=1" is by using a browser (e.g., Firefox nightly)? If its running for you using node then I think the issue could be either with build steps (but you said above this is correct) or the versions of software dependencies.

[kripken]

Oh, I may have misunderstood something above, sorry!

1. The hang in firefox is, I believe, explained by not having -s PTHREAD_POOL_SIZE=8. I see @Priysha-Aggarwal mentioned that before but I missed it. So that doesn't sound like a firefox bug - instead, we should figure out what's going on with that flag (as I said, all the tests seem to use it; I'm not sure why it doesn't have a default value).
2. The [object Uint32Array] is not an integer shared typed array error in node.js is likely because node does not have SharedArrayBuffer support, or something else goes wrong in the creation of the threading code - but regardless, our current threading support can't work in node.js because we require Web Workers, and node doesn't have them. There are some packages on npm that have worker support, but I'm not sure how exactly. So to answer your question: all development so far has gone into getting pthreads working in browsers, it is not expected to work in node.js (but it would be great to get it working there too).

[vivkumar]

Thanks very much @kripken for clarifying my doubts.

[thomasjammet]

Hi all,

I am still having the invalid array type for the operation error log in all Firefox versions on Windows (the last I tried is 62.0b9 (64 bit)).

To reproduce the error you can just compile the tests/pthread/hello_thread.c test application with USE_PTHREADS=1 :
emcc .\hello_thread.c -o main.html -s USE_PTHREADS=1
And of course I have enabled javascript.options.shared_memory. The same application works in Chrome Canary (Version 69.0.3496.0).

Also my version of emcc is 1.38.8 incoming but I have tried with many other versions and I am still having this bug.

Any help would be really appreciated!

[kripken]

@thomasjammet that sounds like a firefox bug (given the same code works in chrome, and you are on a recent emscripten). I'd suggest filing an issue there, https://bugzilla.mozilla.org/ (the right component is Core::JavaScript Engine, this link might work to get there directly, https://bugzilla.mozilla.org/enter_bug.cgi?format=guided#h=dupes|Core|JavaScript%20Engine )

[thomasjammet]

Thanks @kripken,

I prefer to post bugs with short sample code so I have tried to dig into the reason of the issue. It seems that in the main.js application compiled the main buffer (used to initialize the HEAPU32 array) is an ArrayBuffer instead of a SharedArrayBuffer.

And strangely in Chrome it is a SharedArrayBuffer as expected. Could it be a buffer initialization error?

[kripken]

Could be, yeah. You can add more debug logging around where we create the buffer. That should be the same line in both browsers, and something like new ArrayBuffer(TOTAL_MEMORY) - so it seems odd one browser could have a different type than the other. But perhaps control flow doesn't reach there, like maybe one browser does not detect SAB support earlier on?

[koconnor]

@kripken, @thomasjammet:

An update for you all. I am using emscripten 1.38.10 on MacOS X 10.13.5. I have a pthreads test app that will give the 'invalid array type' message from this line:
Atomics.store(HEAPU32, (PThread.mainThreadBlock + 116 ) >> 2, tlsMemory); // Init thread-local-storage memory array.
with FireFox Quantum 61.0.1 (64-bit) with config javascript.options.shared_memory -> true
(as it should with the Spectre vulnerability 'fix'.)

But it does work with both Firefox Nightly (63.0a1 (2018-07-27) (64-bit)) and Chrome Version 70.0.3504.0 (Official Build) canary (64-bit) with the appropriate configs. @thomasjammet, it may work for you with similar versions, I don't have a Windows machine to try those tests.

On to other issues!

[thomasjammet]

Thanks @koconnor, I updated to FireFox Nightly 63.0a1 (64 bit) and it works fine!