Emscripten: Working with Audio Worklet

[jsroga]

Hey guys,
I have a hard time integrating WASM + Emscripten with Audio Worklet.
He's how we tried to do it:

1. We are downloading and compiling .wasm file from the main thread.
2. We are instantiating Emscripten's WASM module in the main thread

Module.instantiateWasm = (imports, successCallback) => {
       wasm.then((wasmBinary) => {
          WebAssembly.instantiate(new Uint8Array(wasmBinary), imports)
            .then((output) => {
              $window.Module.testWasmInstantiationSucceeded = 1;
              successCallback(output.instance, output.module);
            })
        });
        return {};
      };

1. We are sending wasmModule and wasmMemory to AudioWorklet, thanks to the patch you did on message port
2. From AudioWorklet thread we wanted to instantiate it by passing its' memory and module, like that:

AudioWorkletGlobalScope.Module['instantiateWasm'] = function(imports, receiveInstance) {
  AudioWorkletGlobalScope.wasmInstance = new WebAssembly.Instance(module, imports);
  receiveInstance(AudioWorkletGlobalScope.wasmInstance, AudioWorkletGlobalScope.Module.wasmModule);
  return AudioWorkletGlobalScope.wasmInstance.exports;
}

but this is problematic. To do that we need to import Emscripten glue-js output file in AudioWorklet context. So I've tried to do something like that: audioCtx.audioWorklet.addModule('wasm/sndt.js') but it didn't work since Emscripten is not prepared to work in the AudioWorklet context.

Best,
Jacek.

[niekvlessert]

Hi,

I'm following (kind of) the same path and I'm running into these issues as well. At a certain point I just tried to inline the Emscripten code in the worklet, that didn't work obviously. The console didn't give me a lot (yet), but I discovered it's not allowed to set variables outside of a function in the worklet. I stopped there because I figured it would be difficult to get my 1MB Emscripten 'blob' in 1 or more functions.

Since a lot of audio applications use transpiled C code I guess it would be a nice addition.

Regards,

Niek

[FalkorX]

I'm not sure this helps, but I link my C/C++ object files for use in the AudioWorkletProcessor with

emcc -s SINGLE_FILE=1 -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s ASSERTIONS=0 -O0 -o <OUT>.js

and use Module['ENIVIRONMENT']='WORKER' in the pre.js. I register the Processor class in the post.js. Through this I can directly import the result with audioCtx.addModule(<OUT>.js). This way I don't even have to send the wasm from the main thread to the render thread, but load it directly there.

[niekvlessert]

@FalkorX Sure sounds like a solution! So you generate the whole worklet? Is there an example online?

The code that comes out overhere starts with

var Module;

if (typeof Module === 'undefined') Module = {};
...

Not a great start for a worklet I'd say. What would I add to the pre.js to get this to work?

I tried to add the class ... extends AudioWorkletProcessor { process { ... } } to the pre.js, but it ends up half way of the file, and the register to post.js, removing the --embed-file thing I have changed that, but the module is not loadable at all, since var Module; etc. is in there. Probably something wrong with the Module['ENVIRONMENT']='WORKER'

[FalkorX]

@niekvlessert Sorry, I don't have it online, and yes, I generate the whole worklet, no extra steps needed.

My pre.js is really only the line Module['ENVIRONMENT'] = 'WORKER'; and in the post.js, I define the processor and just use the Module there:

// post.js
registerProcessor('ProcessorName', class extends AudioWorkletProcessor {
  ...
  process(input, output) {
    ...
    Module.process();
    ...
  }
}

If you're worried about polluting your AudioWorkletGlobalScope's namespace, you can also just use the -s MODULARIZE_INSTANCE=1 linker option to keep the Module in a closure and -s EXPORT_NAME='"<MODULE_NAME>"' to give it a unique name.

[niekvlessert]

@FalkorX Amazing, I never thought I would be possible to create some javascript code, put a class behind it and then just use the class. I have one issue left; I want to access the Module in the process function in the class as you said, but the Module variable does not exist in the class, only outside of it?

[FalkorX]

That is not a problem, the Module will keep existing in the class's (and thus the process function's) closure.

var Module = ...; // *
...
class Proc {
  process() {
    Module.func(); // refers to *
  }
}

[niekvlessert]

@FalkorX My mistake, it works fine. I'm pretty sure I got undefined for Module, but it probably had something to do with the fact that I was using an old version of Emscripten and had some browser issues.

This is a nice method, for me at least, thanks!

[flatmax]

Hey there, this approach seems to give the following error :

/usr/local/emsdk-portable/emscripten/1.37.35/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:272
        throw new JS_Parse_Error(message, line, col, pos);
        ^
TypeError: {} is not a function

Where my post.js contains the following :

class AudioProcessor extends AudioWorkletProcessor {
  process(inputs, outputs, parameters) {
    console.log('processed once and exiting')
    return false;
  }
}
registerProcessor('audio-processor', AudioProcessor);

Any suggestions ?

[niekvlessert]

Well, not right now, more information is required. Is the whole source online somewhere? Maybe, If it's easy to fetch and is ready to build on Linux, I can try to build it for you.

[flatmax]

Yes, here it is :
https://github.com/madChopsCoderAu/WASMAudio/tree/AudioWorklet

It is the AudioWorklet branch of that code base.

[FalkorX]

I had the same (or a similar) problem: the class and extends keywords are ES6 features, just as let, const and some others. The Emscripten uglifier currently only understands ES5. This is why you need to compile with -O0 or -O1, because it doesn't invoke the uglifier. -O2 and -O3 won't work.
However, the people here are alrady working on this issue, see https://github.com/kripken/emscripten/issues/6041.

[flatmax]

I made this change - now post is correctly added. When run in the browser this is executed in test-element.html :

      runAudioWorklet(){
        if (this.context==null)
          this.context = new AudioContext();
        this.context.audioWorklet.addModule('libwasmaudio.js').then(() => {
          let oscillator = new OscillatorNode(this.context);
          let audioWorkletNode = new AudioWorkletNode(this.context, 'audio-processor');
          oscillator.connect(audioWorkletNode).connect(this.context.destination);
          oscillator.start();
        });
      }

There is now a bug where the code doesn't seem to be executed by the addModule command :

test-element.html:47 Uncaught (in promise) DOMException: Failed to construct 'AudioWorkletNode': AudioWorkletNode cannot be created: The node name 'audio-processor' is not defined in AudioWorkletGlobalScope.
    at context.audioWorklet.addModule.then (http://127.0.0.1:8081/components/test-element/test-element.html:47:34)

I have updated the repo branch AudioWorklet :
https://github.com/madChopsCoderAu/WASMAudio/tree/AudioWorklet

[flatmax]

OK - got it.
The problem is when you modularize : -s "MODULARIZE=1" -s EXPORT_NAME="'libwasmaudio'"
The code doesn't get called, once removed, then the WASM code gets compiled.

[ArnaudBienner]

I wanted to do the same thing and I managed to get this done by using es6 modules.
They seem to be widely supported now (Edge, Chrome, Safari and current Firefox beta according to [1]).

As suggested in #6284 I added export default Module; using --post-js option then use import Module from 'myLib.js'; in my other JavaScript module. This other JavaScript module could be the one defining defining the AudioWorklet class and being imported in the AudioWorkletGlobaleScope using addModule.

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

[TimDaub]

Hi,

I'm running into the same problem now again. Since your conversation, defining ENVIRONMENT as worker in pre.js has been deprecated. Now it's recommended to use the argument -s ENVIRONMENT=worker.

However, an AudioWorklet is not a worker. If we still use the above used approach, then we'll run into the problem that in a Worklet WorkerGlobalScope.self is not defined.

If I simply compile without an environment parameter, add my registerProcessor call as --post-js, I run into the following error:

Cannot assign to read only property '__wasm_call_ctors' of object '[object Object]'

Essentially a -s ENVIRONMENT=worklet option would be nice.

Edit:

The solution by @ArnaudBienner seems to work, requires though a browser that can handle es modules. So as long as every browser supports audioworklets and es modules, it's possible. IMO however there should be a way to do it without modules.

[Archie3d]

Hi,

What is the latest state of this issue?
I am compiling with -s ENVIRONMENT=worker but the audio worklet still fails to load:

// in generated glue code js
if (ENVIRONMENT_IS_WORKER) {
 _scriptDir = self.location.href;
}

with the error Uncaught ReferenceError: self is not defined.

Also further down it is looking for window or importScripts which are not available from within an audio worklet:

if (!(typeof window === "object" || typeof importScripts === "function")) throw new Error ("...");

Now if I don't compile it as worker environment, I get another exception because my WebAssembly uses Pthreads and Fetch API (don't even know whether it is possible to use threads from within audio worklet).

[flatmax]

The simple AudioWorklet C++ project has moved to here : https://github.com/flatmax/WASMAudio/tree/AudioWorklet-litelement

It was working before, but unfortunately something has changed. It now returns a new error :
DOMException: Failed to construct 'AudioWorkletNode': AudioWorkletNode cannot be created: The node name 'audio-processor' is not defined in AudioWorkletGlobalScope.

You can check some of the old tricks to getting it working on that branch ... if you get past the problem I mentioned link back.

[Archie3d]

@flatmax thanks, I guess it's because how post-js (which contains the worklet class) gets appended with MODULARIZE=1 - it goes inside the generated module now, and not in a global scope.

[kripken]

--extern-post-js is an easy way to append code outside the modularize scope.

[flatmax]

Thanks for the tips!

When I use the --extern-post-js the browser can't find the libwasmaudio
module - probably because it hasn't been compiled in the browser yet.

The binded js file starts with this :

var libwasmaudio = (function() {

and this function ends with this :

})();

I would expect the browser to have executed that function which compiles
the WASM in the browser after executing the following in the webapp :

    this.context = new AudioContext();
    this.context.audioWorklet.addModule('libwasmaudio.js').then(() => {

But for some reason it doesn't seem to run the libwasmaudio function
from the binded libwasmaudio.js file.

On 15/7/20 3:56 am, Alon Zakai wrote:
>

|--extern-post-js| is an easy way to append code outside the
modularize scope.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/emscripten-core/emscripten/issues/6230#issuecomment-658324536,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFLUBYFKGPXPACTNTEEW6TR3SL47ANCNFSM4EQI6PGQ.

[flatmax]

Got it working again.

I added an external post.js file which runs the module like so : libwasmaudio();
Now the module runs in the browser as expected.

If anyone needs an AudioWorklet and Emscripten reference in the future, checkout WASMAudio (the AudioWorklet litelement branch) :
https://github.com/flatmax/WASMAudio/tree/AudioWorklet-litelement

[Archie3d]

Thanks, it works, unfortunately I still cannot use Pthreads since Worker in not defined in audio worklet's scope :(

[flatmax]

I also get problems when I add "-s ENVIRONMENT=worker" to [the bind command].(https://github.com/flatmax/WASMAudio/blob/AudioWorklet-litelement/src/Makefile.am#L39) :

@emcc --bind --llvm-opts 1 --memory-init-file 0 -s ENVIRONMENT=worker -s MODULARIZE=1 -s EXPORT_NAME="'libwasmaudio'" -s SINGLE_FILE=1 -s "BINARYEN_METHOD='native-wasm'" -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 -s BINARYEN_ASYNC_COMPILATION=0 -s ASSERTIONS=0 $(AM_CXXFLAGS) -O1 --pre-js ../js/pre.js --post-js ../js/AudioProcessor.js --extern-post-js ../js/post.js .libs/libwasmaudio.so -o .libs/libwasmaudio.js

The browser reports :
libwasmaudio.js:106 Uncaught ReferenceError: self is not defined

[Archie3d]

Right, I had to add something like this to pre.js

var self = {
    location: {
        href: "https://localhost:4443" // URL where the module was loaded from
    }
}