Emscripten: How to properly import modules and bundle

[vncoelho]

What is the correct way to inject multiple libraries?

For example, we want to inject this Crypto: https://github.com/NeoResearch/neo3-cpp-core/blob/master/libs/lib/node_modules/crypto-js/crypto-js.js

Together with our functions that use it:
https://github.com/NeoResearch/neo3-cpp-core/blob/4bf8bead8fd97cde6233c4b91e903226c9e5140b/src/libcore-js/libcore_exports.js#L146-L185

We are currently doing it with MergeInto:

On Node we do not have problem for testing because we use Require.

lNeo3 = require('./build/librarytest'); //librarytest is our library generated with emscripten
let lt_cryptojs = require('./libs/lib/node_modules/crypto-js/crypto-js.js');
lNeo3['CryptoJS'] = lt_cryptojs;

However, we are facing problem when dealing with the .html and modules include, because the crypto-js is not loaded first and we have other issues.
What is the recommendation ?

[kripken]

Not sure what you mean by MergeInto? Perhaps the mergeInto JS library helper function?

Is the question how to use a bunch of normal JS from a JS library? (I'm not sure from the large files linked to - perhaps a small example of what you want would help understand the question.)

[igormcoelho]

Thanks for the reply @kripken , I'll explain what @vncoelho mentioned and our workaround to fix it.

Our project neopt-sdk-js depends on two external JS projects:

• crypto-js
• csbiginteger-js

We convert C++ project neo3-cpp-core into JS with emscripten and need to use functions from these two libs inside our mergeInto functions.

After struggling a little bit, we found this solution (which we don't know it's the best way in emscripten), using two different --pre-js (one for web js, and another for node js):
prefix-web.js

Module["csBN"] = csbiginteger.csBigInteger;
Module["CryptoJS"] = CryptoJS;

prefix-node.js

Module["csBN"] = require('csbiginteger').csBigInteger;
Module["CryptoJS"] =  require('crypto-js');

This way we compile two times, one for a web js output, and another for a node js (commonjs) output. We wonder: is there a better way to have a "single" module loading for both web and nodejs versions?

The bad thing is that html version depends on having a <script></script> tag before emscripten to dump the dependencies into global scope, but it works. For node version, it works fine without any extra change from user pespective (just require('emscripten-output.js') in their project).

I tried to use require.js library, without success, since many errors happened with define(), but honestly we are experienced in C++, but not that much in JS :smile: (and these infinite types of modules...)

Anyway, congratulations for this amazing emscripten project!

[igormcoelho]

Just to summarize the question in a simpler way: is there any official way to include JS dependencies from third-party into the Module generated by emscripten?

[kripken]

Adding a dependency in a prior script tag is an option, yeah, but it has the downsides you mention.

How I would do it is using a --pre-js as you said. Then it's inside the emscripten code, and if you use MODULARIZE then it's all modularized together, and nothing but the Module reaches the global scope. --pre-js is exactly meant to include more JS code that you need for some reason.

You could make a single build for web and node this way, if you bundle both --pre-js files and pick the right one at runtime. 2 separate builds might be better for size and startup though.

[vncoelho]

Hi @kripken, as my brother said, we are thankful and happy with all efforts that were already conducted in this brilliant project.

You could make a single build for web and node this way, if you bundle both --pre-js files and pick the right one at runtime. 2 separate builds might be better for size and startup though.

This we already did, dividing two --pre-js files, which we call prefix-node.js and prefix-web.js

let lt_csbn = require('csbiginteger');
let lt_cryptojs = require('crypto-js');

Module["csBN"] = lt_csbn.csBigInteger;
Module["CryptoJS"] = lt_cryptojs;

console.log("prefix: will need csbiginteger now...");
Module["csBN"] = csbiginteger.csBigInteger;
console.log("prefix: will need CryptoJS now...");
Module["CryptoJS"] = CryptoJS;

But the question is, how to bundle all together? We were still not able to obtain a single file that contain our script and dependencies.

[kripken]

You should be able to do something like this:

if (typeof require === 'function') {
  // This is node.js
  let lt_csbn = require('csbiginteger');
  let lt_cryptojs = require('crypto-js');
  Module["csBN"] = lt_csbn.csBigInteger;
  Module["CryptoJS"] = lt_cryptojs;
} else {
  // This is not node.js
  Module["csBN"] = csbiginteger.csBigInteger;
  Module["CryptoJS"] = CryptoJS;
}

That is, connect the right thing at runtime. You can bundle csbiginteger in the build, and the node.js path will just not use it.

(Note that you may want to do more than just check for require, emscripten does ENVIRONMENT_IS_NODE = typeof process === 'object' && typeof process.versions === 'object' && typeof process.versions.node === 'string';).

[igormcoelho]

Thanks a lot for the advices @kripken .
We managed to have a first operational version, based on emcc version 1.39.4 (from docker image trzeci/emscripten in a gist). We were currently using 1.39.16, and downgraded to 1.39.4.
The downgrade made C++ bindings stop working (so we moved back to extern "C" functions) and even in pure C, that simple gist example only worked for 1.39.4 (in 1.39.16 it compiles, but doesn't work on practice).

---

Here is the long answer, for those who may need to study our solution in some future (we created a gist with slices of the used code): https://gist.github.com/igormcoelho/5d7e1ba721c0f1e4ec9eec07b6c97d41

---

One final advice: npm serve will refuse to offer .wasm files correctly... so I created my own express.js server for that (needs version 4.17.1 and type-is 1.6.18).
(this avoids warning "wasm streaming compile failed: TypeError: Response has unsupported MIME type")

package.json

{
  "name": "serve wasm correctly",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "dependencies": {
    "express": "^4.17.1",
    "type-is": "^1.6.18"
  }
}

server.js

var path = require('path');
var express = require('express');
var app = express();
var staticPath = path.join(__dirname, '/public');
app.use(express.static(staticPath));
// thanks to: https://stackoverflow.com/questions/50589083/typeerror-failed-to-execute-compile-on-webassembly-incorrect-response-mime
app.listen(8080, function() {
  console.log('Express version: ', require('express/package').version);
  console.log('listening on port 8080');
});

[igormcoelho]

I guess you can close this @vncoelho.