Graaljs: Support ES Modules (in plain JS when embedding in Java)

[p-bakker]

I looked, but only found http://mail.openjdk.java.net/pipermail/graal-dev/2017-August/005050.html from 2017 and https://github.com/graalvm/graaljs/blob/master/graal-nodejs/doc/api/esm.md which seems related only to the Node impl. and I'm looking to just use plain EcmaScript from within Java

The docs of Graal JS says it's compatible with ES2018, but ESM has been part of ES since version 6

[woess]

There is experimental support for ES modules in non-Node.js mode. If you eval a Source, name of which ends with .mjs, it is parsed and evaluated as an ES module. Imported modules are loaded via the file system (requires allowIO(true)) and cached by path/name. The return value is the result of the last statement just as with scripts (not the exports!).

It's experimental because the Module Loader is non-standard (not part of the ECMAScript spec) and will probably change. Plus, it's not been tested that much yet. You are, of course, welcome to try it and provide feedback.

[pmlopes]

@woess could you point to the code where this loader lives?

[p-bakker]

@woess not quite sure I understand what you wrote: you said it returns the result of the last statement of the script, so not the exports. the way I understand that is that this implementation doesn't work at all like ES Modules should and thus aren't very useful. Or am I missing something?

As for imported modules being loaded through the FileSystem: This I can customize by providing a custom FileSystem implementation through the Builder for my Context?

[iamstolis]

@woess could you point to the code where this loader lives?

See JSContext.createModuleLoader().

[iamstolis]

@woess not quite sure I understand what you wrote: you said it returns the result of the last statement of the script, so not the exports.

I believe that @woess wanted to warn you that the result of the evaluation of the (top-level) module through Context.eval is the value of the last statement, not the exports. For example:

Context context = Context.newBuilder("js").build();
Source source = Source.newBuilder("js", "export default 42; 211;", "test.mjs").build();
System.out.println(context.eval(source));

prints 211 (i.e. nothing related to the export).

the way I understand that is that this implementation doesn't work at all like ES Modules should and thus aren't very useful. Or am I missing something?

This is a misunderstanding. The requirements on exports/imports/module execution should be handled according to the specification. The specification just does not care about the case mentioned above. Technically, the end of Context.eval() corresponds to section 8.6.3.c:
If all Job Queues are empty, the result is implementation-defined. i.e. the behavior is implementation-defined (even for execution of scripts). So, the current behavior (while not being optimal probably) does not violate the specification, in fact.

As for imported modules being loaded through the FileSystem: This I can customize by providing a custom FileSystem implementation through the Builder for my Context?

Yes, that should work.

[p-bakker]

Excellent, thank you, makes sense now

[p-bakker]

One more Q about this: am trying to implement a custom Filesystem and am testing it with this code:
`Source source = Source
.newBuilder("js", "import * as entity from 'Entity'; export const x = 1", "someRule.mjs")
.build();

try {
Value result = context.eval(source)
} catch (e) {
}`

The import in the source I'm evalling triggers a call to Path parsePath(String path) method of my custom FileSystem. However, the string value I get as parameter in the parsePath method is a fully qualified path to a non-existing file on my local file system: C:\Users\Paul\eclipse\java-oxygen\eclipse\Entity

Is this to be expected and should I somehow start stripping the unwanted stuff? I'd expected to just get 'Entity' as a value.

My custom FileSystem is to load 'virtual' files, being it from a database, from memory, from some url etc.

[woess]

@p-bakker That's a bug. The path should be resolved/canonicalized using the custom FileSystem, but it's not. We'll fix this.

[p-bakker]

k, great, tnx

Are there any nightly builds available, so I wouldn't have to wait for RC11?

[woess]

@p-bakker the bug is fixed now, but didn't make it into RC11. RC12 is planned for Feb 1.
I'm afraid we don't have nightly builds available, but you can build GraalVM (or JS) from source.

[pete-by]

Playing with ESM from Java. I have a js folder in a user.dir with two files index.mjs and module1.mjs:

js/module1.mjs:

export const sqrt = Math.sqrt;
export function square(x) {
    return x * x;
}
export function diag(x, y) {
    return sqrt(square(x) + square(y));
}

js/index.mjs:

import { square, diag } from "module1";
console.log(square(11)); // 121
console.log(diag(4, 3)); // 5

In Java I build a context and then run:

Source index = Source.newBuilder("js", new File("js/index.mjs")).name("index.mjs").build();
context.eval(index);

Unfortunately, it fails with an error (due to NoSuchFileException). However, it runs just fine if I add the mjs extension in the import declaration:

import { square, diag } from "module1.mjs";

I tried to override behavior with custom FileSystem but it did not work because a module file (TruffleFile) in JSModuleLoader#resolveImportedModule() is created before any call my custom FileSystem and I did not find a way to alter it before it is passed to the source builder. Moreover, in the FileSystem I do not have a context of whether the call was result of a module import or just a plain js was requested. Would be great to have an option to customize JSModuleLoader.

[eliasvasylenko]

I think it's worth pointing out that the ".mjs" thing is entirely a node convention, and is broadly considered to be a bit of an ugly hack. Afaiu they need it because they have to support es6 modules simultaneously with traditional node modules, and they need to know before parsing the file which kind of module to expect. I question whether it makes much sense to adopt the ".mjs" convention outside of the context of a node app.

Maybe you have a similar problem and you absolutely need to know before parsing whether a file is going to be a module or not. But even if that is the case, wouldn't this be better served by providing a new API point for module loading rather than overloading this one with a weird, ad-hoc filename matcher to determine behaviour?

[woess]

Maybe you have a similar problem and you absolutely need to know before parsing whether a file is going to be a module or not.

@eliasvasylenko yes, exactly. We need to know whether a Source is a module or a script. In order for a Source to be parsed as a module it needs to either have a file name ending in ".mjs" or the (unofficial) mime type application/javascript+module. So alternatively, you can specify the mime type via the Source.Builder. For import statements, the file name does not matter, as we always know it's a module.

[woess]

@pete-by We don't have any plans to allow for a custom module loader currently.
Note that you can always load the file yourself and pass it to Source.Builder.content, if you want to specify a File but don't want it to be loaded by Source.Builder.build(). From JS, paths and files should always be resolved and loaded using the polyglot FileSystem (if not, it's a bug).

[pete-by]

My JS files are stored in JCR, I have recently resolved my problem with loading modules without requiring them to have '.mjs' extension, but I basically had to implement the Java's FileSystem over the JCR for that and provide it to the builder. I appreciate everyone's comments in the threads, they are really helpful.

[Melechoris]

I have another question regarding loading of ES modules in JS when embedded in JAVA.
Is there a way to dynamically load ES modules at runtime (rather than using the import statement)?

I understand that for loading JavaScript source code the load(source) function can be used. But if I change the example provided by @pete-by to this:

load("module1.mjs");
console.log(square(11)); // 121
console.log(diag(4, 3)); // 5

It fails due to ReferenceError: square is not defined
Can I somehow use the load() function to load additional modules during runtime?

[Lzw2016]

Request support for custom file system examples
graal-sdk-20.0.0.jar org.graalvm.polyglot.io.FileSystem
Context.newBuilder(...).fileSystem(new CustomFileSystem()) Don't work

The module cannot be initialized assert moduleRecord.getStatus() == Status.Instantiated;

org.graalvm.polyglot.PolyglotException: java.lang.AssertionError

    at com.oracle.truffle.js.parser.GraalJSEvaluator.innerModuleEvaluation(GraalJSEvaluator.java:583)
    at com.oracle.truffle.js.parser.GraalJSEvaluator.innerModuleEvaluation(GraalJSEvaluator.java:595)
    at com.oracle.truffle.js.parser.GraalJSEvaluator.moduleEvaluation(GraalJSEvaluator.java:554)
    at com.oracle.truffle.js.parser.GraalJSEvaluator$2.evalModule(GraalJSEvaluator.java:280)
    at com.oracle.truffle.js.parser.GraalJSEvaluator$2.execute(GraalJSEvaluator.java:273)

[wirthi]

We have moved further regarding ECMAScript module support. I am closing this issue, if you have concrete problems on the latest GraalVM 20.0.0, please open a new ticket.

[sparkyspider]

Can we link to a working example of how to import ES modules please? Or documentation?

[eleinadani]

@sparkyspider yes, we will add a few more examples and expand the documentation on ES modules. Thank you.