Graaljs: Arrays passed to JS using ProxyArray do not mimic JS arrays

[edopalmieri]

I have a Java host application that is executing JS scripts as guest using the Polyglot API. I would like to pass Java arrays (ArrayList) to the JS side and treat the array on the JS side as if it was a "native" JS array. As suggested in this SOF post:
https://stackoverflow.com/questions/51707039/injected-members-from-the-host-language-to-arrive-to-guest-language-as-guest-lan
I am using ProxyArray to achieve that. Specifically, I am using ProxyArray.fromArray() to pass the Java ArrayList to the JS side using bindings:

Value guestLanguageBindings = context.getBindings("js"); guestLanguageBindings.putMember("arrParamName", ProxyArray.fromArray(arrayArg));

However in the JS side (guest language) the array is not seen as a JS array, instead it is of type: org.graalvm.polyglot.proxy.ProxyArray (also a Java type). This prevents me from being able to use the array as if it were a regular JS array. Statements like arr[0] work, however specific calls such as arr.sort() fail.

The screenshot below illustrate a debugging session where variable is passed as described above with the resulting error.

[wirthi]

Hi Edoardo,

thanks for your request.

The ProxyArray on the JS side currently behaves close to a JavaScript object (but not like a JavaScript Array, true). Accessing it from JS will work in most cases if you do whatever you would do with a plain JavaScript object in such a situation, e.g. calling Array.prototype.foEach.call(variable, function .... ).

The problem is that we treat the ProxyArray just like any other (interop) object; specifically, it is missing the prototype link to Array.prototype, so it does not have the sort or forEach functions. We are aware of that and are looking into improving that behaviour in the future.

Note that Array.prototype.sort is currently not supporting ProxyArray at all. I've opened a ticket, we should resolve that shortly.

Best,
Christian

[edopalmieri]

Hi Christian,

Thank you for your explanation. It makes sense, I understand now the behaviour I was seeing and I was able to do a couple of tests based on your suggestions to be able to use the ProxyArray in the JS side (calling foreach, push, etc).

With my application I am using the polyglot API to try to execute scripts provided by users, which are unaware of the polyglot requirements/limitations. As a consequence, the JS provided will not be coded as we are discussing now, but in the more "straightforward"/traditional approach (eg: myArray.foreach() instead of Array.prototype.foreach.call(myArray, function()) ).

For this purpose the improvements you are working on would be extremely useful. In the meantime I will revert to using JSON representations of arguments passed to the guest language side so that "guest language developers" can work as they are used to with the only expectation that the arguments they will receive will be in standard JSON form.

I have a feeling that would be the only way to being able to execute guest code that is "graal unaware" at this stage, especially if trying to do it with different languages too (JS, R, Python, and Ruby). Would you agree?

Once agin thank you for your explanation and impressive work with the framework.
Edoardo.

[wirthi]

Hi Edoardo,

as you seem to be in control of the Java side of things, you can also prepare a proper JSArray with code like that:

        Value jsArray = context.eval("js", "new Array();");

        jsArray.setArrayElement(0, 1001); //array will grow automatically, JS semantics
        jsArray.setArrayElement(1, 1002);
        jsArray.setArrayElement(2, 1003);

        guestLanguageBindings.putMember("variable", jsArray);
        context.eval("js", "print(variable);");

This might not be the optimal solution (performance, need to copy), but will give a 100% correct JS Array to your JavaScript code.

[edopalmieri]

Hi,

Indeed I am in control of the Java side and of course I could create the JS array already from the Java side! I just did not think about it. Sure, it may not be the most efficient solution, but I am sure it beats converting the array to JSON.

Thanks for the suggestion, I just gave it a go and it works great!

I have a slightly off topic question: when it comes to passing the array to the guest side, I am trying to use the top level bindings (instead of the polyglot bindings - simply because polyglot bindings require the guest developer to know how to retrieve them through the polyglot library). I can set members on the top level bindings in JS no problem, but I get errors when doing the same in R and Python. I am reading from the docs that some languages may not offer access to the bindings. With which languages can I use the top level bindings?

thanks again!
Edoardo.

[wirthi]

Hi Edoardo,

Can you post the code you tried with R and Python that failed?

You can either do it with context.getPolyglotBinding() or with context.getBindings("language"). For the first one, see the examples at https://www.graalvm.org/docs/graalvm-as-a-platform/embed/#computed-arrays-using-polyglot-proxies - That's one shared bindings for all the languages. They have the disadvantage that you need to explicitly import the value. The later one is the one you were using in your example above. They should work for R and Python as well, so if you can post your code, we can investigate why it's not.

thanks,
Christian

[edopalmieri]

Hi,

Sure, here are a couple of straightforward tests that show the problems I am seeing:

R

public void execRWithBindingsError() {

    Context testContext = Context.newBuilder().allowNativeAccess(true).build();

    testContext.getBindings("R").putMember("someText", "Hello there!");

    testContext.eval("R", "print(someText)");               
}

Error:

java.lang.IllegalArgumentException: Invalid member value 'Hello there!'(language: R, type: character) for object 'com.oracle.truffle.api.vm.PolyglotLanguageBindings@2f2e4bde'(language: R, type: Object) and member key 'someText'. at com.oracle.truffle.api.vm.PolyglotValue.invalidMemberValue(PolyglotValue.java:283) at com.oracle.truffle.api.vm.PolyglotValue$Interop$PutMemberNode.executeImpl(PolyglotValue.java:1931) at com.oracle.truffle.api.vm.PolyglotValue$PolyglotNode.execute(PolyglotValue.java:551) at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callProxy(OptimizedCallTarget.java:262) at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callRoot(OptimizedCallTarget.java:251) at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary(OptimizedCallTarget.java:241) at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.doInvoke(OptimizedCallTarget.java:226) at org.graalvm.compiler.truffle.runtime.GraalTVMCI.callProfiled(GraalTVMCI.java:86) at com.oracle.truffle.api.impl.Accessor.callProfiled(Accessor.java:733) at com.oracle.truffle.api.vm.VMAccessor.callProfiled(VMAccessor.java:93) at com.oracle.truffle.api.vm.PolyglotValue$Interop.putMember(PolyglotValue.java:1327) at org.graalvm.polyglot.Value.putMember(Value.java:263) at Test.execRWithBindingsError(Test.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)

Python

    public void execPythonWithBindingsError() {

    Context testContext = Context.newBuilder().allowIO(true).build();

    testContext.getBindings("python").putMember("someText", "Hello there!");

    testContext.eval("python", "print(someText)");              
}

Error:

org.graalvm.polyglot.PolyglotException: java.lang.AssertionError: Returned scope variables object must be a TruffleObject. at com.oracle.truffle.api.vm.PolyglotLanguageBindings.<init>(PolyglotLanguageBindings.java:61) at com.oracle.truffle.api.vm.PolyglotLanguageContext.initializeLanguageBindings(PolyglotLanguageContext.java:145) at com.oracle.truffle.api.vm.PolyglotLanguageContext.getHostBindings(PolyglotLanguageContext.java:119) at com.oracle.truffle.api.vm.PolyglotContextImpl.getBindings(PolyglotContextImpl.java:630) at org.graalvm.polyglot.Context.getBindings(Context.java:375) at Test.execPythonWithBindingsError(Test.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) Original Internal Error: java.lang.AssertionError: Returned scope variables object must be a TruffleObject. at com.oracle.truffle.api.vm.PolyglotLanguageBindings.<init>(PolyglotLanguageBindings.java:61) at com.oracle.truffle.api.vm.PolyglotLanguageContext.initializeLanguageBindings(PolyglotLanguageContext.java:145) at com.oracle.truffle.api.vm.PolyglotLanguageContext.getHostBindings(PolyglotLanguageContext.java:119) at com.oracle.truffle.api.vm.PolyglotContextImpl.getBindings(PolyglotContextImpl.java:630) at org.graalvm.polyglot.Context.getBindings(Context.java:375) at Test.execPythonWithBindingsError(Test.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) Caused by: Attached Guest Language Frames (0)

Indeed, I am trying not to use the polyglot bindings to avoid requiring to the "guest language developers" to have to explicitly import them.

Thanks!
Edoardo

[lukasstadler]

I looked at the R issue - there's a bug which makes it only accept R-specific data types for bindings, which will be fixed soon.
@timfel - would be great if you could take a look at the Python issue.

[edopalmieri]

Speaking of using R and Python via the Polyglot API, I noticed that Python needs allowIO() to be set to true in order to execute, while R needs allowNativeAccess() set to true.

Is it not possible to run these languages with the default settings, ie "no access" allowed, similar to how I am running JS?

The Docs do not go into great depths in explaining what happens when setting these access variables to true. How would they affect the ability to sandbox the guest code that is being executed?

[edopalmieri]

@lukasstadler , @timfel I believe the Python issue has been fixed in RC5. I was running RC4 when doing the test above. I have updated to RC5 now and re-ran the test and it is working as expected.

[lukasstadler]

JS is somewhat special in that it doesn't have a base library, i.e., a directory of modules or packages provided with the language runtime. Python and R do, and parts of their base library are used during startup - which is why they require IO access. R even runs C code during startup, so it also needs the native access permission.

It may be desirable to allow access to the base library and installed modules even without IO permissions, but since the code that loads modules/packages uses vanilla file IO, it's not that easy to determine which accesses should be allowed (and which ones denied). This is something that we're looking into, but it's not a trivial issue, considering all the different deployment scenarios.

However, as long as the native access permission is not needed, accesses to the file system can be sandboxed by implementing a custom file system which provides a view of the base library.

Sandboxed execution in the presence of native code is also something that we're working on - our plan is to provide a preview of this functionality in GraalVM EE end of September.

[edopalmieri]

Hi @lukasstadler thanks for the explanation. From what I understand, once you roll out the feature for sandboxing execution in the presence of native code in combination with a custom file system implementation we should be able to fully sandbox execution of any language?

[wirthi]

Hi @edopalmieri

it will work in languages that supports it and implements e.g. the Virtual File System. Not necessarily in "any" Truffle language. Graal.js e.g. supports it in the core language, but not in the Node.js extension at the moment.

The issue around Array.prototype.sort for Polyglot objects was solved in https://github.com/graalvm/graaljs/commit/304e908a3ae1758fe62174ca3d6ac98bd57ef783

Unless there is anything open regarding JavaScript, I'd close this issue. If there is anything open for R or Python, please consider submitting a ticket in the respective repositories.

Best,
Christian

[edopalmieri]

@wirthi thanks for the explanation.

Indeed, I have closed the issue. Thanks for your help.
Edoardo.

[p-bakker]

In https://github.com/graalvm/graaljs/issues/45#issuecomment-415390159 @wirthi said:

The problem is that we treat the ProxyArray just like any other (interop) object; specifically, it is missing the prototype link to Array.prototype, so it does not have the sort or forEach functions. We are aware of that and are looking into improving that behaviour in the future.

Has that happened yet? If not, is there a case for it I can subscribe to?

[wirthi]

Hi @p-bakker

@iamstolis is working on a related issue in #88. I'll ask him to look into the ProxyArray case as well. In any case, this will be an experimental feature for the moment (i.e., you will have to pass a flag when creating the JS engine).

-- Christian

[iamstolis]

@p-bakker, I have just pushed an experimental support for Array.prototype on the prototype chain of array-like non-JavaScript objects (including ProxyArray wrappers), see https://github.com/graalvm/graaljs/issues/88#issuecomment-453021299. Please, give it a try and let us know if it works for you.

[p-bakker]

Tnx! See it didn't make the cut for RC11. Could really use some nightly builds :-) Is that something to ask for within the context of GraalJS or is that something do be asked for in the overall GraalVM project?

[iamstolis]

FYI: js.experimental-array-prototype option that adds Array.prototype on the prototype chain of array-like non-JavaScript objects will be removed in GraalVM 1.0.0-rc14. It will replaced by js.experimental-foreign-object-prototype option. We changed the name of this option because besides adding Array.prototype it also adds Function.prototype on the prototype chain of executable non-JavaScript objects and Object.prototype on the prototype chain of all non-JavaScript objects now.

[greenbird-chr]

@iamstolis

A year ago you had a Gitter chat with a user that provided the test below.

The issue on rc13 then was that it failed for ProxyArray. You reported it worked for both on rc14 with the updated property name (js.experimental-foreign-object-prototype).

On 20.1.0 it works for ProxyArray but fails on splice on the ArrayList with TypeError: undefined is not a function. Is this a regression?

class PrototypeTest {

    private void test(final Object listObj) {
        final Engine engine = Engine.newBuilder().option("js.experimental-array-prototype", "true").build();
        final Context context = Context.newBuilder("js").engine(engine).build();
        final Value value = context.eval("js", ""
                + "(function() {"
                + "     return {"
                + "         foo: function() {"
                + "             var list = this.createList();"
                + "             console.log(list);"
                + "             list.splice(3, 1);"
                + "             console.log(list);"
                + "         }"
                + "     }; "
                + "}())");
        value.putMember("createList", (ProxyExecutable) args -> {
            return Value.asValue(listObj);
        });
        value.invokeMember("foo");
    }

    @Test
    void testProxyArray() {
        Assertions.assertDoesNotThrow(() -> {
            test(ProxyArray.fromList(new ArrayList<>(Arrays.asList("foo", "bar", "yak", "grzl"))));
        });
    }

    @Test
    void testArrayList() {
        Assertions.assertDoesNotThrow(() -> test(new ArrayList<>(Arrays.asList("foo", "bar", "yak", "grzl"))));
    }
}

[iamstolis]

@greenbird-chr

Is this a regression?

No, it is not. The configuration of the Engine and Context in your test-case is to blame. The test-case is using the obsolete name of the option ("js.experimental-array-prototype"). The correct name is "js.experimental-foreign-object-prototype". Moreover, it is an experimental option. So, you have to use

allowExperimentalOptions(true).option("js.experimental-foreign-object-prototype", "true")

instead of

option("js.experimental-array-prototype", "true")

This change ensures that testProxyArray() passes. Unfortunately, for testArrayList() you need one more change: you should allow list access in your Context i.e. you have to add something like

.allowHostAccess(HostAccess.newBuilder().allowListAccess(true).build())

into a code that builds Context.

Note that various sensitive operations (like access to fields and methods of Java objects) are disabled by default. So, when it comes to Java-interoperability it is always a good idea to try to run your code with allowAllAccess(true) to figure out whether the failure is caused by some missing permission. Of course, do not use allowAllAccess(true) in production. It should be replaced by appropriate fine-grained set of permissions.

[greenbird-chr]

@iamstolis
Thanks. allowListAccess did the trick.

[mpetzold]

The solution presented in #291 maybe also a solution to this issue without using experimental features.

[klenksn]

You could have a look at the sj4js library. It extends GraalJs with exactly these kinds of features...