Sdk: Ambiguous rules for Isolate message (Illegal argument in isolate message)

[duzenko]

Consider the following (originally flutter) code

class ExternalClass {
//  final VoidCallback _internalClosure = () => print('zxc');
  anyMethod() => print('asd');
}

class Worker {
  final ExternalClass externalReference;
  final SendPort sendPort;
  Worker({this.externalReference, this.sendPort});
}

void backgroundWork(Worker worker) async {
  worker.externalReference?.anyMethod();
  worker.sendPort.send('qwe');
}

Future main() async {
  final externalObject = ExternalClass();

  ReceivePort receivePort = ReceivePort();
  Worker object1 = new Worker(
    externalReference: externalObject,
    sendPort: receivePort.sendPort,
  );
  await Isolate.spawn(backgroundWork, object1);
  receivePort.listen((data) {
    print(data);
  });
}

It works as intended.
Now uncomment the line with the _internalClosure and now it throws an exception:

E/flutter (29643): [ERROR:flutter/lib/ui/ui_dart_state.cc(148)] Unhandled Exception: Invalid argument(s): Illegal argument in isolate message : (object is a closure - Function '<anonymous closure>':.)
E/flutter (29643): #0      Isolate._spawnFunction (dart:isolate-patch/isolate_patch.dart:549:55)
E/flutter (29643): #1      Isolate.spawn (dart:isolate-patch/isolate_patch.dart:389:7)
E/flutter (29643): <asynchronous suspension>
E/flutter (29643): #2      main (package:pavement/main.dart:32:17)
E/flutter (29643): <asynchronous suspension>
E/flutter (29643): #3      _runMainZoned.<anonymous closure>.<anonymous closure> (dart:ui/hooks.dart:229:25)
E/flutter (29643): #4      _rootRun (dart:async/zone.dart:1124:13)
E/flutter (29643): #5      _CustomZone.run (dart:async/zone.dart:1021:19)
E/flutter (29643): #6      _runZoned (dart:async/zone.dart:1516:10)
E/flutter (29643): #7      runZoned (dart:async/zone.dart:1500:12)
E/flutter (29643): #8      _runMainZoned.<anonymous closure> (dart:ui/hooks.dart:221:5)
E/flutter (29643): #9      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:305:19)
E/flutter (29643): #10     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)
E/flutter (29643): 

What exactly rule does this code with a private closure breaks? Why is the message so ambiguous?
What classes can we and what can we not send into an isolate as a message? Where is it stated in the docs?

Note that the code above is the just a repro case. The real use case is much more complicated and took a significant effort to debug and analyze, as well as to create a repro case starting from a business flutter app.

The exception is thrown from the 'external' code that is already a head bump as it's not easy to see what condition fails. Where do I go looking for the source code for that part? Supposedly this is a dart VM?

[srawlins]

I'd guess that at a minimum, closures are not allowed. I'll let the VM team answer more generally.

[duzenko]

I'd guess that at a minimum, closures are not allowed. I'll let the VM team answer more generally.

Can't I access the externalObject instance via a global variable or a static field? What is the point of this limitation?

[a-siva]

The VM currently only allows closures of top level methods or static methods in a class to be sent in isolate messages. This is currently not documented here https://api.dartlang.org/stable/2.6.0/dart-isolate/SendPort/send.html, we should improve that documentation.

[duzenko]

@a-siva
I'm sending an object instance, not a closure here.

[a-siva]

Sending an object entails serializing all it's fields, you are sending object1 which is a Worker object, one of it's fields is externalReference which is of Type ExternalClass and one of the fields of ExternalClass is a closure (_internalClosure).

[duzenko]

Why serialize at all here? I might want to send a ByteBuffer with 100MB of data to an isolate. What will happen to it?
Are all objects linked to the message via member references get serialized as well?
Does it mean that all changes made to passed objects are lost to the caller?
What if I want to do some parallel processing on a single block of data?
Does this 'serialization' happen on the main thread? How is it implemented? Where is its source code?
Is there any way I can get 'regular' SMP in Dart? I.e. just use what the underlying platform (Android/iOS) already supports.

[hutchings90]

@duzenko
I tore my hair out trying to figure this out. What solved my issue is realizing that I could call .then((Isolate isolate) { ... }, thereby having access to my object's code.

I have no idea why a static or top-level function is required. Maybe it provides some useful capabilities, but, in my case, it was completely useless and misleading. In order to avoid that error, I just made a top-level function that has an empty body. Let me know if this helped, if you have questions, or if you need further clarification.

Here is an example of what I did:

topLevelSpawnEntryPoint(Map<dynamic, dynamic> message) {}

class MyState extends State<MyPage> {
    @override
    void initState() {
        Isolate.spawn(topLevelSpawnEntryPoint, {}).then((Isolate isolate) => proveItWorked());
    }

    proveItWorked() {
      print('It worked');
    }
}

[duzenko]

@duzenko
I tore my hair out trying to figure this out. What solved my issue is

I think your case is different. You need a completion notification but I want to send an object reference that has complicated links to other objects which leads to unwanted serialization error.

[rdnobrega]

@duzenko Hi, I had a somewhat similar issue. My workaround was to use good old class derivation. If you came up with another solution I would be happy to hear. Here's the code:

````
import 'dart:async';
import 'dart:isolate';
import 'dart:io';

void main() async {
var manager = await WorkManager.start((data) {
stdout.write('[MAIN] RECEIVED: $data\n');
}, Miner());

await Future.delayed(Duration(seconds: 1));
manager.send('sending message');
}

//Wrapper class for duplex comm
class WorkChannel {
ReceivePort receiver;
SendPort sender;
//Just a wrapper
send(dynamic data) => sender?.send(data);
bool get isChannelReady => sender == null;
}

//Manager - the parent thread part of the communication
class WorkManager extends WorkChannel {
Isolate workerTask;
Function(dynamic) onMessageReceived;

//setup duplex comm. receives sendport from minion
setupMessage(dynamic data) {
if (data is SendPort) {
sender = data;
} else {
onMessageReceived(data);
}
}

//creates a manager and registers a callback for when mainthread receives messages on the minion worker
static Future start(
Function(dynamic data) onMessageReceived, Worker minion) async {
var manager = WorkManager();
manager.onMessageReceived = onMessageReceived;
manager.receiver = ReceivePort()..listen(manager.setupMessage);
minion.sender = manager.receiver.sendPort;
manager.workerTask = await Isolate.spawn(_createWorkerTask, minion);
return manager;
}

//isolate-thread code. Send back a receiver for the manager
static _createWorkerTask(Worker worker) {
worker.receiver = ReceivePort()..listen(worker.messageReceived);
worker.send(worker.receiver.sendPort);
worker.work();
}
}

//boilerplate code to be overriden
abstract class Worker extends WorkChannel {
work();
messageReceived(dynamic data);
}

//example class
class Miner extends Worker {
int id = 1;
int counter = 0;
@override
messageReceived(data) {
stdout.write('[MINER #$id] received: $data\n');
}

@override
work() {
Timer.periodic(Duration(seconds: 1), (timer) {
send('Miner #$id working at ${counter++}');
});
}
}
```

[szotp]

Documentation is still confusing:

https://api.dart.dev/stable/2.6.0/dart-isolate/SendPort/send.html

it is also possible to send object instances (which would be copied in the process). This is currently only supported by the dart vm.

So it works in only command line apps? Only in flutter's debug mode? Does it work in the browser?
https://flutter.dev/docs/cookbook/networking/background-parsing#notes-on-working-with-isolates

Isolates communicate by passing messages back and forth. These messages can be primitive values, such as null, num, bool, double, or String, or simple objects such as the List in this example.

So I guess flutter supports it fully, even o the web?

Also, how this can work without reflection in AOT mode?

[mraleph]

@szotp

So it works in only command line apps? Only in flutter's debug mode? Does it work in the browser?

As the comment states: it works everywhere where Dart VM is used. Which is in CLI and native Flutter applications irrespective of mode (debug, profile and release all use Dart VM).

@mit-mit do we have some term to replace Dart VM in that comment. Should we rewrite it to say "Dart Native Runtime"?

So I guess flutter supports it fully, even o the web?

Flutter Web is not a released thing - so documentation is not updated to reflect its limitations. Consider filing a bug on Flutter repo.

Also, how this can work without reflection in AOT mode?

Similarly to how GC works - you don't need to know a lot of things (e.g. field names), you need to know just how many fields are there and which ones are pointers and which ones are primitives.

[szotp]

Thanks for explanation, it makes sense now. I raised this topic because I've seen multiple people confused by this.

I think this method requires a good detailed explanation, including what can't be copied, what's the performance of copying, in which environments Dart VM is not used (Is this only in regular Dart transpiled to js?), etc.

[mit-mit]

do we have some term to replace Dart VM in that comment.

Yes, I'd use the term 'Dart Native platform' as on https://dart.dev/platforms