Moor: Unable to send MoorIsolate instance to isolate

[tigloo]

I'm trying to send a MoorIsolate instance to an isolate that requires database access.

The setup is as following:

• Flutter mobile app
• Main isolate instantiates the moor isolate as documented at https://moor.simonbinder.eu/docs/advanced-features/isolates/#initialization-on-the-main-thread
• A process that runs in the background (but in the context of the main isolate) requires database access
• A process that handles communication with an external device and runs in its own isolate requires database access

When trying to send the MoorIsolate instance to another isolate like this:

_toIsolate.send([IsolateCommands.receiveDatabaseIsolate.index, databaseIsolate])

I get the following error:

illegal argument in isolate message (object is a regular dart instance)

It seems that I can only send the SendPort instance encoded within ServerKey to the isolate. I tried to change Moor's code to expose the SendPort instance and only send that, but then Moor itself fails trying to establish a connection to its own isolate with the same error.

It seems like MoorIsolate cannot be sent to another isolate.

[tigloo]

...and just for clarification: we don't need the isolate for performance reasons, we only need it because the documentation states that concurrent accesses to the same database corrupt the database. I saw the moor_ffi explicitly turns thread safety off. If there is any other way to handle concurrent accesses, that would also be an option.

[rohansohonee]

I am also waiting for a solution to this problem for a long time.

[rohansohonee]

The documentation states this under:

Common operation modes

The MoorIsolate object itself can be sent across isolates, so if you have more than one isolate from which you want to use moor, that’s no problem!

One executor isolate, multiple client isolates:
The MoorIsolate can be sent across multiple isolates, each of which can use MoorIsolate.connect on their own. This is useful to implement a setup where you have three or more threads:

• The moor executor isolate
• A foreground isolate, probably for Flutter
• Another background isolate, which could be used for networking.

You can then read data from the foreground isolate or start query streams, similar to the example above. The background isolate would also call MoorIsolate.connect and create its own instance of the generated database class. Writes to one database will be visible to the other isolate and also update query streams.

Documentation link

But this does not work.

@simolus3 Please update the documentation reflecting that MoorIsolate cannot be sent over to other isolates as the following error occurs:
illegal argument in isolate message (object is a regular dart instance)

[tigloo]

A simple workaround could be to add a serialization method to MoorIsolate that just returns the used SendPort and a deserialization method that re-instantiates MoorIsolate based on that SendPort. The class essentially only wraps a SendPort.

I tried passing the SendPort inside the MoorIsolate into my isolate, which works. However, MoorIsolate.connect() from the isolate then breaks because Moor itself tries to pass the MoorIsolate instance to its own isolate. This also fails.

The first part would be a very simple PR but I'm not sure about Moor's internal communication with its own isolate.

[simolus3]

I've now changed all of moor's internal isolate communication to only use primitive objects by using a custom serializer.

With the next update, you'll also be able to obtain the underlying SendPort with MoorIsolate.connectPort. That port can safely be sent across isolates, the receiving end can reconstruct a MoorIsolate out of that by using the new MoorIsolate.fromConnectPort constructor.

So far, I couldn't find a good way to obtain an environment where primitive objects are required. If I run tests with pub run test I can send arbitrary objects, it's the same with dart2native. Does release-mode Flutter require primitive communication? I want to make sure that my tests would fail if I send a non-primitive object. If you now more about this, help would be much appreciated.

[tigloo]

Very cool, I'm looking forward to test this! Flutter seems to require primitive communication, at least that's the environment I experienced the error in. I was a bit surprised that it was unable to send a primitive that's just wrapped in another class, but maybe it just looks at the top-level type.

[simolus3]

I've now added an integration test for Flutter here. I'm running that test with flutter --release lib/moor_ffi.dart, which works for me.

If you want to, you can try to use the latest version from develop by adding this to your pubspec:

dependency_overrides:
  moor:
    git:
      url: https://github.com/simolus3/moor.git
      ref: develop
      path: moor
  moor_ffi:
    git:
      url: https://github.com/simolus3/moor.git
      ref: develop
      path: moor_ffi
  moor_generator:
    git:
      url: https://github.com/simolus3/moor.git
      ref: develop
      path: moor_generator
  sqlparser:
    git:
      url: https://github.com/simolus3/moor.git
      ref: develop
      path: sqlparser

I'll close this as it appears to be fixed, but let me know if you run into any problems on develop.

[rohansohonee]

Can the documentation be updated on how we must use the new Isolate features?

[simolus3]

I've updated the section here, it now describes how MoorIsolates can safely be shared across isolates. Is that what you need?

[rohansohonee]

I am using Daos. How must I access the database now inside a dao?

[tigloo]

This may be a stupid question but what would be the best way to test the new version? I tried changing my pubspec.yaml to point to the "develop" branch of Moor, but I'm getting a version error with sqflite.

Can I point only Moor to the develop branch and override versions to the currently released once for moor_ffi, moor_generator and sqflite? What would be a working configuration?

[kuhnroyal]

You need to override all 4 dependencies, moor, moor_generator, moor_ffi and sqlparser. And remove the sqflite direct dependency.

[tigloo]

Here's my pubspec (excerpt):

dependencies:
  moor:
    git:
      url: https://github.com/simolus3/moor.git
      path: moor
      ref: develop
  moor_ffi:
    git:
      url: https://github.com/simolus3/moor.git
      path: moor_ffi
      ref: develop
dev_dependencies:
  moor_generator:
    git:
      url: https://github.com/simolus3/moor.git
      path: moor_generator
      ref: develop

I get the following error:

Because every version of moor_generator from git depends on moor from hosted and carity_pro depends on moor from git, moor_generator from git is forbidden.
So, because carity_pro depends on moor_generator from git, version solving failed.
Running "flutter pub get" in carity-pro...                              
pub get failed (1; So, because carity_pro depends on moor_generator from git, version solving failed.)

[simolus3]

I get the following error:

You need to use dependency_overrides for every package developed in this repo. The section from my answer above should work, let me know if it doesn't.

If you don't want to use dependency_overrides, you can also use my custom pub server. I can't make any promises on availability though.

dependencies:
  moor:
    hosted:
      name: moor
      url: https://kneipe.simonbinder.eu/moor/dev
  moor_ffi:
    hosted:
      name: moor_ffi
      url: https://kneipe.simonbinder.eu/moor/dev
dev_dependencies:
  moor_generator:
    hosted:
      name: moor_generator
      url: https://kneipe.simonbinder.eu/moor/dev

In that case you don't need to override the sqlparser dependency, but I'd still recommend to use git and dependency_overrides.

I am using Daos. How must I access the database now inside a dao?

The same? Typically you would construct your dao with MyDao(database), right? That doesn't change with isolates. To construct a database attached to another isolate, you can follow the guide and use isolate.connect(). You can then access that database inside a dao like you regularly would.

[tigloo]

Apologies, I didn't see you already posted the dependency overrides above. I now have the following error when compiling:

 - 'BatchedStatements' is from 'package:moor/src/runtime/executor/executor.dart' ('../../.pub-cache/git/moor-572efd31ff29a163525d0de4bd8c4ac133b2ee8e/moor/lib/src/runtime/executor/executor.dart').
Change to a supertype of 'BatchedStatements', or, for a covariant parameter, a subtype.
  Future<void> runBatched(List<BatchedStatement> statements) async {
                                                 ^
../../.pub-cache/git/moor-572efd31ff29a163525d0de4bd8c4ac133b2ee8e/moor/lib/src/runtime/executor/helpers/delegates.dart:106:16: Context: This is the overridden method ('runBatched').
  Future<void> runBatched(BatchedStatements statements) async {
               ^
../../.pub-cache/hosted/pub.dartlang.org/moor_ffi-0.4.0/lib/src/vm_database.dart:48:32: Error: 'BatchedStatement' isn't a type.
  Future<void> runBatched(List<BatchedStatement> statements) async {
                               ^^^^^^^^^^^^^^^^
Target kernel_snapshot failed: Exception: Errors during snapshot creation: null
Failed to build bundle.
Error launching application on iPhone 6s.
Exited (sigterm)

I tried flutter clean and flutter pub cache repair but the error still comes up.

[rohansohonee]

@simolus3
Dao constructor

MyDao(MyDatabase db) : super(db);

I am actually getting an error from the editor:

db is undefined

I was earlier able to access the db instance. Now that I am using moor isolates, daos have broken.

I require the db instance to access data from different tables.
Can you provide an example on how I should do this?

[rohansohonee]

@tigloo You must override moor_ffi, replace moor_flutter with that.

[simolus3]

db is undefined

Ah right, sorry. This was another breaking change in moor 3.0. Replacing db with attachedDatabase should work. I've also added an extension to keep the old db getter with 91115311cf0d295d13144aab323f88260b051f58