Moor: Running Moor in an isolate

[cadaniel]

I'm wondering if it's possible to use moor in an isolate. I'd like to fetch data from my server in the background, and then sync it with the database locally.

Would this be possible?

[simolus3]

I've never tried to run moor in another isolate, but there are some limitations that come to my mind:

1. We use sqflite as a backend, which uses Flutter platform channels. I'm not sure if those can be run on a background isolate, but it should be straightforward to find out. We're also going to switch from sqflite to a dart:ffi based implementation in the future (which could improve things), but there is no ETA yet.
2. Moor uses zone values to manage transactions, this will definitely not work _across_ isolates. As long as every interaction with moor is on the same isolate (even in a background isolate), that shouldn't be a problem. But sending a database object across isolates and operating on both of them will cause major problems.

I'd like to fetch data from my server in the background

You mean in a background isolate so that the main thread doesn't experience UI lags during json serialization? Or entirely as a background process (like a background service that operates while the app is closed)? For the first scenario it probably makes sense to keep moor in the main isolate as it doesn't do a lot of calculation, you could send the data from the background isolate to the main isolate and use that one to store it in moor. If you need a background-running app, it depends on whether sqflite supports that or not. I don't see a reason why that shouldn't be possible though. The only problem is that the database code should really only ever run on one isolate, so maybe some additional synchronization logic is required here.

[cadaniel]

Yeah. What I'd like to do is queue up bulk data updates and update them into the DB. I'd ideally like to have the DB operations done on a background thread, so the next time they're accessed they're updated.

AFAIK you cant use platform channels in isolates, so that could be a problem with this, but maybe with dart:ffi this becomes possible. I guess I'll have to do all the networking and json processing and maybe I can throw the objects to save across the isolate.

[simolus3]

I'd ideally like to have the DB operations done on a background thread, so the next time they're accessed they're updated.

The experimental moor_ffi package will support background isolates in two ways:

1. Background database, foreground application code. The queries are sent from the main (or any other) isolate and handled on the background isolate.
2. Entirely background: Just create your entire database instance on a background isolate

We still don't support sharing a database instance across isolates, that would become problematic with features like transactions which require a lock.

maybe I can throw the objects to save across the isolate

The background mode of moor_ffi isn't doing much more than that, unfortunately. We can't share ffi objects across isolates, so we just run the synchronous db in a background isolate and then send the data via message passing. It works and reduces load on the main thread, but overall it's going to be much much slower.

[jaumard]

@simolus3 how would you manage this use case:

We have a huge synchronization of data with our backend, so we do https call/parsing paginated and upsert the result in the local database, on old device this synchronization of data slow down the app. My idea was to put all that synchronization into an isolate to do safe in the main one, but the main one has also read/write access to the database for the 'light' UI actions.

Is it possible to have two connexion to the database ? one per isolate, or maybe you have another idea on how to manage this ? But this use case look quite common to me (data sync between app and backend)

[simolus3]

Sharing a database instance over isolates will definitely break something (at least the stream queries, which aren't synchronized over isolates).

If you use the ffi implementation, you could have a setup with three isolates:

• an isolate for database actions
• an isolate for synchronization
• the ui isolate

You would perform all database operations on the same isolate (use the VmDatabase with background: false because you take care of the isolates yourself). It would probably require a lot of boilerplate code to setup the database calls then, as the instance can't be shared :/

I can think of an API that would make things a lot easier here, but it probably requires some internal refactoring in moor. I'll play around with some approaches, but I can't guarantee anything soon.

[jaumard]

No problem @simolus3 ! Thanks for taking the time to answer ! Your idea is quite nice actually ! For sure for now it will involve a lot of boilerplate but it's nice anyway :)

If moor can support this out of the box that would be a huge advantage imho ^^ but I can imagine that it's a lot of work and take time to be implemented

[simolus3]

I now have a general idea for an isolate based api and how it could be implemented in moor. We add a new MoorIsolate (name pending) class and two methods:

• MoorIsolate.inCurrent() uses the current isolate as the one where queries will be executed
• MoorIsolate.spawn() spawns another isolate to run queries

The returned MoorIsolate is a simple object that can be sent across isolates - it only contains the internal SendPort necessary to connect. It also contains a connect() method which can be used to obtain the QueryExecutor, StreamQueryStore and SqlTypeSystem used by the higher-level methods. All of those are internally synchronized over isolate communication without requiring additional work from the user.

So, a typical use-case could look like this:

@UseMoor
class MyDatabase extends _$MyDatabase {
  MyDatabase(DatabaseConnection connection): super.fromConnection(connection);
  // ...
}

void main() async {
  final background = await MoorIsolate.spawn(_openConnection);
  final database = MyDatabase(await background.connect());
}

DatabaseConnection _openConnection() => DatabaseConnection.fromExecutor(VmDatabase(...));

In scenarios where you're using more than two isolates (e.g. one for UI, one for networking and one for the database), you could also send the MoorIsolate instance over a SendPort for the networking isolate to use.

I've started make some internal adjustments so that this api can be supported, but it will take some time to finish. I'm open to any suggestions for the api of course.

[simolus3]

The api is ready on the develop branch and will be a part of the next moor release. Here is the current version of the documentation: https://develop--moor.netlify.com/docs/advanced-features/isolates/
It takes some work to setup, but then you can seamlessly use the database like you would without the isolate api. It also works across many isolates: If you perform an update in one isolate, query streams in all isolates would update etc.

[VadimOsovsky]

@simolus3 waiting for the next release! any updates on when the next version is coming?

[simolus3]

I plan on releasing this weekend :)

Edit: Moor 2.1.0, which I've just released, contains this feature. Docs are available over here

[rohansohonee]

Hi @simolus3

Exception has occurred.
FlutterError (ServicesBinding.defaultBinaryMessenger was accessed before the binding was
initialized.
If you're running an application and need to access the binary messenger before `runApp()` 
has been called (for example, during plugin initialization), 
then you need to explicitly call the `WidgetsFlutterBinding.ensureInitialized()` first.

If you're running a test, you can call the `TestWidgetsFlutterBinding.ensureInitialized()` 
as the first line in your test's `main()` method to initialize the binding.)

• I am following the docs and trying to access getApplicationDocumentsDirectory.
• I also included void main() async { WidgetsFlutterBinding.ensureInitialized(); as the error said to do yet it throws an error.
• It seems the method you have provided in the docs is not working anymore.

[rohansohonee]

Hi @simolus3

I think you need to add WidgetsFlutterBinding.ensureInitialized(); to moor_ffi when we are trying to spawn an isolate.

[simolus3]

Neither moor nor moor_ffi has a dependency on Flutter, so it can't use the binary messenger or initialize the widgets binding. I'll take a look at what's going on here, thanks for the report!

[rohansohonee]

So now how to access database from application documents directory??

[simolus3]

I couldn't reproduce the problem you posted earlier, even if I call MoorIsolate.spawn before runApp.

So now how to access database from application documents directory??

You could use the path_provider package together with a LazyDatabase:

import 'package:path_provider/path_provider.dart';
import 'package:path/path.dart' as p;

import 'package:moor/moor.dart';

DatabaseConnection createConnection() {
  final db = LazyDatabase(() async {
     final appDocDir = await getApplicationDocumentsDirectory();
     final file = File(p.join(appDocDir.path, 'db.sqlite'));
     return VmDatabase(file);
  });
  return DatabaseConnectiion.fromExecutor(db);
}

If you've shipped a version built with moor_flutter before, I'd recommend to use the approach shared here to make sure you're always using the same file as FlutterQueryExecutor.inDatabasePath. For new projects using moor_ffi, I'd recommend to use path_provider.

[rohansohonee]

I am on the latest Flutter version.
Which one are you on??

[rohansohonee]

I am using the exact code.
I get an error on getApplicationsDocumentDirectory

[rohansohonee]

Should i downgrade my Flutter to a previous version??
My current version is 1.12 i.e it is the stable channel

[simolus3]

I was on the stable channel at 1.12 as well, so no downgrading is necessary.

I get an error on getApplicationsDocumentDirectory

Ouch, I think I get the problem now - calling getApplicationsDocumentDirectory on a background isolate is not going to work, since it uses method channels internally. Method channels aren't available for background isolates. It's possible to fix this, but it requires some more code and knowledge about how isolates work in Dart. The idea here is that we choose the database path in the main isolate. We then start the background isolate ourselves and pass the path to its init function:

Future<MoorIsolate> _createMoorIsolate() async {
  // this method is called from the main isolate. Since we can't use
  // getApplicationDocumentsDirectory on a background isolate, we calculate the
  // database path in the foreground isolate and then inform the background
  // isolate about the path.
  final dir = await getApplicationDocumentsDirectory();
  final path = p.join(dir.path, 'db.sqlite');
  final receivePort = ReceivePort();

  await Isolate.spawn(
    _startBackground,
    _IsolateStartRequest(receivePort.sendPort, path),
  );

  return (await receivePort.first as MoorIsolate);
}

void _startBackground(_IsolateStartRequest request) {
  // this is called from the background isolate! Let's create the database
  // from the path we received
  final executor = VmDatabase(File(request.targetPath));
  // we're using MoorIsolate.inCurrent here as this method already runs on a
  // background isolate. If we used MoorIsolate.spawn, a third isolate would be
  // started which is not what we want!
  final moorIsolate = MoorIsolate.inCurrent(
    () => DatabaseConnection.fromExecutor(executor),
  );
  // inform the starting isolate about this, so that it can call .connect()
  request.sendMoorIsolate.send(moorIsolate);
}

class _IsolateStartRequest {
  final SendPort sendMoorIsolate;
  final String targetPath;

  _IsolateStartRequest(this.sendMoorIsolate, this.targetPath);
}

You can then connect to that isolate via

final isolate = await _createMoorIsolate();
final connection = await isolate.connect();

YourDatabaseClass.connect(connection);

I imagine this can be a pretty common and non-obvious error, so I'll expand the docs on how to handle this. In the meantime, here is the mini-project where I use this strategy.

[rohansohonee]

Hey @simolus3

You are right. Currently isolates cannot use method channels.
There is a plugin called flutter_isolate which uses FlutterNativeView to allow method channels in an isolate.
You can check that out and see if it can be implemented in your package.

I will also try implementing your solution.

[rohansohonee]

Hi @simolus3

I have implemented your solution and this is what I have discovered:

• Firstly the issue has been resolved but when updating my table from background isolate does not seem to synchronize changes to ui isolate where i am watching my table.
• Secondly how to shutdown the isolate? It continues running even when i call db.close()!!

I know db.close() just releases the resources of the database but your docs have not mentioned on how to shutdown the isolate.

[rohansohonee]

Any idea on how to access database in a custom isolate that was spawned say using the compute function.

[simolus3]

There is a plugin called flutter_isolate which uses FlutterNativeView to allow method channels in an isolate.

Both moor and moor_ffi can't depend on Flutter or any Flutter-only package. The package looks interesting though, maybe it can make some setup operations easier for users.

updating my table from background isolate does not seem to synchronize changes to ui isolate where i am watching my table

I'll take a look at that, thanks. Moor has a synchronization mechanism for updates across isolates, so in theory this should work.

Secondly how to shutdown the isolate

You can use await MoorIsolate.shutdownAll(), which also terminates the background isolate.

Any idea on how to access database in a custom isolate that was spawned say using the compute function.

You could pass the MoorIsolate instance along the message parameter. So maybe writing a wrapper class that contains the original message and the MoorIsolate to use could work. The compute isolate would then create it's own DatabaseConnection and use that to create the database via Database.connect.

[rohansohonee]

Hi @simolus3

Can I start a moor isolate inside the compute isolate??
Will synchronisation work??

Also I will try passing the moor isolate instance as you have mentioned above and test whether synchronisation across isolates is actually working or not.

[rohansohonee]

Hi @simolus3

Only primitive types can be passed, so passing MoorIsolate instance is not possible.
Can you provide a solution for this use case???

[simolus3]

Can I start a moor isolate inside the compute isolate??

In the compute isolate, you could use MoorIsolate.current instead of MoorIsolate.spawn. But I don't think that's how the compute() function is meant to be used: AFAIK, it's designed to run calculations that eventually finish. As soon as a moor isolate is started, it will basically run an infinite loop until stopped. This is so that any other isolate can connect to a MoorIsolate at any time.

Will synchronisation work??

I just added a unit test to verify that synchronization works across multiple isolates (8987da453b21b0364ae40f36b4971d4449f9f812). In my scenario, there are three isolates:

1. the main isolate running the test, this is also the one listening to a stream
2. the MoorIsolate
3. a background isolate connecting to the 2nd isolate to execute a write

So synchronization across isolates seems to work in moor. If you share more details about how you exchange the MoorIsolate, I can help more.

Only primitive types can be passed, so passing MoorIsolate instance is not possible.

That's not true - most Dart objects can be sent over isolates. The difference is that, if an object is not a Capability, it will be copied. The MoorIsolate class was designed with this in mind, so it can find the right isolate to connect to even when sent across a SendPort. We have many tests passing a MoorIsolate across isolates.