Moor: Version for the Dart VM

[simolus3]

Maybe we eventually can ship a version of moor that works on the Dart VM by using dart:ffi.

[Mike278]

https://github.com/putraxor/sqlite3_ffi slightly improved version of the Dart sample SQLite ffi project

[simolus3]

There is a working VM version on the ffi branch. It still needs some smaller improvements, but all features are supported.
In package:moor/moor_vm.dart , there is VmDatabase(file) and VmDatabase.memory() which can be used instead of a FlutterQueryExecutor.

[rodydavis]

So how to i get the FFI support in Flutter? I am using the experimental branch of moor. and Setting it up correctly but it doesn't compile. Which version of Dart/Flutter is needed to make this work? and which branch of Flutter? Currently im on the master branch

[simolus3]

I didn't try the FFI version on Flutter yet. The ffi branch works on my Dart installation (currently 2.4.1), but I know there are breaking changes on dart:ffi in the next version, which Flutter on master might have already picked up. I'll try to get it running on the master channel of Flutter now and I'll let you know when it works.

[simolus3]

@AppleEducate (and everyone else, of course), I just managed to get the ffi version running on the latest Flutter master version. To use the ffi version, your pubspec should look like this:

dependencies:
  moor: ^1.7.0 # we don't need moor_flutter when using FFI
  flutter:
    sdk: flutter

dev_dependencies:
  moor_generator: ^1.7.0
  build_runner:
  flutter_test:
    sdk: flutter

dependency_overrides:
  moor:
    git:
      url: git://github.com/simolus3/moor.git
      ref: ffi
      path: moor/

If you already used the git dependency and get compilation errors, try flutter packages upgrade. You need to use commit 78bb23a or later. Next, you would have to change your build setup so that the compiled apps include sqlite. For my example project, I pretty much copied the setup from this repository.

Finally, replace the moor_flutter import with package:moor/moor.dart and package:moor/moor_vm.dart. The VMDatabase class can be used instead of FlutterQueryExecutor.

Optionally, you might want to override moor_generator to a version that is consistent with the moor version.

dependency_overrides:
  moor_generator:
    git:
      url: git://github.com/simolus3/moor.git
      ref: ffi
      path: moor_generator/

Also, please remember that the ffi branch is based on develop and contains some changes that might be unstable.

[rodydavis]

Thank you so much! I’ll try this now 😁

[rodydavis]

With the new Flutter 1.9 released and better support for dart:ffi does this change anything?

[simolus3]

Not (yet), I'm afraid. There are two aspects currently blocking this:

1. The changelog of Dart 2.5 considers ffi a "preview" and the docs still point out that breaking API changes in dart:ffi are expected. Personally, I don't think they will introduce breaking changes after making an announcement like that, but it could happen. I guess we can work around this by releasing a package which requires a sdk version >=2.5.0-dev <2.6 and update our code if a breaking change is introduced.
2. AFAIK it's not entirely clear how we would publish the sqlite binary on pub. We can probably just make the moor_flutter package a flutter plugin and include the native code via gradle or cocoa pods. It seems like there are no plans to support shipping native libraries with VM apps (https://dartbug.com/36712) though.

The second point seems annoying for Desktop applications, but I hope it's not going to be a problem when we put the code importing dart:ffi in the moor package and then make moor_flutter a dummy package which just exports moor and bundles the native library. As sqlite is installed on virtually all desktop computers, they would probably be able to use the VMDatabase without a special setup. On phones, the library would be available because we're bundling it via moor_flutter.

I'll try to get something stable that supports ffi on the beta branch this weekend or next week. You would have to use a bunch of dependency_overrides in your pubspec to use it, but no special build setup should be required. I'll let you know when it's done!

[rodydavis]

Ok so good point..

I current have a project shared where it dynamic imports the web VMDatabase for flutter web and the flutter_moor for iOS/Android.

The only reason I was looking at FFI was for desktop, but you said it should already have SQLite. What different approach could I take for desktop? Maybe you could do moor_vm, moor_flutter, moor_desktop

[simolus3]

I think we'll keep the code that uses ffi in the main moor package - it is separated from the rest of the library so tree-shaking will get rid of it in projects that don't use it.

What different approach could I take for desktop?

I'm not sure what you mean with that. Let's say sqlite is available on all desktop computers your app runs on. In that case, you would just depend on moor and use a VMDatabase instead of a FlutterQueryExecutor - no further work is required.

If you want to be 100% sure sqlite is available, you would probably have to bundle it yourself (moor sadly can't help you there because we can't distribute native libs via pub). We currently open the DynamicLibrary in moor, but I think we can expose an api that lets users provide their own opening behavior.
I've never used Flutter for Desktops apps, so I'm not sure how they're packaged. But if you ship a compiled version of sqlite with your app when releasing, moor would probably expose an api that lets you return DynamicLibrary.open('relative/path/to/the/bundled/sqlite.so') (or dll or dylib, depending on the platform). Would that work for your use case?

[rodydavis]

I will create an example Flutter project today that runs mobile, web and desktop and
I think it will make more sense what I’m talking about.

The VMDatabase may work on the desktop. I’ll check and see. It works really well on the web.

[rodydavis]

Ok so here is an example of it working across a Flutter web and mobile project. Using correct packages at compile time. https://github.com/AppleEducate/moor_shared

It should be noted all flutter projects now include a web folder. Maybe an approach like this could work for flutter_moor.

I will work on the desktop now

[rodydavis]

OK i pushed the desktop part.

This is the part i am trying to figure out:

import 'dart:io';

import 'package:moor_flutter/moor_flutter.dart';
import 'package:moor/moor_vm.dart';

import '../database.dart';

Database constructDb({bool logStatements = false}) {
  if (Platform.isIOS || Platform.isAndroid) {
    return Database(FlutterQueryExecutor.inDatabaseFolder(
        path: 'db.sqlite', logStatements: logStatements));
  }
  if (Platform.isMacOS || Platform.isLinux) {
    final _path = Directory('${Platform.environment['HOME']}/.config');
    final _file = File('$_path.sqlite');
    return Database(VMDatabase(_file, logStatements: logStatements));
  } else if (Platform.isWindows) {
    final _path = Directory('${Platform.environment['UserProfile']}\\.config');
    final _file = File('$_path.sqlite');
    return Database(VMDatabase(_file, logStatements: logStatements));
  }
  return Database(VMDatabase.memory(logStatements: logStatements));
}

When running on MacOS i get

[ERROR:flutter/lib/ui/ui_dart_state.cc(148)] Unhandled Exception: SqliteException: unable to open database file, unable to open database file

If i comment out the path the data base works but only in memory.

import 'dart:io';

import 'package:moor_flutter/moor_flutter.dart';
import 'package:moor/moor_vm.dart';

import '../database.dart';

Database constructDb({bool logStatements = false}) {
  if (Platform.isIOS || Platform.isAndroid) {
    return Database(FlutterQueryExecutor.inDatabaseFolder(
        path: 'db.sqlite', logStatements: logStatements));
  }
  // if (Platform.isMacOS || Platform.isLinux) {
  //   final _path = Directory('${Platform.environment['HOME']}/.config');
  //   final _file = File('$_path.sqlite');
  //   return Database(VMDatabase(_file, logStatements: logStatements));
  // } else if (Platform.isWindows) {
  //   final _path = Directory('${Platform.environment['UserProfile']}\\.config');
  //   final _file = File('$_path.sqlite');
  //   return Database(VMDatabase(_file, logStatements: logStatements));
  // }
  return Database(VMDatabase.memory(logStatements: logStatements));
}

[simolus3]

On the develop branch, we now have the new moor_ffi package which contains both the relevant Dart code and the build setup for native integration. The reason I've moved this functionality out of moor into moor_ffi is that we have to use a pretty tight environment.sdk constraint in the pubspec (only Dart 2.5.x is allowed) and that I'd like to keep features that depend on a "preview" api out of the main package.

At the moment, we only support Android out-of-the box, I'll add support for iOS and macOS later next week when I have access to a Mac. On these platforms, you will be able to use moor_ffi as a regular Flutter plugin without changing your build setup.
Also, neither moor or moor_ffi depends on Flutter, so this can also work for standalone Dart apps or for unit tests, but only if sqlite is already available on your system (we can't bundle it without flutter).

So this is the current overview of supported platforms:

• Android: Supported. I noticed that on the x86 and x86-64 abis, the flutter.so files are missing, so this only supports arm7 and arm8-64 at the moment. I will look into that. We compile sqlite from source to bundle it, which requires the Android NDK to be installed.
• iOS: Not supported at the moment, hopefully I get to do that soon
• macOS: Same, but it should already work because sqlite is available as a system library
• Linux: Will probably not be supported soon without further setup because the Flutter tool doesn't pick up these dependencies automatically. However, sqlite is available on virtually all distros, so the current version does work already.
• Windows: What tooling is concerned, it's on the same state as Linux, but I'm not sure whether sqlite is available without us bundling it.

To use this, add this to your pubspec:

dependencies:
  moor: any
  moor_ffi: any

dev_dependencies:
  moor_generator: any  # and build_runner, etc.

dependency_overrides:
  moor_ffi:
    git:
      url: git://github.com/simolus3/moor.git
      ref: develop
      path: moor_ffi/
  moor:
    git:
      url: git://github.com/simolus3/moor.git
      ref: develop
      path: moor/
  sqlparser:
    git:
      url: git://github.com/simolus3/moor.git
      ref: develop
      path: sqlparser/

Finally, an open question is whether we should call sqlite functions on a background isolate. Sending loads of data back and forth between isolates has a considerable performance impact, but if we load sqlite on the main isolate we'd make synchronous reads/writes to the file system on that isolate... Probably best to make it a configurable parameter.

[mit-mit]

Hey guys, very very cool that you are working on this!

The changelog of Dart 2.5 considers ffi a "preview" and the docs still point out that breaking API changes in dart:ffi are expected. Personally, I don't think they will introduce breaking changes after making an announcement like that, but it could happen.

Breaking API changes will happen. You can browse through the work items under MVP in https://github.com/dart-lang/sdk/projects/13#column-4690637. There are some of these we can't make just yet, as we plan on using extension methods.

I don't mention this to discourage you, but just to be fully transparent.

[simolus3]

Thanks @mit-mit, good to know that! I will put appropriate warnings in the README if we release this before dart:ffi gets stable.

---

I just tried to run @AppleEducate's moor_shared project with the new moor_ffi package and it worked on iOS and macOS without changing the build setup. I'll make some more changes here so that the migration is easier and then open a PR.

For now, it seems like we can just DynamicLibrary.open('libsqlite3.dylib') on both platforms. In the future, we should probably use DynamicLibrary.executable(), but that's not available on the Dart 2.5 release yet.

[simolus3]

Just released moor_ffi version 0.0.1 on pub!

As the readme points out, all warnings still apply: It's an experimental project, it may not work on non-stable Dart versions and shouldn't be used on production apps yet. Feel free to try it out though! I appreciate any kind of feedback. If you run into any problems, please open another issue.

[rodydavis]

Thank you so much 😎

[simolus3]

FYI for those of you using dev releases of Dart 2.6 or non-stable Flutter versions.

I've just published version 0.1.0-dev.1 of moor_ffi which supports Dart 2.6.0-dev.8.0, where most breaking changes were introduced. If you're only using public apis of moor_ffi, there are no breaking changes. Work on that version is happening on ffi-dart-2-6 branch, which I'll merge to master as soon as Dart 2.6 gets released.

[rodydavis]

Thanks @simolus3!