Moor: Support for Web Workers

[marcgraub]

Context

Hi! I made an offline web app, it synchronizes data with an API on the background and stores it in a local database with moor.

Problem

The problem I have comes when I make inserts to the database, the UI freezes for some ms, for example, every ten minutes I check for new clients and normally, I make inserts with batches of 50.

Solution

I've seen you have been working on implementing IndexedDB as a storage engine, it would be great to give the implementation some basic support for Web Workers.

With the current implementation the window object isn't accessible from the worker, we can pass a bool to indicate the IndexedDB lives inside a worker and use WorkerGlobalScope to get access to IndexedDB.

I gonna do a PR proposal.

Another concern I have is if stream queries are going to be updated when table receives an insert in another web worker. I think not.
If not, exists some method to trigger manually the auto update of a query stream?

I made some tests with the code and web workers seems to solve my problem.

[festelo]

Hi @simolus3 Would you like to check https://github.com/festelo/moor/tree/master_worker/moor/lib/src/web/worker ?
Transactions are not implemented currently.
To make it work - this file should be compiled into js and placed somewhere in the app.

Usage:

  final storage = MoorWebStorageFactory.indexedDb(
    name,
    inWebWorker: true,
    migrateFromLocalStorage: false,
  );
  return WebDatabase.withDelegate(
      MoorWorkerClient(workerPath, sqlJsPath, storage));

inWebWorker must be true and migrateFromLocalStorage must be false.
workerPath - path to the compiled file.
sqlJsPath - path to sql-js script.

[festelo]

I made this because I had performance issue. But as i found out later, the problem was not in moor... Anyway maybe this will be helpful

[kanekv]

How does web storage work? Does it overwrite whole database on every modification?

[simolus3]

Unfortunately that's what we need to do, yes.

[kanekv]

@simolus3 thanks a lot! Is there a way to manually manage when save happens? E.g. not save it on every modification but once in a while?

[simolus3]

Sorry for the slow response. You can wrap the MoorWebStorage to avoid storing all the time:

class _OnlyStoreWhenNeeded implements MoorWebStorage {
  final MoorWebStorage inner;
  Uint8List _pending;

  _OnlyStoreWhenNeeded(this.inner);

  Future<void> save() async {
    final pending = _pending;
    if (pending != null) {
      _pending = null;
      await inner.store(pending);
    }
  }

  @override
  Future<void> close() async {
    await save();
    await inner.close();
  }

  @override
  Future<void> open() {
    return inner.open();
  }

  @override
  Future<Uint8List> restore() async {
    return _pending ?? await inner.restore();
  }

  @override
  Future<void> store(Uint8List data) {
    _pending = data;
    // nope, not doing it.
    return Future.value();
  }
}

Use it like this:

YourDatabase(WebDatabase.withStorage(_OnlyStoreWhenNeeded(MoorWebStorage.indexedDb('db'))));

Closing the database will then store its data.

[kanekv]

@simolus3 Great, thanks!

[simolus3]

In the next moor version (4.1), it will be much easier to use web workers with moor. The necessary apis are on develop now!

I started to write the documentation for this feature. As you can see, it doesn't require too much code to set up. One downside is that local storage is not available at all in workers, so you can only use IndexedDb. So you might want to check whether IndexedDb is available and only use workers if it really is.
There's an example in the repository in extras/web_worker_example/ too.

Another thing: If you use shared web workers, stream queries and updates synchronize across tabs in real time!

[julien100000]

Any prediction when version 4.1 will be available? :)

[simolus3]

I've just published moor version 4.1.0 which contains this feature. The docs explain this here, this repo also contains a working example too.