Moor: [web] - Setting the value of 'moor_db_str_db' exceeded the quota

[rodydavis]

js_helper.dart:1340 Uncaught Error: Failed to execute 'setItem' on 'Storage': Setting the value of 'moor_db_str_db' exceeded the quota.
    at alF.Kn (html_dart2js.dart:27588)
    at alF.z7 (web_db.dart:119)
    at alF.wN (web_db.dart:65)
    at acS.$0 (engines.dart:81)
    at basic_lock.dart:31
    at amU.a (async_patch.dart:305)
    at amU.$2 (async_patch.dart:330)
    at Object.r (async_patch.dart:235)
    at nv.Ue (basic_lock.dart:15)
    at nv.mY (basic_lock.dart:15)

Having issues on the latest with Flutter web, the local storage seems to be maxing out.

[rodydavis]

It is throwing here:

void _storeDb() {
    if (!isInTransaction) {
      final data = _db.export();
      final binStr = bin2str.encode(data);
      window.localStorage[_persistenceKey] = binStr;
    }
  }

I'm going to see if i can add a disable cache option to Web Database.

[rodydavis]

Im experimenting here: https://github.com/AppleEducate/moor/blob/master/moor/lib/src/web/web_db.dart

[simolus3]

Local storage quota is around 10mb for most browsers (5 for Safari WebViews, but that's still a lot for most app databases). Can you check whether the same code you're using on the web yields a database file that big on other platforms?

Maybe we have to use something like the Filesystem API where it's available, but support across Browsers is very limited. Can you also check for other content in LocalStorage that might be contributing to this?

[rodydavis]

Yeah the app db on this app can be big. But it’s the same database for every user. It’s this app https://hymns-for-worship.web.app Basically it loads I. The songs from a json file. Other than that I use local storage for a few settings

[rodydavis]

And I’ll check on mobile, it works super fast there ✅

[yshahak]

I am getting the same error. I am seeding the WebDatabase with a file from assets(I wrote a custom quick and dirty implementation of moor-web). The size of this file is 6.3Mb. This part is working just fine, but after I am doing some update querying I am running into this issue.
Specifically, this is the query I'm running:

clearLabels() async {
    return (update(lessons)
      ..where((t) => t.label.equals('').not())
    ).write(LessonsCompanion(
      label: Value(''),
    ),
    );
  }

[simolus3]

Is this still a problem when using IndexedDB? That implementation should consume less space (as we don't have to encode blobs as strings) and browsers generally allow more space for IndexedDB.

[jamie1192]

hey @simolus3 , I've come across this problem so unless I'm doing something dumb, it is indeed still an issue.

I've used Rody's 'moor_shared' project as a starting point for cross-platform Moor setup, while also referring to your answer from here as to how to use an existing db asset (11.5MB) in web, this is what i'm currently doing in my web setup:

MyDatabase constructDb({String dbName, bool logStatements = false}) {
  print("Constructing web database");
  return MyDatabase(
    WebDatabase.withStorage(
      MoorWebStorage.indexedDbIfSupported(dbName),
      logStatements: logStatements,
      initializer: () async {
        print('Initializer triggered');
        final storage = MoorWebStorage(dbName);
        // Copy from asset
        var data = await rootBundle.load(join('assets', dbName));
        final bytes =
            data.buffer.asUint8List(data.offsetInBytes, data.lengthInBytes);
        await storage.store(bytes);
        return bytes;
      },
    ),
  );
}

On latest Chrome (85.0.4183.121), i'm getting this error in the console:

Uncaught (in promise) DOMException: Failed to execute 'setItem' on 'Storage': Setting the value of 'moor_db_str_db' exceeded the quota.
    at Storage.[_setItem] (http://localhost:58446/dart_sdk.js:96301:27)
    at Storage.[dartx._set] (http://localhost:58446/dart_sdk.js:96231:21)
    at moor_web._LocalStorageImpl.new.store (http://localhost:58446/packages/moor/moor_web.dart.lib.js:132:38)
    at http://localhost:58446/packages/myProject/infrastructure/database/platform/web.dart.lib.js:31:23
    at Generator.next (<anonymous>)
    at http://localhost:58446/dart_sdk.js:37211:33
    at _RootZone.runUnary (http://localhost:58446/dart_sdk.js:37065:58)
    at _FutureListener.thenAwait.handleValue (http://localhost:58446/dart_sdk.js:32049:29)
    at handleValueCallback (http://localhost:58446/dart_sdk.js:32596:49)
    at Function._propagateToListeners (http://localhost:58446/dart_sdk.js:32634:17)
    at _Future.new.[_completeWithValue] (http://localhost:58446/dart_sdk.js:32477:23)
    at async._AsyncCallbackEntry.new.callback (http://localhost:58446/dart_sdk.js:32499:35)
    at Object._microtaskLoop (http://localhost:58446/dart_sdk.js:37326:13)
    at _startMicrotaskLoop (http://localhost:58446/dart_sdk.js:37332:13)
    at http://localhost:58446/dart_sdk.js:32851:9

[simolus3]

@jamie1192 You're using a different storage in the initializer. You could try it with this:

MyDatabase constructDb({String dbName, bool logStatements = false}) {
  print("Constructing web database");
  final storage = MoorWebStorage.indexedDbIfSupported(dbName);
  return MyDatabase(
    WebDatabase.withStorage(
      storage,
      logStatements: logStatements,
      initializer: () async {
        print('Initializer triggered');
        // Copy from asset
        var data = await rootBundle.load(join('assets', dbName));
        final bytes =
            data.buffer.asUint8List(data.offsetInBytes, data.lengthInBytes);
        await storage.store(bytes);
        return bytes;
      },
    ),
  );
}

[jamie1192]

Ohh I see, I was hoping it was just something dumb. Thanks very much!