Dexie.js: Is there a way to get only part of the document? [Maximum IPC message size exceeded]

[Bnaya]

I have documents that one of the field is big.
When the result is alot of rows, the the operation is failed with Maximum IPC message size exceeded error.
Is there a way to exclude some of the fields on the query?

[dfahlander]

IndexedDB has no easy way of selecting a subset of properties, but it is doable using compound keys (Strategy B below).

That said, the issue could also be due to that toArray internally calls IDBIndex.getAll() which might result in too heavy load for IndexedDB. So it could probably be worked around by avoiding .toArray() and use each instead and build up the result manually without including the large property (This is strategy A below)

Stragegy A - avoid IDBIndex.getAll() and use each instead.

Force fetching query results one by one, using Collection.each and at the same be sparse with the total memory allocation

function map (coll, mapperFn) {
    var result = [];
    return coll.each(row => result.push(mapperFn(row))).then(()=>result);
}
// Call it like this
var coll = db.docs.where('queryField').equals(queryValue);
map(coll, doc => ({wantedField1: doc.wantedField1, wantedField2: doc.wantedField2}))
    .then(result => {
        console.table(result);
    })
    .catch(err => console.error(err.stack || err));

Stragegy B - natively select subset of fields

This strategy would maybe be even more optimized than Strategy A. In case Strategy A takes too long time to execute, consider this one instead.

This strategy requires that you've defined a compound index that contains both the index you want to query as well as the fields you want to return. You can use Collection.keys() to only get the keys you are querying, and if that key is compound and also includes the fields you want to return, you can use that index using between() instead of equals():

var db = new Dexie('dbname');
db.version(1).stores({
    docs: 'id, [queryField+wantedField1+wantedFeld2]'
});

// Then query it like this
db.docs.where('[queryField+wantedField1+wantedFeld2]')
    .between([queryValue, Dexie.minKey, Dexie.minKey], [queryValue, Dexie.maxKey, Dexie.maxKey])
    .keys(keys => keys.map(key=> ({wantedField1: key[1], wantedField2: key[2]}))
    .then(result => {
        console.table(result);
    })
    .catch(err => console.error(err.stack || err));

[dfahlander]

An alternate strategy (C) could also be to split up your heavy field into its own table and reference it from the main table. You can then use dexie-relationships addon to select whether or not to include the referred items.

[Bnaya]

Thank you very much!

BTW is there any technical limitation that there's no reduce method for Collection?

[dfahlander]

No. Pull requests welcome ;)

For map, there is a limitation in current design of Collection. A new and rewritten Collection will eventually replace current and not have this limitation.

[dfahlander]

BTW did you test strategy A or B? Did it work?

[Bnaya]

I went with A, even i think i can use A with my current indexes.
It was much simpler

[raphinesse]

I just released dexie-batch to address a similar issue in a recent project. It is brand new and documentation is still very terse, but it already serves me well in production.

It provides an each variant, that uses batched transfer in the background to cut down total transfer overhead. Give it a try and let me know what you think.