Dexie.js: Advice on encryption

[stutrek]

EDIT: You can find dexie-encrypted here: https://github.com/mark43/dexie-encrypted

I'm a long time user of Dexie, it's truly a great library, thank you for building and maintaining it.

I was very surprised to that I couldn't find an addon to encrypt as the issues show it to be a common use case. I'm about to try to make this addon, and I would like it to be transparent to the consumer after initial configuration.

This is the strategy I was planning on taking, I am very interested to hear any thoughts you may have.

1. a crypto key is set on the db, probably with something likedb.setCryptoKey()
2. there is a configuration per table that says which paths to encrypt.
3. an addon with Dexie.use() that uses webcrypto as data goes in and out

The part I am most unsure of is upgrading. With Dexie.use(), will it just work?

for (2) it will either have to be a change to to the versions API to pass in an object, a new symbol in the index string, or a method that's called after creating the database.

Option A - my favorite - change stores API to take a config object

var db = new Dexie("FriendDatabase");
db.setCryptoKey(myKey);
db.version(1).stores({
     friends: { keys: "++id,name,age", encrypt: "address,phoneNumber" }
 });

Option B - make a new symbol for keys to encrypt
I used "$" as an example. I have no strong opinions about the symbol.

var db = new Dexie("FriendDatabase");
db.setCryptoKey(myKey);
db.version(1).stores({
     friends: "++id,name,age,$address,$phoneNumber"
 });

Option C - a separate call made after creating the database. This one may make upgrading a bigger challenge because the encryption info doesn't come until later.

var db = new Dexie("FriendDatabase");
db.setCryptoKey(myKey);
db.version(1).stores({
     friends: "++id,name,age"
 });

db.friends.encryptFields(['address', 'phoneNumber'])

[dfahlander]

Nice that you do this. Definitely needed. However the problem with webcrypto is that the API is async so it won't survive transactions (unless the browser vendors have done something exceptional around this).

Regarding upgrade, I would recommend to store a metadata table with the encrypted fields info, and on event db.on("ready") diff the current setting with the stored one to detect if rencryption etc is needed, and if so, provide that under the hood before leaving over to the application.

db.use() is a good choice and a preferred way of hooking in to db operations as it gives full freedom to act as a proxy between the API and the underlying IDB. That said, the middleware api is still not written in stone, but if it changes, your architecture would still remain same so it would be easy to adjust you add-on if so.

Strategies for handling transactional encryption needs to be solved. Some ideas:

• Skip webcrypto and use js implementation of crypto algorithms instead

• Use Dexie.waitFor() when calling webcrypto apis to keep transaction alive.

*

[dfahlander]

Regarding adding a metadata table, this has currently to be done by overriding _parseStoresSpec as done in dexie-observable and dexie-relationships, so you can return a stores-spec that always includes your metadata table.

Something like this:

    ...
    db.Version.prototype._parseStoresSpec = Dexie.override(db.Version.prototype._parseStoresSpec, origFunc => function (stores, dbschema){
  return origFunc.call(this, {
    ...stores,
    _cryptoMeta: 'id' // your added table
  }, dbschema);
});

[stutrek]

I like the idea of storing it in the DB, I was trying to find a way to keep it in configuration so we could push changes out without a full deploy. Were you imagining something like this?

var db = new Dexie("FriendDatabase");
db.setCrypto(myKey, myTableConfigs);

I was planning on using Dexie.waitFor, the downside of CPU usage would happen with a JS implementation of crypto, and I'll be running in a worker anyway.

[dfahlander]

I think you provide the configuration using something like db.setCrypto() but also store that in db so you can diff. Db would reveal the state of the data while config defines the new set of fields to encrypt. A diff would reveal action needed on existing data.

Don't know whether waitFor or js implementation of crypto would be best performance wise. Probably a webassembly based crypto module would be the optimal solution as you are already in a worker you could utilize heavy cpu without slowing down the gui thread. But using webcrypto with waitFor seems the natural, least complex way to start.

[stutrek]

I have a repo up, I'd appreciate any feedback you have. https://github.com/mark43/dexie-encrypted

Notes:

• I used the old hooks, even though I know you're supposed to use the alpha version in production, it's not what you get when you npm install dexie, and I don't like using anything labelled alpha.
• tweetnacl for crypto - it's much faster than WebCrypto and synchronous.
• Instead of modifying dexie you pass in your db. It's like this because package managers often provide different instances to different modules, breaking the magic of extending. I have ran into this with dexie-import-export.

[ejdaly]

In #374 an option was suggested:
_Use WebCrypto, but prepare the encryption before starting an indexedDB transaction_

I can't think of anything inherently wrong with this approach. Is there a benefit to putting encryption / decryption within the indexedDB transaction?

[stutrek]

That's a very good question. The reason I did it this way is because Dexie's hooks are within a transaction. To implement this without hooks you'd have to override all the gets, puts, filters, and more.

Also, tweetnacl is much faster than webcrypto at the moment. Depending on how much of this is because of async overhead, it may be faster for a long time.

[ejdaly]

I'm surprised tweetnacl is so much faster that webcrypto, but I guess it would have an advantage when working on smaller payloads (where the async overhead would be more significant).

To implement this without hooks you'd have to override all the gets, puts, filters, and more

Yeah that's a fair point. I don't think there's any easy way to have the full Dexie (or just IDB) API on an encrypted database without making some trade offs (which is probably why it's difficult to find addons or libraries of this type!)

In your library (which is excellent btw) you get the full API, with the trade off of un-encrypted keys. You could hash the keys (#255) but you lose a lot of the query API functionality. I guess you could encrypt the keys and load them into memory for querying..., and maybe then using Table.bulkGet(), but I think that would require reading and decrypting the full db to build that in-memory index (and would require a lot of additional coding..).

[dfahlander]

@stutrek The library seems to be very easy to use with minimal config and easily gain encrypted db. Awesome work!

[stutrek]

@dfahlander thanks! It would have been much worse without your help.

@ejdaly It would be very cool to encrypt the keys too. You'd still get exact matches if you keep the actual data in the encrypted bundle and use a fast hash algorithm for the IDB index. I don't need this for my particular use case, so it's unlikely that I'll get around to it anytime soon :(.