Keepassxc: Support bidirectional sync with a different database file [$10]

[TheZ3ro]

Expected Behavior

Option to Sync (on changes) from another external file.

Would be nice if with the current database Autoreload and database Merge we should implement an option on KeePassXC to watch on an external source (a file placed on another drive/folder) and when there are changes on that file, merge with the current opened database.

This can be useful with Cloud/Sync client like Dropbox, GDrive, WebDav, so one can listen on an external file and KeePassXC will continuously merge the changes from that file.

This is related to https://github.com/keepassxreboot/keepassxc/issues/22

[droidmonkey]

Implemented in #93

[TheZ3ro]

I don't think #93 can Sync(and merge) from an external file selected by the user (other then the currently opened database file) ?
Or am I wrong?

[droidmonkey]

93 merges if there are unsaved changes to the database. I suppose what this issue is asking for is a "merge by default" behavior which would be trivial to add to the existing autoreload code. Perhaps change the autoreload option to a spin box selector with three options "auto", "ask", "merge"

[TheZ3ro]

What I think it's supposed to do this feature is:
Add an option to the top menu (like the Merge from another database) but instead of selecting a file and merge it, it should be able to select a file, open a filewatcher on it and merge everytime there is a change on that file (and maybe overwrite the file that is been watched).
Also nice if this can be done on multiple file

This will finally fix every behaviour with multiple cloud and sync service.

What do you guys think?

[vith]

You want to open a.kdbx, then select from a menu to watch for changes in b.kdbx? I don't think I understand what problem that would solve or I have misunderstood.

[TheZ3ro]

Current behaviour

With #47 and #92 we have this scenario:

I have a main.kdbx file on my PC, and a cloud.kdbx file in another folder downloaded from a cloud/sync client.
Suppose that the main database have password that must not end into the cloud database.

So I can only address this by:

1. Opening the main.kdbx file, Selecting the Merge option in the menu and merge the changes every time I want to use that database.

1. Using 2 different database (in the case that we supposed this the way to go)

If I want to Merge multiple file I must do that manually every time

Expected Behavior

With an option to "Continuously Merge from another database", KeePassXC will check for changes on selected files and merge them automatically

Notes

This issue was requested some time ago, I'm reporting here what I've understand about it and asking if it's useful or useless to have this option on KeePassXC

This is also a KeePass feature

[droidmonkey]

Synchronization is a feature, which we more or less support. What you are describing is really a combination of features and is described here: https://sourceforge.net/p/keepass/discussion/329220/thread/be5d5787/

One thing to note that we currently don't do, before the db is saved KeePass checks to see if there were changes to the underlying file and asks to merge if so.

[philiplb]

FTP would be awesome, too.

[tohn]

One thing to note that we currently don't do, before the db is saved KeePass checks to see if there were changes to the underlying file and asks to merge if so.

This feature would be awesome! Otherwise I still have to use KeePass to sync several databases ...
Is there any plan to implement this?

[phoerious]

We kind of support that already. When an opened file was changed externally, KeePassXC automatically reloads the changes. What we don't have is an explicit check before saving, but auto reload and merge should usually be quick enough.

[tohn]

Thanks, but that's not what I meant. Let me describe it better:

We have several central databases on a fileshare, and each user has copies of these databases in their homes.
Now I change something in my local copy and save my file, but I also want to include my change to the central database. But in the meantime someone else also changed something (different) in their local copy but synced already (with KeePass) with the central database. If I click on merge, I would assume, that my changes are merged into the central db and their changes are merged into my database.

This works already fine with KeePass (File -> Synchronize -> Synchronize with File...).

[phoerious]

I'm not sure if this isn't a little out-of-scope. On the other hand, KeePass does support something like that apparently.

[jeroen7s]

i would definitely not considder this out of scope, I too have experienced data loss in a similar situation and a "master-local" synchronization scheme would definitely solve this,
(as described on keepass website: here, see how it works section)
it help avoiding data loss on all major cloud providers and FTP, SMB etc... without having to implement a full cloud sync implementation for every cloud provider

When an opened file was changed externally, KeePassXC automatically reloads the changes.

key word is opened, if your dropbox file changed while your keepassdb was not opened you will still lose data, especially if you change the db on multiple systems.

and this is not just in a complex multi-user environment.

implementing such a master-local (or cloud-local, or how you want to call it) synchronization scheme shouldn't be that difficult.
local files sync using keepass with files stored in dropbox folder, and will never be accidentally overwritten by dropbox, the cloud files might be overwritten multiple times, but local files simply sync with it and will never lose data.
this is what keepass2android does by default, so does Keeweb, and so should keepassXC imho, since there's not really any downside in doing this, besides maybe wasting a few extra KBs for having 2 keepass databases.

[jeroen7s]

anyone interested in implementing such a master-local synchronization scheme?
I'd think that preventing data loss would be very high on the priority list.
this weekend is the 4th time I noticed data loss on my Keepass database synced with Dropbox.
it's a real pity cause this makes it impossible for me to recommend KeepassXC to my friends.
I'd like to implement it myself but i know absolutely nothing about coding in C++ and the Qt-framework

[TheZ3ro]

@jeroen7s this is already possible but it's not automatic.
When using KeePassXC: merge the master database with your local database, save and update the master database.
The issue we are discussing here is to make this process all automatic :)

[jeroen7s]

i'm definitely in favor of making it automatic, not many users like to do that stuff manually, just for protection against data corruption, especially non-technical users.

[ajvsol]

This is definitely needed badly, I've been using the aforementioned KeePass sync feature with triggers to sync the master cloud<>local database for several Windows KeePass users.

Now that I have to account for two additional Mac users I naturally found KeePassXC, but the lack of automatic sync makes this unsuitable for a multi-user environment. Currently looking at Passman as an alternative.

[roomcays]

I also cast my vote for this feature as this is the only thing that stops me from moving from KeePass to KeePassXC (Mono < Qt).
How about setting some bounty? I would participate as a donor :)

[tohn]

Since this is still relevant, I created (as suggested by @roomcays) an issue on bountysource: https://www.bountysource.com/issues/39075071-option-to-sync-on-changes-from-another-external-file

[sultanahamer]

I have an idea which makes this implementation easy with the help of syncthing or any out of box syncing solutions. Let KeepassXC store its local version(lets call it cached version) of database file(may be in application space somewhere). Once user opens keepass database, keepass can check signature of both database files and then ask user to merge / ignore cached version. The reason behind this is, as a user, I would not have so many database files so caching should not be a big deal. And this way, as a user I would have only 1 file to manage unlike keepassx solution of sync.

What this solves:

1. changes from server side should be easily merged
2. changes from client side should be merged
3. simultaneous changes in both sides should be mergable unless its a merge conflict(when a single password entry is changed in both sides)

What still remains:

1. User would not be able to select what to keep in case of merge conflict(when a single password entry is changed in both sides) arises. But this should be solved with some small effort of building new UI dialogs show casing changes in both files and let user select what to override with.
2. Moving database file from one directory to other might just recreate another cache copy.

[jeroen7s]

@sultanahamer you're basically discribing how KeeWeb and Keepass2Android handle it

[sultanahamer]

@jeroen7s If it fixes the issue, then it should be alright unless it comes at big cost. I am up for development if someone(dev from xc) guides me a bit in the start up

[TheZ3ro]

This idea seems pretty good but we will need a major code-refactor before implementing something this big

[nopivnick]

if you're running KeePassXC on a Linux box, as a kludge you can install Mono + Keepass.exe + KPScript.exe and run a sync with a simple script.

you'll need to provide your password whether or not you've already got the .kdbx file open in KeePassXC (at least that's been my experience) but either way it's a lot faster than firing up the entire KeePass.exe GUI just to use its sync functionality.

also, I suspect you can forgo installing mono-complete (which is a bit of a beast) since there are no KeePass.exe plugins involved if all you're after is using KeePass.exe to sync.

to automate things a bit more you could probably use inotify-tools to trigger the sync when KeePassXC saves changes to the .kdbx file, though I haven't tried.

this excellent blog post by @publicus breaks down how to properly install KPScript.exe on a Linux box so you can run it from terminal.

again -- presuming you're running KeePassXC on Linux -- the KPScript.exe kludge addresses some of the questions and concerns raised in issues #637, #818, and #841 as well.

fwiw.

EDIT: also #2184.

[publicus]

@droidmonkey, thank you very much for your work on this! And for your and the whole development team's work on the project! I'm really excited to see this in action, and grateful for your work!

[droidmonkey]

It works really well too! The real thanks needs to go to @ckieschnick and HickNHack software who developed this and the advanced merging code to begin with.

[akontsevich]

Whether it works correctly now and exactly like analogous function in original KeePass? Thanks!

[droidmonkey]

It does as long as you compile with the "WITH_XC_KEESHARE_SECURE" setting enabled. Most platforms will have this setting compiled in. The exceptions are macos and ubuntu <= Trusty due to missing libquazip5.

Also, you're very welcome @publicus !

[mstarke]

@droidmonkey Thank you for offering the platform to start developing the KeeShare Extension in the first place.

[m0rphU]

Came here by #637 which is the exact use case I try to replicate with KeeShare now.
tl;dr of that request: Synchronize two identical databases locally, so that one of the copies can go into a cloud synced folder without conflicts.

This works 90% for me now, with one drawback: I can't find an option to import from a *.kdbx database file with a composed master key (password + key file in my case). This means that I would need to leave the cloud copy with less security than the local copy.
Am I missing something here?

[mstarke]

@m0rphU the current system is fixed to passwords only. It is surely a good point to extend in the future.

[droidmonkey]

You can also make your cloud copy with a much longer password. It is effectively the same thing

[m0rphU]

@mstarke Thanks for the clarification. I suppose integrating all options is mainly a GUI thing? With the regular merge options, I can unlock via all regular options.
Choosing a longer password, is currently not an option for me. The cloud copy is mainly used on Android and iOS touchscreen devices.

For now, I'll keep using the cloud version as the primary one on my PC as well. So I have no blocker for migration from KeePass to KeePassXC.
However, I know that this will lead to sync conflicts at one point or another. Then I will see how good merge and KeeShare work ;)

[mstarke]

@droidmonkey You can also make your cloud copy with a much longer password. It is effectively the same thing

Isn't it harder to guess both key file and password instead of only having to guess a (possibly long) password?

[droidmonkey]

In the backend, the master key is formed by appending the SHA256 of the keyfile data (in most cases) to the SHA256 of the password.

https://github.com/keepassxreboot/keepassxc/blob/8bfc53923445a7afe216503a0630c55f01037106/src/keys/FileKey.cpp#L106-L109

So yes, having to guess two SHA-256 values instead of just one is more difficult. But brute forcing a password grows exponentially more difficult with each additional character. Even a 20 random character password (especially with symbols) is impossible to guess in the current lifetime of the universe with a massive cracking exercise.

@m0rphU I know KeePass2Android supports filling the master password with your fingerprint. This could alleviate the need to type in a very long password for the cloud storage every time you use your database.

[mstarke]

I was just about to check my code since I was sure that the key file is hashed most of the time of it's longer than 32 bytes. Otherwise I would have implemented it wrongly in KeePassKit. But I just saw you edited your answer 😬. The problem with using the sync container as a standalone store is that you are severely limiting the options for encryption parameters since it's considers an implementation detail and nothing a user should actually know or change. This is a use case we never considered as vital but you are a good example that it might actually be worthwhile to allow for more options on the sync container.

[akontsevich]

Bidirectional sync does not work: it syncs currently opened DB, but other DB file left untouched. This function does not work like original in KeePass which synchronizes both files. Please reopen and fix, still need to use KeePass for sync. Also why do You ask password for another Db file as it has the same pass by default. KeePass does not ask it.

[ocumo]

I have been looking through so many requests about the exact same thing as I need, with different wording, but always the same intention and very clear to me; however, it keeps being "Closed" with a pointer to a feature that does a totally different thing, a much more complex and convoluted feature that is not even yet documented. I have been jumping this for a couple of hours already. I will try once again in some of the "Open" issues about this same thing, (but with slightly different wording), to see if it would finally be appreciated/understood; but I keep my expectations very low...