Keepassxc: Add direct menu option to setup bidirectional sync with another database

[akontsevich]

Summary

Currently bidirectional sync does not work or not implemented. Seems current DB file merging syncs currently opened DB, but other DB file left untouched. This function does not work like original in KeePass which synchronizes both files. Please add the feature or fix it, still need to use KeePass for sync. Also why do You ask password for another DB file as it has the same pass by default. KeePass does not ask it.

Related to #90 and #637.

[droidmonkey]

Bidirectional sync is supported with KeeShare in 2.4.0. See #2109

[akontsevich]

@droidmonkey What is KeeShare, how to use it, see no synchronization there. Very inconvenient comparing to KeePass.

[droidmonkey]

We are almost done with our documentation.

[akontsevich]

@droidmonkey with KeePass I just go to menu, select file - and it is done. Why it can't be done same way in KeePassXC?

[droidmonkey]

We are not aiming to be _EXACTLY_ like KeePass.

[droidmonkey]

I think your request is actually to make a direct menu action that sets up a synchronization with some other file. This can be achieved with KeeShare on the root group, but not from the file menu. If you edit the root group you can setup a sync with some other database file.

[akontsevich]

We are not aiming to be EXACTLY like KeePass.

@droidmonkey Why not to take best from it? I started to use KeePassXC because it worked more convenient than KeePass: Ctrl-C, tray icon, native KDE look and feel, Hi/Low/DPI, etc. Now with recent changes it is not - forced to get back to KeePass.

If you edit the root group you can setup a sync with some other database file.

Very complex and inconvenient, need to setup KeeShare first. Why not reopen this ticket and implement for many many users which want this simple feature?

[akontsevich]

Awesome! Thanks!!! :)

[janedough1331]

I second this. Very inconvenient. And i recall last time i tried using keepassXC i loved it, but this was really the nail in the coffin that made me go back to keepass2

[droidmonkey]

Lol what? The nail in the coffin was a one time setup function that is fully accessible through the group edit??

[Dannniello]

Maybe I'm missing something, but KeeShare Synchronize it is much different option than Synchronize in KeePass.

KeeShare Type: Synchronize is for constant database sharing, i.e. every save will export data to file provided in path. If path is not available - save will generate error with information that export failed.

In my case I need manual "one time only" synchronize option - exactly the same like it is working in KeePass. Why? Because I use it to synchronize database between my laptop and desktop from time to time.

Workaround - use merge command two times:

keepassxc-cli merge -s /path/to/desktop/database.kdbx /path/to/laptop/database.kdbx
keepassxc-cli merge -s /path/to/laptop/database.kdbx /path/to/desktop/database.kdbx

It works, but it is not perfect - files after that will have different checksum... KeePass Synchronize is generating "perfectly" synchronized files (the same checksum).

[droidmonkey]

KeeShare is an entirely different feature than synchronize the database itself.

[dmrzzz]

I feel much enlightened by @Dannniello's comment; I found this issue months ago while looking for a KeePassXC operation called "synchronize" (by analogy with KeePass), but never even noticed the "merge" operation until now.

I get the impression that what OP and several others really want here is not even KeeShare Synchronize at all, but rather a menu action that performs a very (conceptually) simple one-time two-way merge using the same master key.

I think part of the difficulty is that this area of KeePassXC feels a bit like hunting squirrels with tactical nuclear weapons (to borrow a favorite quote from a former teacher). The KeeShare feature and its documentation in QUICKSTART.md are geared toward the complex use case of sharing a subset of your credentials with other people, which is pretty amazing, but not everybody needs that, and it's not at all obvious at first glance how to apply this to the much simpler use case of propagating changes between what we logically think of as two copies of the "same" database (no subsets, no other people, no different keys). Possible solution: a new quickstart section "Using Merge" which covers how to merge changes one-way from database A into database B (and what this means for item history), how to do a two-way merge that updates both A and B, and then how to automatically synchronize them.

[akontsevich]

hunting squirrels with tactical nuclear weapons

LOL @dmrzzz good point! Need to make soft for people needs I think. I think topic has 2 superfluous words: KeeShare is not the only way to implement the feature.

[ocumo]

Thanks @dmrzzz for putting the issue so clear. I share 100% of your points.

The fact is, as so many people trying to move to KeePassXC from KeePass2, I have the exact same need of a bidirectional sync with another database, a very common use case that has been so very well described by others: I just need to sync my database between my laptop and my desktop. I do this with seamlessly with KeePass2, as explained by others. I would like to use KeePassXC, though, but without this being possible except in a convoluted way, either by the "killing a fly with a cannon" or with an inconvenient and risky operation of merging and then copying the merged file over the other, that brings no advantages.

Now that I have finally convinced my wife to use a password manager, I won't even try to explain to her that to sync her laptop database with that in her desktop PC, she would have to start learning how to fiddle with bash commands or to learn a complicated concept of group sharing, which is not even remotely intuitively suggesting "synchronization", other than apparently it happen as some kind of secondary effect as long as one configures certain options in a special way (with no documentation available -at least it has taken me a lot of time of searching and reading to no avail).

After reading that this request has been moved from the v2.5.0 milestone, I truly hope that that doesn't mean the issue will be closed now?

Please dear devs, consider the use case of simple users that are not developers and that need a practical solution for the simple problem that all these people before me have explained so well. As soon as this feature is available, there will be many people that will start using this fantastic fork of KeePass/X/....

Thanks a lot anyway for a fantastic work so far!

[droidmonkey]

@ocumo how are you syncing databases between your phone and computer? Are you using webDAV or some other similar technique? I simply use cloud file storage (One Drive, Google, Dropbox, etc) and that works 100% of the time. Our merge algorithm is very robust.

[akontsevich]

droidmonkey added this to the v2.6.0 milestone 3 minutes ago

Thanks a lot!!!

[ocumo]

@droidmonkey, I do not have a satisfactory solution yet for my phone, although I have been using keepass2Android, but I simply do the silly way: many times I just copy the kdbx file to the phone and though that's one-way only, I mostly have a "phone is read only" silly policy. Other times I have sync with the KeePass2 but that requires a bit of command line work, so it's faster (though dumber) the other way. But I should say that the phone is not a priority for me so far, as long as I find a consistent solution first for all PCs at home.

[ocumo]

By the way, thank you so much for the milestone!

[phoerious]

KeePass2Android can open directly from WebDAV, Dropbox, etc. Changes made to the database on the phone are immediately synced back to your cloud storage. Haven't had any issues with that.

[srguglielmo]

I just wanted to add my two cents. Up until a few weeks ago, I used KeePass 2 (the original project) across Windows, Mac (via mono), and Linux (via mono). All three on an almost daily basis. When MacOS removed support for 32-bit applications (required to run KeePass via mono) recently, I discovered KeePassXC. I love that it runs natively. However, switching to it not only broke my syncing workflow, but makes it very difficult to convince lesser-technical people (like my family and friends) to use.

For my syncing workflow, I have a separate kdbx file stored on each of my devices, plus one in a cloud storage account. The file in cloud storage is my _"source of truth"_. My sync process is:

• Open the local file (e.g. macbook.kdbx) in KeePass.
• Download cloudStorage.kdbx.
• Perform a sync.
• Upload cloudStorage.kdbx (overwriting the original).

Sometimes, I make a change to Site A on my MacBook and a change on Site B from my Linux desktop. I want to keep _both_ changes. I can't do that anymore without opening one file, running a merge, opening the other file, running another merge.

We are not aiming to be _EXACTLY_ like KeePass.

I understand, but two-way sync simply _makes sense_ here. It's not "trying to be like KeePass", it's just being user friendly. It isn't logical to default to one-way sync without a big red bold warning about potential data loss.

Thank you for re-opening and re-evaluating this request!

[droidmonkey]

To be clear, macbook.kdbx == cloudStorage.kdbx? If so, why go through complications and just have one file? Whatever mainstream cloud storage solution you are using is bound to have version integrity.

You mention lesser technical users, but your sync setup is very complex (relatively speaking). Whereby if you just dropped your kdbx into iCloud or Dropbox you'd be 100% functional with KeePassXC.

What I am trying to say is that the _process_ that people are using bi-directional merge/sync for is what is broken. In almost all cases that have been presented to us it is easily solved with a modern process.

[dmrzzz]

Whatever mainstream cloud storage solution you are using is bound to have version integrity.

I'm glad you personally trust your cloud storage provider to be 100% bug free and never choke on a corner case. I don't want to trust mine that much. I've detailed my concerns in https://github.com/keepassxreboot/keepassxc/issues/2747#issuecomment-539714108 but they are equally relevant to this discussion.

In any case, it seems clear that quite a few KeePassXC users would really like to have a simple two-way merge feature, because it would let us use the processes _we_ prefer.

And AFAICT the hard work (the back-end code to perform history merges) is already there -- all that's missing is a single front-end operation which either

• merges A into B and merges B into A

or perhaps

• merges A into B and then saves the result to both A and B (this would make the final checksum match, which @Dannniello had implied a preference for)

Is there some downside to providing such an operation in KeePassXC?

[droidmonkey]

I'm for the front-end operation capability, but it's important to not narrow the perspective too much into it being the ONLY solution to a problem.

[seafox]

You mention lesser technical users, but your sync setup is very complex (relatively speaking). Whereby if you just dropped your kdbx into iCloud or Dropbox you'd be 100% functional with KeePassXC.

The use case you constantly suggesting has several drawbacks already mentioned elsewhere, just to reiterate:

1. Requires relying on and installation of external 3-rd party (commercial, big company or custom installation) software that may not be desirable otherwise.
2. Normal autosync feature of those 3-rd party software directory synchronizers has the potential for data loss as they obviously can't perform KP database merge operations. Doing extra (auto)backups is just complicating issues further, as growing complexity increase the chance of user/operator multiply files handling mistakes.
3. Working offline means no sync, but ability to have flexible extra measures for security, when integrated with triggers/events/scripts on database opening/closing, allows much cleaner solutions without involving 3-rd party folders sync software. So it's just more appropriate for many folks screaming and begging around here!
4. Also one particular feature that KeePass has is the ability to use internal database fields as the source for shared master database location and credentials makes "shared master - local copies" sync use case much more robust (just setup once and distribute) and secure (no need to leave plain open master share database location, last opened/synced files, etc. in KPXC config file)! [Note that all copies of such database sync solution are precisely identical on master and local clients, irrespectively of underlying OS.] So the question is whether KPXC dev team wants to provide some implementation making that possible in KPXC also? If they do, I'm afraid the simple menu option isn't enough to be automated even as KeePass does...

What I am trying to say is that the _process_ that people are using bi-directional merge/sync for is what is broken. In almost all cases that have been presented to us it is easily solved with a modern process.

Wording _modern process_ is a fancy way to push the certain use case over others, see above...

I'm for the front-end operation capability, ...

The counter-arguments were presented...

[ocumo]

Reading the well structured and reasoned post by @seafox, I think there's not much else that I could add, really. Thanks for that. I subscribe all of it.

Still, in my opinion, --and making an effort not to repeat @seafox comments--, the situation I see is very akin as many infamous cases whereby some argue that their "new" way is somewhat superior just because it's different/more innovative/impressive than the "other" way, which is somewhat "wrong".

Sounds like the never ending story of one side committed to more-or-less "introduce" some kind of new paradigm, and the other side not perceiving/buying the need for or advantage in questioning their way. Unfortunately, when that is the case, things not always turn for the best for any of the parties. This is especially the case where the two things, which are _not_ mutually exclusive (no, they are not, far from it!), are being debated _as if they were_, either by one of the parties or both of them.

The world of technology -and software in particular- is full of infamous, ridiculous examples of huge mistakes that started with a "_Everybody is doing it wrong. People have to change and do it my way_" kind of genius inspiration. That's really very, _very_ tricky in life.

What I am trying to say is that the process that people are using bi-directional merge/sync for is what is broken. In almost all cases that have been presented to us it is easily solved with a modern process.

"Modern process" is indeed an unfortunate and super-abused expression that is more common in other areas absolutely extraneous to Engineering, such as Politics or Marketing, where speech "flourishes and thrives" with that kind of stuff. Not so much here: I'm pretty sure none of us here are in those capacities, thankfully. That's a blessing, really, because those expressions are empty containers; things like: "_good quality_", or "_everybody thinks_", etc., we can move on and dismiss those as mistakes; they don't fit in.

I think it's reasonably obvious that there are a significant group of people, arguably at the very least developers in some form or shape, from various parts of the World, that randomly landed in this thread for the exact same reason, only one reason, same argumentation, same or incredibly similar way of resolving an important, common usability issue, ... what are the odds? Really. Think a little bit.

Look at my case. I learned about KeePassXC by total accident, just about 22 hours ago!. I was reading about the changelog of Firefox 70. My eyes catched something lateral about the new version of the Tails distribution moving to KeyPassXC because it's more/better maintained than their previous choice. _"Huh?... another KeePass version?" "Wot, ..Better!?" "The Tor guys say so!?" "woah! I take it! Now! Here I go!!"_

And here I am, with the _exact_ same set of needs and opinions about this excellent program, as a big number of folks with so many different backgrounds, from who knows which different parts of the Planet. There must be something worth to consider about this instead of entering religious argumentation about who's more "modern" or "agile" or "cloudy" or whatever.

I call this one word: Opportunity.

--

[dssouza-ti]

Hi everyone, I'm just another one who would love to see this bidirectional sync option implemented and landed on this issue. Currently that's the only thing preventing me to move 100% from mono keepass2 to keepassxc.

@ocumo @seafox, that's extremely good argumentation, couldn't agree more.

By the way, I use Linux and tried keeshare, merge and sync with google drive and one drive, but none of them works the way bidirectional sync works in keepass2.

I have a team at work that uses the same database offline in their computers, and we sync with a online database that is in one of our servers. Every method cited above couldn't achieve perfectly what a simple click in the "synchronize" option can in keepass2. Each one had a different issue that caused some data loss or some inconvenience in our use case and I'll open issues reporting that later, but the main point is, It's great that we have several ways to sync our databases, yet a so simple option that can help so many people is lacking here and that makes a huge difference to a lot of use cases.

I just love KeePassXC and want so badly to move my entire team from keepass2 to it, but as long as we can't achieve the same sync without problems, we just can't move on.

[droidmonkey]

@dssouza-ti please update to 2.5.0 if you haven't already. It adds support for detecting changes over network shares.

[dssouza-ti]

@droidmonkey I already updated. I was waiting for #3612 to be implemented to run some tests.

As I said, I found some inconveniences (maybe some bugs?) that I'll properly report in a separate issue so we can discuss it without adding unrelated comments to this thread.

By the way, as I also said before, I love KeePassXC, congratulations on the great work! I really am looking forward to the moment I can move my entire team to this software.

[droidmonkey]

Agree thank you for reporting, would like to fix those for 2.5.1

[dssouza-ti]

@droidmonkey I just reported the issues I found in #3790 and #3791

[roberestarkk]

To add to seafox's fantastic response:

1. Normal autosync feature of those 3-rd party software directory synchronizers has the potential for data loss as they obviously can't perform KP database merge operations. Doing extra (auto)backups is just complicating issues further, as growing complexity increase the chance of user/operator multiply files handling mistakes.

This is my main concern. To my knowledge, the way cloud storage providers work is they go:
c# if(remote.modifiedDate > local.modifiedDate) { local.REPLACEWITHREMOTE(); } else if (remote.modifiedDate < local.modifiedDate) { remote.REPLACEWITHLOCAL(); } else { thumb.sitOn(); }
The cloud provider doesn't do any merging, it's just random binary as far as it knows.
Graphical Example

Which means, if I open either of the cloud-managed local files as my local source of truth, it can be pulled out from under me, like the metaphorical rug, should a merge conflict arise.
In the old KeePass2 days, I'd have a separate local solution, whose ONLY method of contact with anything cloud related was with the KP Sync option.
ie:
/home/me/documents/THEDATABASE.kdbx (Open in KP)
/home/me/dropbox/THEDATABASE.kdbx (periodically 'Properly' synced at the KP level, with triggers and sync function, and blind-synced with the cloud)

That way, the cloud files are guaranteed to reach eventual consistency, because the true-local copies are only being changed by the KP-aware sync
Graphical Example

[droidmonkey]

You missed a step. The cloud service will create a conflict file if there is a recent change locally that did not get synced with the remote.

[roberestarkk]

That's true.
But it doesn't really change what I'm saying unless KeePassXC has a mechanism to detect and merge those extra files.

Just add "and an extra file is created in the cloud folder" to each of my examples at the 'dumb sync' stage.
You'll still have one scenario where your actual database has dropped an entry (or entries), and one where it hasn't.

You could add a series of steps to the 'non KP Sync' method for manually resolving those files, except the whole point of computers is to automate boring repetitive tasks like that.
So each individual user would wind up either having to switch back to KeePass2, or run both side-by-side using KP2 as the 'glue' (which is what I'm having to do at the moment), or custom script some way to detect and merge those conflict files.

Or, the most simple and elegant solution. We pester the KPXC devs across multiple GH issues, in the hope they'll recognise the need for such a feature, and we can get it implemented properly and in a single place where everyone can benefit from it.

[seafox]

@roberestarkk Thanks for taking time and efforts to continue pushing KPXC devs towards more robust common sense sync solution! Juggling with both KP & KPXC apps would be a bit messy workflow here, so atm I'm using just KP. We've got some more time to wait for a proper sync ;-)

[aisbergde]

bidirectional merge = sync
this is the most important feature for me.

I use Keepass files as a kind of GIT-repository: I can use copies of the keepass database files at the same time to add or edit different entries and I can merge them all together.

Storing files in a folder under cloud control is not enough. I had a lot of issues using this before sync was possible in Keepass. Files can be changed on two devices at the same time. Cloud services will show a "conflict" and some entries can be lost. Only the sync option will allow to change different items in different environments at the same time.