Keepassxc: Don't error out when duplicate attachments are found

[beks6]

Expected Behavior

Database should open

Current Behavior

Database won't open in KeePassXC on MacOS 10.14.1 with "Duplicate attachment found" error

Possible Solution

Steps to Reproduce (for bugs)

1.
2.
3.
4.

Context

We are using the Database over smb access on several macOS Computer. Opening the database in other programs like KeeWeb or MacPass is working fine. Only KeePassXC is causing this problem.
How can we solve this issue? We also exported the DB to .xml from KeeWeb but searching in a DB with over 100 pw is causing headaches :)

Debug Info

KeePassXC - Version 2.3.4
Revision: 6fe821c

Libraries:

• Qt 5.9.3
• libgcrypt 1.8.2

Operating system: macOS 10.14
CPU architecture: x86_64
Kernel: darwin 18.2.0

Enabled extensions:

• Auto-Type
• Browser Integration
• Legacy Browser Integration (KeePassHTTP)
• SSH Agent
• YubiKey

[droidmonkey]

Here is the offending code block:

https://github.com/keepassxreboot/keepassxc/blob/82bf00de7389db3e52baf11c9295300ad943c643/src/format/KdbxXmlReader.cpp#L873-L881

In my opinion this should not raise an error, only a warning (or perhaps silently remove the duplicate attachment). If it is truly duplicate there won't be data loss.

[beks6]

The point ist we get the warning, but Database won't open.
We investigated also a bit deeper into the problem. The DB is on our share and it seemed the *.kdbx file got corrupted. We even aren't able to configure permissions on the file. Luckily we have version control on our share and have been able to restore the last save. Will inform you if we find more to this problem, but it seems not to be a bug.

But still wondering why KeeWeb or MacPass have been able to open the DB.

Thanks for the fast response

[droidmonkey]

Those other apps are either not doing format checks on open or they are not caring about duplicate attachments. To me its a bug because its an easily fixed error that cannot be fixed because we bail out.

[beks6]

It could also be possible, that this bug is popping up falsely due to a unhandled exception?

We think so because we have no more access to the file, we can't even move it to another folder and we can't manipulate the security privileges. Even with our Admin Domainaccounts we are not able to manipulate the file permissions.

To be mentioned, the DB file is on a Windows share and we are accessing it from macOS through smb.

[phoerious]

We don't use exceptions, but it's possible that the error message itself is misleading. Does it help when you save back the file in one of the other applications that allow you to open it? Perhaps try converting between kdbx3 and kdbx4 (and then back).

[beks6]

Will give it a try tomorrow.

[beks6]

OK guys, it's now getting a serious problem. Yesterday we exported the last working version to another location on the share. Today we made some new entries to the db and now we get again the duplicate warning and are not able to open the database in KeePassXC. I'm wondering what is causing this problem

[phoerious]

Could you build a version where you comment out the line @droidmonkey referenced and then dump the database with keepassxc-cli extract and see if you have any duplicate attachments in the XML data?

[droidmonkey]

Are you using all three apps to access this database? (KeeWeb, MacPass, and KeePassXC)

If you are, it is very likely that one of those apps is duplicating an attachment file incorrectly.

[beks6]

Until last week we had no problems with this usecase. But since last week only KeePassXC is a bit offended and we struggle in finding to root of the problem.

@phoerious could you give me a detailed example how to do this? I tried to check the xml exported from KeeWeb, but the exported xml is not formatted properly and we have more than 100 entries. You could imagine how long it takes to format it properly and check the entries.

Yesterday after restoring the database we already checked our entries for attachments and there are only two or tree entries with an attachment in the database and today nobody added an attachment, just normal pass entries.

[phoerious]

You find build instructions here: https://github.com/keepassxreboot/keepassxc/wiki/Building-KeePassXC

[beks6]

Another maybe important notice, the Windows Version KeePass2 v2.38 is also able to open the database. So it's only affecting KeePassXC, but I can't really say what we changed in our usecases. The last update installation is weeks ago.
Still struggling a bit getting the cli to run :)

[phoerious]

Your database is most likely inconsistent, but the other programs are less strict about it.

[beks6]

ok, I exported now the DB from the Windows client into xml. any hint where to start searching? xml has 57145 rows :)

[phoerious]

Search for <Binaries> or <Binary>.

Quoting our own specification:

   <Binaries> (KDBX 3.1)  A sequence of zero or more "<Binary>" elements
      containing files attached to entries in the database.  Each
      "<Binary>" element is a _BLOB_ containing the file's contents.
      "<Binary>" elements MUST have an "ID" attribute, which is a unique
      _INTEGER_ identifying the _BLOB_ data and a _BOOLEAN_ attribute
      "Compressed", indicating whether the _BLOB_ data is Gzip-
      compressed [RFC1952].
      A database SHOULD NOT contain two "<Binary>" elements with the
      same _BLOB_ data.
      This element is only specified for KDBX 3.1 and lower.  KDBX 4.0
      stores files in its encrypted binary inner header, so this element
      MUST NOT be present if the container format is version 4.0 or
      higher.

You probably have two attachments with the same ID attribute.

[beks6]

OK, I think I've found the bastard. The <Binary> IDs are unique but searching further led to two <Binary>
<Key>Mc_Shark_LIWEST_4061.pdf</Key>
<Value Ref="0" />
</Binary>

entries with identical Key and Value entries. What's best practice resolving this issue?

EDIT: Figured out, one of this entries is in the <History> section

[droidmonkey]

Ahhh, this may be a bug

[droidmonkey]

So the IDs are unique, that is fine. What matters here is that the _BLOB_ sections must be unique as well. Is their duplicate of the actual attachment data?

[beks6]

I assume, the _BLOB_ data is the hash between <Binary ID="0" Compressed="True">_BLOB_</Binary>

If so, I copied the 4 entries each in a separate file (over 300 000 chars), but they seem all to be different

[beks6]

As of now, we deleted the history entry with the attachment referencing the same value id

But still wondering, why this problem popped up now (beginning last week). The history entry is from 08.03.2017!

We'll keep an eye on this

[phoerious]

I don't think that is the bug here. Our specification says that there SHOULD be no two binaries with the same BLOB data and I don't think we throw an error in that case. Having multiple identical binary references is totally fine and normal. If you say that you have four distinct BLOBs in your XML, which are referenced from multiple places, then there is no problem with that. The XML export is also not entirely representative, because the export feature in KeePass may also have fixed the actual problem already, so you don't see it.

[beks6]

KeePassXC is now able to open the DB, but as I already mentioned. I think the warning/message is maybe misleading and caused by another bug.

I'll keep you informed

[rmiller-sgntr]

Hi, I am having this issue as well, working on a shared DB, all my colleagues are using Mac and can use the file, but I am using Ubuntu 18.04 and cannot open it with the duplicate attachment error.

Any progress on this?

[droidmonkey]

Are you ALL using KeePassXC or are some using another application?

[rmiller-sgntr]

My colleagues use MacPass, I get the same error on both KeePassXC and KeePassX (kinda stating the obvious).

[droidmonkey]

I think the root cause of this is MacPass, to be honest. That is the common thread (besides KeePassXC and a team environment) between you and beks6. Why can't everyone just use KeePassXC?

Ah Hah! https://github.com/MacPass/MacPass/issues/788

Also: https://github.com/MacPass/MacPass/issues/857#issuecomment-444090040

[phoerious]

We should still not bail out when that happens, though.

[rmiller-sgntr]

So assuming this issue will resolve once macpass users update to the version with the fix?

[droidmonkey]

You'll need to address that question to the MacPass owner

[droidmonkey]

I fixed this issue, but fwiw this only impacts KDBX 3.x databases. Upgrading to KDBX 4.x will solve the problem.