Keepassdx: wrestling with attachments

[mgkai]

It kind of started in fall, that I noticed attachments (mostly JPG, also PDF) breaking, not immediately but at some point.
The reference to the attachment then is given in red-colored text (normally black) and there is a broken paper clip icon next to it.
Normally you do not have a paper clip icon, only download icon. Does this give any idea what's going on there? TNX for letting me know

In case you wonder, if I download such flagged attachments nevertheless, it will not display in image viewers (or PDF viewer, for PDFs).
Besides, I tried turning off Gzip compression but the issue persists.

[J-Jamet]

The red broken paperclip indicates that there is a problem with your attachment. This cannot be retrieved properly from KeePassDX and is corrupted.

The exception occurs at this point in the code:
https://github.com/Kunzisoft/KeePassDX/blob/4770269f6f99f7b6c61a22a808959ac42f266586/app/src/main/java/com/kunzisoft/keepass/database/file/input/DatabaseInputKDBX.kt#L1004

The attachment is then downloaded bit by bit without base64 decoding.

I recommend that you completely rebuild your database from KeePass2 by exporting the database to XML and re-importing it to eliminate encoding errors.

[LJPapenfort]

I recommend that you completely rebuild your database from KeePass2 by exporting the database to XML and re-importing it to eliminate encoding errors.

Is that a general issue? I realized today, that saving the db in keepassdx leads to random corruption in my attached ssh keys.
Tested it multiple times and keepassdx definitely corrupts attachments.

[J-Jamet]

Is that a general issue? I realized today, that saving the db in keepassdx leads to random corruption in my attached ssh keys.
Tested it multiple times and keepassdx definitely corrupts attachments.

It's really weird, I did a lot of tests without seeing this issue. Can you indicate a specific procedure to reproduce this problem? Because it needs to be resolved very quickly if it does.

[J-Jamet]

This is not a general issue, it only happens if the data flow is broken in one place, I had to manually break it in the tests and for me it only happened if another tool broke the files.

[LJPapenfort]

The database is opened through the nextcloud app, but that never posed an issue with other keepass implementations.
I have seen this lately by using the autofill. After specifying the right entry the information seems to be saved in the db, which then consequently leads to corruption of the keys (in completely unrelated entries).
I have seen this kind of corruption before, but at that time I wasn't aware that it came from this app. Also I'm pretty sure that I wasn't using autofill at that time yet.

[J-Jamet]

So we have to find out exactly what is wrong. Honestly, at the moment I have no idea and cannot reproduce the problem. Which is even more problematic, because the cause has to happen for a specific reason.
Are there any items in your ssh keys that are deleted or redundantly replaced after a save? Can you create a database with no real information with attachments that reproduce the issue? and upload it here?

[LJPapenfort]

Not sure if there is a pattern, I will check that and try to setup a simple test case.

[LJPapenfort]

Ok, this is just confusing:

1. Adding a single named entry to the same folder where the ssh keys reside corrupts one of my keys already.
2. Created a new database, adding a new ssh key and doing the same procedure is fine.
3. Taking my db and stripping everything away, adding the new ssh key and doing the same is also fine.

Seems to be a very weird edge case.

[LJPapenfort]

I recommend that you completely rebuild your database from KeePass2 by exporting the database to XML and re-importing it to eliminate encoding errors.

This actually seems to fix the issue with my db. Can't reproduce the corruption with the reimported version.

[J-Jamet]

Okay, so for me it comes from a corruption in your database header which must be badly formatted.
Since the file data is stored in the header, it only takes one bit of offset in the description of an item size to break the save.
The export and re-import solution allows the file to be completely rebuilt.
(It's as if a hard drive's allocation table was created incorrectly, formatting it makes it work properly.)

I will be able to look for solutions to fix the header dynamically, but that requires a lot more study to detect a bad formatting and it would be necessary to reproduce the bugs which modify the header of the file without completely breaking it.

[mgkai]

Wow, lot of work done here today; thank you! As for me, I did this XML recreation (using the https://keepass.info desktop client), now waiting & checking occasionally.
My routine involved just one database, located on cloud service, and access this database either with KeePass.info's Windows client or with KeePassDX. Maybe there was conflicting access at some point. I also find it hard to structure this setup in a way that the exact breaking step could be more specifically provoked and nailed.
At the moment, I still sync the data via cloud service but use 2 copies, one only accessed by KeePassDX, the other only by beforementioned Windows client.

[mgkai]

Of the 2 copies I created after XML recreation, the desktop-only version is still OK, while the attachments on the Android-only copy (used with KeePassDX exclusively) are broken. I have now created a single-entry database, Android-only, that I can forward for inspection once attachments break again. -- I think breaking of attachments may happen when editing entries in KeePassDX.

[J-Jamet]

Thank you @mgkai for your investigation.
It really bothers me that this happens. Can you upload a version of your database without sensitive information that does not yet have corruption and a simple little procedure to corrupt an attachment. It would be very useful and allow me to identify and solve the problem much faster.

[bluezed]

Crap! I only came to KeepassDX (from Keepass2Android) last week because DX seems to work better with Firefox Autofill. Now I just noticed that ALL my Attachments are corrupted! What the hell happened? How do I get them back!? ☹️

[J-Jamet]

@bluezed At the moment, I can't even reproduce the issue on my side. So the first step is to identify exactly what happened to create a fix. You can help us by creating a reproducible procedure.

[bluezed]

Sorry @J-Jamet I overreacted earlier due to the shock of all my attachments being corrupted. Luckily I could restore it all from backup.

Anyway... I use the latest KeePass2 on PC with OneDrive as the cloud service. It looks like as soon as I make any change to an entry using KeepassDX the attachments become corrupted. I reproduced it with a simple database with only 3 entries. The third one has a text-file attached and when I change the password of that entry and press save then the attachment becomes corrupted.

Attached the database before and after corruption. Password is _secret_.

Databases.zip

I've now changed back to Keepass2Android for the time being...

Hope this helps though as I quite liked the DX app so far...

[J-Jamet]

Thanks, that will help a lot. I will study the problem in depth and of course I will keep you informed of the progress.
Indeed, good idea to use KeePass2Android until the update is available and if you have this issue with your database. Sorry for this problem, I am trying to resolve it as quickly as possible.

[LJPapenfort]

Yep, happened to me again too.
But not sure how to isolate / reproduce the problem.
Hope the example above gives a hint!

[mgkai]

Yep, happened to me again too.
But not sure how to isolate / reproduce the problem.
Hope the example above gives a hint!

On my test database with 1 single entry, I added some text to the notes field of that entry, then locked the database, went back in, and bingo: attachment was broken. On another test database, I changed the password, locked and went back in => attachment broken.
Both of them stored on my Android cell locally, ie neither of them synced to cloud service

[J-Jamet]

I just figured out where the problem came from (In fact there are two different issues in this thread.)

1. The main problem occurs in kdbx3.1 databases in specific cases. There was code left in KeePassDX used for an old KeePass file format that allows you to encrypt a binary contained in a single entry. But this method is normally no longer used with the new formats (kdbx4) and causes conflicts in a particular case (when the protection attribute was present in the definition of a binary https://github.com/keepassxreboot/keepassxc-specs/blob/669af9172dfbae595f90cb6fe6639f62fbd07dbb/kdbx-xml/kdbx4.0-schema.xsd#L261).
   Fix: I therefore force the use of the most efficient method (https://github.com/keepassxreboot/keepassxc-specs/blob/669af9172dfbae595f90cb6fe6639f62fbd07dbb/kdbx-xml/kdbx4.0-schema.xsd#L190) compatible with kdbx3.1 during save which optimizes the storage of binaries in the meta node with references and not in each entry separately (in addition optimizes the storage of binaries and decrease the size of the final file : 2.9kB to 2.4kB)
   databases_binary_828.zip

2. The second issue is the one I had described (https://github.com/Kunzisoft/KeePassDX/issues/828#issuecomment-752648447). In this case, I can't do much, you have to rebuild the database with the technique of XML export / import from KeePass for computer to reformat the file.

I'm making other improvements and creating a new version of the app.

[J-Jamet]

Version 2.9.8 was created specifically to fix this issue.