Keepassdx: Out of memory on trying to open database saved by keepassxc

[wheybags]

Hi,
I'm getting an oom when I try to open my database saved by KeepassXC, no matter what database settings I use. I have seen https://github.com/Kunzisoft/KeePassDX/issues/154 but I think some better workaround should be found, as the number of dbs that fail to open will only increase with time.
If I create a db with default settings in the latest version of keepassxc (2.4.1 at time of writing), it loads fine, but when I import all my old data into it, it stops working.
If I open the db and resave it with original Keepass2 version 2.41 and set encryption algo to AES/Rijndael and derivation function to AES-KDF, then it works, but If I open and re-save in keepassxc it breaks again.
I gather the issue is that KeepassDX tries to allocate a block of memory in one go that is too large so some android system heuristics kick in and deny it?
Is there any way to diable this on a per-app basis, or maybe chunk it into a series of smaller allocations?

[ladidadida]

I am facing the same issue since I added exactly one more password.

Do you need any log-data? Unfortunately I have not found out how to create/collect log data.

[lgrahl]

Same here. Happy to help as well.

[J-Jamet]

Can you upload a database that causes the problem but works with KeePass 2 and KeePassXC? It will allow me to see more easily where the problem is coming from.

[lgrahl]

I have to kindly refuse that and hope you understand. Even encrypted, I don't want to hand out this database. I'm no stranger to Android app development if that helps.

[J-Jamet]

It is obvious that I do not specifically require your database with sensitive data but any database that has the problem to debug it.

[lgrahl]

I'd certainly do that if I had another database with that problem. :slightly_smiling_face:

Anyway, I made a quick build and here's the stack trace:

E/com.kunzisoft.keepass.database.action.LoadDatabaseRunnable: No memory to load your entire database.
    java.lang.OutOfMemoryError: Failed to allocate a 1750217576 byte allocation with 3427143 free bytes and 188MB until OOM, max allowed footprint 6854287, growth limit 201326592
        at com.kunzisoft.keepass.stream.LEDataInputStream.readBytes(LEDataInputStream.java:108)
        at com.kunzisoft.keepass.database.load.ImporterV4.ReadInnerHeader(ImporterV4.java:228)
        at com.kunzisoft.keepass.database.load.ImporterV4.LoadInnerHeader(ImporterV4.java:211)
        at com.kunzisoft.keepass.database.load.ImporterV4.openDatabase(ImporterV4.java:183)
        at com.kunzisoft.keepass.database.load.ImporterV4.openDatabase(ImporterV4.java:77)
        at com.kunzisoft.keepass.database.Database.loadData(Database.java:161)
        at com.kunzisoft.keepass.database.Database.passUrisAsInputStreams(Database.java:139)
        at com.kunzisoft.keepass.database.Database.loadData(Database.java:121)
        at com.kunzisoft.keepass.database.Database.loadData(Database.java:110)
        at com.kunzisoft.keepass.database.action.LoadDatabaseRunnable.run(LoadDatabaseRunnable.java:71)
        at java.lang.Thread.run(Thread.java:764)

Edit: size is indeed 1750217576 in ReadInnerHeader which is odd.

[ladidadida]

I tried to create a new database with enough entries to get the same behavior, but it is not that easy. My orginal faulty db is 28kB of size, the new one I created for testing is now 40KB. But the new one, also created with KeepassXC can now be loaded by KeepassDX without any problem.

I have pasted my new keepass-db together with keefile in one zip file. The password is 'abc123!!'

@
keepassfiles.zip

[theScrabi]

Youp happens to me to now. Just one more key and im out of memory :)

[lgrahl]

Is there a spec for the database/header format? Then, I might be able to investigate this with my database. I couldn't find a spec when I was writing a small KeePass Python library a few years ago.

[J-Jamet]

@ladidadida Thank you for your test database, I will debug with it, it's perfect!
@lgrahl https://gist.github.com/lgg/e6ccc6e212d18dd2ecd8a8c116fb1e45 (or best to look in the original KeePass source code)
I will check this once I have completed the migration to AndroidX and integration of the new fingerprint API. Sorry for this issue anyway.

[J-Jamet]

@ladidadida I can't replicate the issue with your database, it works very well with KeePass DX on my devices. (Even on my Galaxy S2 I9100 an Galaxy S I9000 with low memory, it only takes time)

[J-Jamet]

~According to what's written @wheybags, I think the problem comes from one or more entries imported from an old database, there may be an unrecognized format or a TAG xml that is not closed, or an object of history that is not good, hard to say. KeePass XC may use a binary save method that takes a lot of memory during opening with KeePass DX. For files I have already made a stream system that saves the data temporarily on the internal memory so no need to allocate per block of memory.~
False hypothesis

[ladidadida]

@ladidadida I can't replicate the issue with your database, it works very well with KeePass DX on my devices. (Even on my Galaxy S2 I9100 an Galaxy S I9000 with low memory, it only takes time)

Hi @J-Jamet, maybe my post was a little confusing. I wanted to state, that the database I uploaded is OK. I was NOT able to generate a DB that shows the same behaviour even though it already contained more entries than my real faulty DB.

[J-Jamet]

Hi @ladidadida I did not understand that, sorry.
If we can not reproduce the anomaly it is problematic. If the problem comes from a data formatting error that is dynamically fixed on desktop apps, it will be very hard to fix (especially if I can't debug :/).

[J-Jamet]

I think I solved the problem. The utility methods in MemUtil still used Java code. After converting the file to Kotlin, I could reproduce the OOM, so I improved the copyStream method that did not handle a buffer for the InputStream. I think some virtual machines automatically compensated the memory by an automatic buffer. You will tell me if it is resolved for you in the next version 2.5.0.0beta23 or if you compile the develop branch.

[J-Jamet]

Normally corrected, tell me if it's ok with version 2.5.0.0beta23

[leviwheatcroft]

I'm still having this issue with 2.5.0.0beta24

Reluctant to share the db it's occurring with.

The db is indeed generated / edited by KeePassXC, downloaded to android via nextcloud.

Anything I can do to help track this down ?

Edit: actually I just tried the workaround offered by @PeteS4 in #350 - (adding another blank key in KeePassXC) which seems to have resolved this.

[lgrahl]

Sorry for radio silence. Also still an issue here and I can offer debugging.

[theScrabi]

By the way. I had this issue before. Turns out this happens when you have files saved in your database. I removed some files than it worked again. Now when it happens again just remove some more files.

I suggest to not store files in keepass databases but instead put them into an encrypted volume like veracrypt or crypted 7zip.

[leviwheatcroft]

@theScrabi can you be sure that the files are actually the problem? I mean, when this occurs, it seems like any edit to the database in KeePassXC might resolve it. So removing a file might work just because it's an update, it doesn't necessarily mean that the file was the problem.

[theScrabi]

I could look for the original issue. for me it compleatly solved the issue. My database is now working since weeks again. If I would just change it without removing files it would just work until i change it again. Try it for your self.

[J-Jamet]

These are the theoretical cases of OOM that I just referenced. Since the error can occur in many places and I can not reproduce each case, I'm updating the code to be as clean as possible and you will tell me in the next update if the problem is still present in your specific case:

• The parser is an XML pull and the nextText() method that retrieves data from the XML may not work properly if there is a large amount of data to retrieve. (No real solution, maybe use a parser SAX but less fast and forced to redo all the parser.)
• The data is only exchanged with byte arrays, which uses a large amount of information for the transfer. (Better to use stream with buffer and store temp files in app storage) (Completed development)
• Zip files are only done on the fly by converting byte array. (Which can not be done with large volumes, solution is to use stream) (Completed development)
• Binary data is not stored in the same way for KDBX3.1 or KDBX4.0 files, which can cause bugs.(I added the version number of the KDBX file used in the settings and try to harmonize the binary storage but it's rather complicated. https://github.com/keepassxreboot/keepassxc-specs/blob/master/kdbx-xml/rfc.md) (Completed development)
• The Argon2 key derivation with a large RAM set in KeePass XC prevents proper opening. (solution is to lower the memory used and increase the iterations. Maybe a fix for Argon2 can be done, if anyone can chack that, it would be cool)

[J-Jamet]

I made a first code improvement pass but there is still work. I'm going to centralize these elements in a single new issue.

[J-Jamet]

I have improved the system for recognizing variables in C which are no longer cast without verification. I also updated BouncyCastle.
Those who had the Out Of Memory problem, can you tell me if it is resolved with version 2.5RC2?
In any case, I will try to improve the threads with coroutines.

[J-Jamet]

Since I don't have an answer, I think the problem is solved for you. So I close the issue. If someone encounters the problem again, please open a new issue and link the number.

[TheLonelyGhost]

FWIW, I haven't had OOM issues for the past few updates. LGTM!

[J-Jamet]

@TheLonelyGhost Very good! :+1: (when I saw the notification of your message, I thought there was a problem. :face_with_thermometer:)