Realm-cocoa: writeCopyToPath:encryptionKey:error: creates a file that cannot be opened

[jasper-chan]

I'm trying to use writeCopyToPath:encryptionKey:error: to compress my realm file, but I'm noticing that the copied version cannot be opened. I've triple checked to make sure the encryption key is correct, and I have disabled all functions that use Realm to isolate just the copying process. I have also put the copy function in an autoreleasepool as per some suggestions, but to no avail. The original file can be opened with the key, but the copy cannot.

I have tried this on 0.97.0 and 0.98.0

Thanks

[jasper-chan]

I just wanted to add if I repeat the same exercise without encryption, the realm file opens fine.

[jasper-chan]

Just for more info, I've already posted this issue on stackoverflow and have been in talks with Tim.

http://stackoverflow.com/questions/35297551/realm-file-size-increased-dramatically/35306967?noredirect=1#comment58443843_35306967

[TimOliver]

Sorry for the delay @jasper-chan! I missed this one last week.

In any case, I'm working on an app that incorporates an encrypted realm, so I just added a bit of extra code to save a duplicate of the encrypted file, quit the app, and then open the duplicate.

It worked fine for me, so I'm wondering if this is actually a bug. What kind of data are you trying to store in this Realm file?

Ideally, would you be able to provide enough of your app to us for us to test this out ourselves?

Thanks a lot!

[jasper-chan]

@TimOliver Thanks for replying! So we have a lot of data models, and each of them store an assortment of property types. I think its safe to assume we use every type imaginable.

Short of asking you guys to sign an NDA before giving you guys our source code, or possibly doing a screenshare/Skype session... would it help if I provided you the realm files? IE. the original one and the copied one?

[jasper-chan]

@TimOliver So a little bit of good news, I've been doing a lot of testing today and I think I came to the same conclusion as you, in that it is not the writeCopyToPath:encryptionKey:error: that is causing the problem. However, I am unsure what I could be doing in my code that is causing that function to create a file that cannot be decrypted. Would you be able to take a look at it? I think I've figured out which function it is, but I'm not sure what I could be doing wrong to trigger this problem.

[TimOliver]

Hi @jasper-chan!

Oh! Uh, haha wow okay. That makes it sound like the problem is deeper than we thought.

This might be a complete no-brainer, but as a test, have you tried doing the write to copy at the very beginning of your app delegate's didFinishinLaunchingWithOptions: before any other interactions with your app?

If it still fails after that, that would definitely seem to imply something is wrong with Realm and not your app.

Sure! If you'd like please send your Realm files (and the encryption key in base64) to [email protected] and I'll take a look at them. We'd definitely be okay with signing an NDA to get access to the code as well, but hopefully we can resolve this before needing to go that far.

Thanks!

[jasper-chan]

@TimOliver
So I just tried your test for sanity sake and the outcome is still the same. I'll send you the Realm files with the base64 encryption key at the email you provided for inspection on your end. Hopefully we are able to figure out what the problem is without having to do the whole NDA dance.

Thanks again for your help!

[jasper-chan]

@jpsim May I ask what you are waiting for from me?

[jpsim]

My mistake, I hadn't seen the email you just sent. The ball is in our court now. Thanks for the information!

[TimOliver]

Hi @jasper-chan!

Thanks a lot for sending us those files. I filed it as an internal issue with the engineers working on Realm's core, and heard back immediately that the structure of the copy's header looks very 'inconsistent' with what a properly formatted encrypted Realm should look like.

I'm going to try and write a small sample app to see if I can take the 'original' Realm file and export a functioning copy myself to confirm.

[TimOliver]

Thanks a lot for your patience @jasper-chan!

We had a look at both of your Realm files and have two points of clarification we'd like to follow up:

1) We made some improvements to encryption in Realm 0.98, so we'd recommend you update to the latest version of Realm, and only testing there. I've uploaded a new copy of the Browser incorporating 0.98.2 which you can use to manually open the Realm files: https://github.com/realm/realm-browser-osx/releases/tag/0.98.2

2) How exactly are you generating and storing the encryption key? There's quite a few zeroes in that key, so we're guessing you're not explicitly generating 64 bytes of data for it.

Thanks a lot! Have a nice weekend!

[jasper-chan]

Hi @TimOliver!

1) I just downloaded Realm 0.98.2 and retried the test, but the issue still persisted. I also tried using the new Realm browser you uploaded.

2) The encryption key is a UUID converted to 64 bytes of data via padding zeros at the end. It is currently being stored in the keychain using a third party library called Lockbox (https://github.com/granoff/Lockbox).

[bdash]

@jasper-chan, can you please share the specific steps that you're using to test this? Are you immediately attempting to open the newly-written Realm in the same process? Are you opening it during the next launch of your application? Are you opening it afterwards in the Realm Browser?

One thing we'd like to rule out is an issue that was fixed in v0.97.1. #3075 would result in Realm failing to open encrypted files written using writeCopyToPath:encryptionKey:error due to a problem while reading the files (not writing). The version of Realm Browser in the App Store is using a version of Realm that is affected by this issue, which is why Tim has pointed you at a new version of the Browser.

To expand on Tim's comment about encryption keys: UUIDs are not a secure method for generating an encryption key as they do not contain a sufficient amount of random data. You should consider generating secure random data using SecRandomCopyBytes instead.

[jasper-chan]

@bdash
Steps:
1) Comment out the code writeCopyToPath:encryptionKey:error
2) Clean install my app
3) Function runs to insert objects into my database
4) Stop app
5) Uncomment the code writeCopyToPath:encryptionKey:error
6) Comment out my insertion method call
7) Run app
8) Stop app
9) Attempt to open realm file with realm browser.

Thank you very much for your recommendation for generating random data using SecRandomCopyBytes. I will research into this option some more :)

[bdash]

Can you please clarify:

1. Which version of Realm your app is using when you do this?
2. Which version of Realm Browser you are using in step 9?

Thanks!

[jasper-chan]

@bdash

1. I am using 0.98.2 realm framework.
2. Realm Browser is Version 0.98.2 (55)

[bdash]

Thanks, @jasper-chan. Getting these details nailed down helps us to rule out known issues and focus our investigation. I'll take another look at the encrypted files you provided with this new information in mind.

[jasper-chan]

@bdash Thank you very much! Please let me know if there is anything else I can do to help with your investigation!

[TimOliver]

Hi @jasper-chan!

Just to provide you with another quick update! We managed to determine that this is a legitimate bug on behalf of Realm's internal engine, and we've given it a high priority to be fixed. The timeline should hopefully only be a week or so.

In the meantime, sadly, all I can recommend is you avoid saving a copy of a compressed Realm for the time being. Sorry about that!

[jasper-chan]

@TimOliver
Glad to hear you are able to reproduce the issue! I eagerly await for the fix :)

[jasper-chan]

Hi @TimOliver I was wondering if there is an update with regards to this bug?

[jasper-chan]

@TimOliver @jpsim
I was wondering if there is an update for this bug?

[jpsim]

@jasper-chan thanks for your patience with this issue. We're evaluating a potential fix at the moment, it's undergoing code review and further testing. I can't provide a timeline at the moment. We'll update this issue with more information as soon as we have it.

In the meantime, the only way to definitively avoid this bug is to not use -[RLMRealm writeCopyToPath] or -[RLMRealm compact].

[jasper-chan]

Hey @jpsim @TimOliver @bdash
I hope you all had a restful Easter weekend! I was wondering if there is anything new with regarding this issue? We're getting a little nervous about this problem due to the lack of updates.

[jpsim]

Sorry for the lack of updates, I've just messaged someone on our core team who's been looking into this to see if we've made any progress. We'll update this issue as soon as we know more.

[sdwornik]

Hi @jpsim @TimOliver @bdash
jasper-chan and I are quickly approaching the release date for our app and depend heavily on this fix. It's been two months now and we need to update our own timeline to know if this is something that will be in a release within the next two weeks, or does it now affect our critical path and what your timeline for its release looks like?
Let us know.
Thanks

[jpsim]

A fix for this is currently in review in our core database engine (internal link: realm/realm-core#1638). Barring any problems discovered during the review process, we should be able to include this in a release this week.

[jasper-chan]

Hi @jpsim
We are excited to hear this news! Best of luck to everything and we look forward to this upcoming release!

[mikemee]

Now watching this too and confirming that it seems to be the same problem reported here https://stackoverflow.com/questions/36349216/first-realm-db-can-be-open-encrypted-but-writecopytopath-cannot

The only other insight / question I had from my own further testing was that if I immediately open the copy/compressed file from the same program after compressing it, then I can exit the program and later open it with the key correctly. However, if just copy and immediately exit, then I cannot open it. I was hypothesizing that there might be a background write that hadn't completed. Based on the above though, it seems that perhaps that's not it. _(Fortunately I can ship with the slightly larger version for now - opening it to workaround the problem immediately grows the file.)_

[sdwornik]

Hi @jpsim,
Are you guys on track to releasing the update this week?

[jpsim]

I'd encourage everyone who's been affected by this issue to try again using Realm built from source from the core-0.97.4 branch to see the fixes included there address the problem for you.

[sdwornik]

Thank you @jpsim will give it a try.

[mikemee]

try again using Realm built from source from the core-0.97.4 branch

For those on the realm team (or others with special access), correct? Otherwise I'm not seeing how to do this, but am happy to try it with my repro scenario if it helps out. Or I'll just wait for the next official release, no worries.

[jpsim]

For those on the realm team (or others with special access), correct?

No, this is available publicly. You can build Realm from source if you use CocoaPods, and specify the branch to use in your Podfile:

use_frameworks!
target 'MyApp' do
  pod 'Realm', git: '[email protected]:realm/realm-cocoa.git', branch: 'core-0.97.4'
  pod 'RealmSwift', git: '[email protected]:realm/realm-cocoa.git', branch: 'core-0.97.4'
end

Obviously you can omit the pod 'RealmSwift' line if you're only using Realm Objective-C.

[mikemee]

Ah, got it. Thx. (I had checked the github tags & branches dropdown but not the "105 Branches" link.)

I'm using Carthage, but I'll either work out how to do the same thing there or will switch over to CocoaPods and confirm/deny the fix here (assuming that is useful).

[jpsim]

Even easier with Carthage. Just write this in your Cartfile:

github "realm/realm-cocoa" "core-0.97.4"

[mikemee]

Now works as expected in my scenario that previously had problems. Realm Browser 0.98.6 still doesn't open it (as you likely expect), but with Swift code linked to core-0.97.4 I can now open a copied file successfully now after exiting / copying / all I can think of.

Thanks! Looking forward to the Java update so I can confirm on Android and reduce shipping size by 400k. (But will likely ship with 0.88.3 in the meantime).

PS: need to remove "," from Cartfile example above

[jpsim]

Thanks for testing @mikemee! The fix has now been merged into master, so be sure to point there, rather than the now defunct core-0.97.4 branch, if you have yet to test this out.

[jasper-chan]

@jpsim @bdash @TimOliver
I just tried Realm Framework 0.99 with the new writeCopyToURL function, and our issue is still unresolved. Perhaps we can setup a time to do a Skype chat and show you guys how we produce this problem?

[jpsim]

@jasper-chan sorry to hear the fix included in 0.99 didn't address the issue for you. We'll coordinate on our end to assign someone from our team to take the time to have a Skype call with you to identify steps to reproduce.