Realm-cocoa: Uncontrolled increase of Realm database file size until app crash

[numair]

Goals

I want the database to remain true-to-size

Expected Results

Going in and out of the app should result in no significant change in the size of the database file.

Actual Results

Going in and out of the app repeatedly results in a Realm Incorrect Version Exception Error. The number of records in the database remains constant, but the size of the database begins to rapidly balloon on each stop/start. Eventually the database file size expands beyond 1/2GB and the app crashes on boot because the database file is too big.

Steps to Reproduce

Happens randomly in use of a Realm database in an iOS app v2.3

Code Sample

N/A

Version of Realm and Tooling

Realm framework version: 2.3

Realm Object Server version: N/A

Xcode version: 7.3.1

iOS/OSX version: 10.3

Dependency manager + version: Cocoapods 1.2.x

[numair]

@shuhaodo can you add your experiences here as well?

[shuhaodo]

1. For every read/write operation, a new Realm() instance is created. Since the realm instance is cached for the same thread, we didn't notice performance issue till version 2.3.

2. Also noticed thousands of RealmConfiguration instances were created and release in a short time. In the end there is only one instance persisted in the memory.

This can be reproduced easily with multiple threads rapidly read/write concurrently.

[luna-kx]

Hi @numair. Thanks for reporting this. I believe this is related to a few other issues that we've seen reported but would you be able to share a sample project that demonstrates this issue? One of the Cocoa engineers will follow-up as soon as we can.

[MikeKasperlik]

We experience this (I guess it is the same, if not I will open a separate issue) also in the wild and I'm not sure what causes this. Our app stores data around ~20mb in the realm database at most, but we keep get reports of this:

fatal error: 'try!' expression unexpectedly raised an error: Error Domain=io.realm Code=9 "mmap() failed: Cannot allocate memory size: 3221225472 offset: 0" UserInfo={NSLocalizedDescription=mmap() failed: Cannot allocate memory size: 3221225472 offset: 0, Error Code=9}: file /Library/Caches/com.apple.xbs/Sources/swiftlang/swiftlang-800.0.63/src/swift/stdlib/public/core/ErrorType.swift, line 178

As this happens on app start and only for some users this is hard to track.

At the point of crash we have no open instances of realm and we try to do the following:

func compactRealm() {
        let defaultURL = realmConfig!.fileURL!
        let defaultParentURL = defaultURL.deletingLastPathComponent()
        let compactedURL = defaultParentURL.appendingPathComponent("default-compact.realm")

        if FileManager.default.fileExists(atPath: compactedURL.path) {
            try! FileManager.default.removeItem(at: compactedURL)
        }

        if FileManager.default.fileExists(atPath: defaultURL.path) {
            autoreleasepool {
                let realm = try! Realm(configuration: realmConfig) ## Here it crashes
                try! realm.writeCopy(toFile: compactedURL)
            }

            try! FileManager.default.removeItem(at: defaultURL)
            try! FileManager.default.moveItem(at: compactedURL, to: defaultURL)
        }
    }

The stack trace is also rather unintresting:

Crashed: com.apple.main-thread
0  libswiftCore.dylib             0x1028f4a18 _assertionFailed(StaticString, String, StaticString, UInt, flags : UInt32) -> Never + 164
1  libswiftCore.dylib             0x1029147c8 swift_unexpectedError_merged + 476
2  libswiftCore.dylib             0x1029145d0 swift_errorInMain + 26
3  OurApp                        0x100a61c8c static Database.(compactRealm() -> ()).(closure #1) (Database.swift:75)
4  libswiftObjectiveC.dylib       0x10306972c autoreleasepool<A> (invoking : () throws -> A) throws -> A + 68
5  OurApp                        0x100a6186c static Database.compactRealm() -> () (Database.swift:79)
6  OurApp                        0x100a613dc static Database.setup() -> () (Database.swift:60)
7  OurApp                        0x100ac4318 OurApp.init()
8  OurApp                        0x100ac4d80 static OurApp.setup() -> () (OurApp.swift)
9  OurApp                        0x1001b2d74 specialized AppDelegate.application(UIApplication, didFinishLaunchingWithOptions : [UIApplicationLaunchOptionsKey : Any]?) -> Bool (AppDelegate.swift)
10 OurApp                        0x1001ab970 @objc AppDelegate.application(UIApplication, didFinishLaunchingWithOptions : [UIApplicationLaunchOptionsKey : Any]?) -> Bool (AppDelegate.swift)
11 UIKit                          0x189f472dc -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 380
12 UIKit                          0x18a153800 -[UIApplication _callInitializationDelegatesForMainScene:transitionContext:] + 3452
13 UIKit                          0x18a1592a8 -[UIApplication _runWithMainScene:transitionContext:completion:] + 1684
14 UIKit                          0x18a16dde0 __84-[UIApplication _handleApplicationActivationWithScene:transitionContext:completion:]_block_invoke.3151 + 48
15 UIKit                          0x18a15653c -[UIApplication workspaceDidEndTransaction:] + 168
16 FrontBoardServices             0x18594f884 __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__ + 36
17 FrontBoardServices             0x18594f6f0 -[FBSSerialQueue _performNext] + 176
18 FrontBoardServices             0x18594faa0 -[FBSSerialQueue _performNextFromRunLoopSource] + 56
19 CoreFoundation                 0x183d55424 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
20 CoreFoundation                 0x183d54d94 __CFRunLoopDoSources0 + 540
21 CoreFoundation                 0x183d529a0 __CFRunLoopRun + 744
22 CoreFoundation                 0x183c82d94 CFRunLoopRunSpecific + 424
23 UIKit                          0x189f4045c -[UIApplication _run] + 652
24 UIKit                          0x189f3b130 UIApplicationMain + 208
25 PACEApp                        0x1001112d0 main (AppDelegate.swift:32)
26 libdyld.dylib                  0x182c9159c start + 4

Fabric device stats tells us that there is around 2-5% of ram left when this happens but I think that is due to us trying to allocate GB's of data...

Realm v2.4.4
Swift 3.0.2

I can't reproduce this so sadly no sample project/... If you can give me a starting point I can try to pin this

[MikeKasperlik]

What I also noticed is that we experience this on nearly all device types and the pretended size realm wants to mmap is only one of these four:

• Cannot allocate memory size: 3221225472
• Cannot allocate memory size: 2147483648
• Cannot allocate memory size: 1744830464
• Cannot allocate memory size: 2415919104

As the data we store is highly user dependent it would surprise me if these users have the same database sizes.

[jpsim]

Realm's file size expansion logic is deterministic and expands the file to predictable values, so it's not at all surprising that only a small finite number of sizes are being reported as not able to be allocated.

[hyouuu]

I also experienced this for some users with:
fatal error: 'try!' expression unexpectedly raised an error: Error Domain=io.realm Code=9 "mmap() failed: Cannot allocate memory size: 1744830464 offset: 0" UserInfo={NSLocalizedDescription=mmap() failed: Cannot allocate memory size: 1744830464 offset: 0, Error Code=9}: file /Library/Caches/com.apple.xbs/Sources/swiftlang/swiftlang-802.0.48/src/swift/stdlib/public/core/ErrorType.swift, line 182

What is the waiting for user label for in this case @jpsim ?

[bdash]

Many instances of unexpectedly large file sizes are due to user code unintentionally keeping old versions of data alive within the Realm file. We refer to this as version pinning. It is discussed in the File size and tracking of intermediate versions section of the Realm documentation.

[shuhaodo]

The instructions were followed but the issue only started to happen in realm 2.2 ~ 2.4 version.

[bdash]

I'd really prefer if folks would file their own, separate issues about the problem they're having since it's quite likely the problem is in the manner in which the app is using Realm rather than necessarily something in Realm itself, and that's very difficult to discuss when multiple users whose apps are doing things differently are all chiming in on a single GitHub issue. It's difficult to even determine whether that's the case when multiple people are chiming in on one issue because if I ask a question about the manner in which your app is using Realm, I'm going to get different answers from every person involved. If GitHub supported threaded conversations that may be manageable, but it doesn't so separate issues are what we really need to deal with this in a useful fashion.

[numair]

Hi Mark -- it is clear that a lot of people are having issues with
uncontrolled expansion of the database file size. I hope the Realm team
takes this seriously as we are going to have to migrate off Realm back to
CoreData -- and advise others to do the same -- as there is zero excuse for
a bug that turns an app into a complete lemon (due to crash on startup due
to database file size) in production code.

[bdash]

We do take it seriously, but we need more information in order for this to be actionable. Firstly, we need to understand the manner in which your app is using Realm, and we can't usefully do that for multiple people in a single GitHub issue.

@numair, since you filed this issue we can happily gather information about your situation here. A few questions:

1. Can you reproduce this problem yourself at all, or has it only been reported by users?
2. Can you reproduce it on demand, or does it only appear to happen at random?
3. Can you describe how your application uses Realm with respect to multiple threads. For instance:
   a. Do you _only_ use Realm on the main thread?
   b. Do you use Realm on dispatch queues?
   c. Do you use Realm on NSOperationQueues?
   d. Do you use Realm on background threads that you've created?
   e. On which threads / queues do you write to the Realm?
   f. On which threads / queues do you read from the Realm?
4. Is your Realm file only accessed by a single process at a time? Multiple process may be involved if you have multiple apps in an app group that share a Realm, or if you have application extensions that share a Realm with your main app.
5. How frequently do you write to the Realm?
6. Do you have multiple threads attempting to write to the Realm simultaneously?
7. Have you tested with the most recent version of Realm? v2.5.0 includes a fix for a problem that could lead to unexpected file size growth when you have write contention (i.e., multiple threads attempting to write to the Realm concurrently for an extended period of time).

For other people that are seeing a similar issue, please open a new GitHub issue and provide the information described in the issue template along with answers to these questions.

[shuhaodo]

I have got some additional info:

1. It still happened in v2.5.1
2. It happened to tens of thousands of users, causing a significant drop of DAU, and we can reproduce it easily too.
3. The app uses realm in multiple threads on the main thread and background threads through GCD. We don't use NSOperationQueue.
4. Write happens both on main thread and bg thread, but mainly on bg thread (>98% of the writes)
5. Read happens on both main and bg threads too, about 40% & 60%, it's hard to put an accurate number as we do not limit read access to certain threads.
6. We have app extensions so realm file is in shared group. But the extensions are rarely used.
7. Write to realm very frequently. We are an email app, lots of I/O happen when app comes to foreground.
8. Yes, multiple threads are writing simultaneously.

I will open a new issue as you requested. https://github.com/realm/realm-cocoa/issues/4837

[jpsim]

First, let me reiterate that large Realm files are likely due to holding old versions of data in a background thread for extended periods of time. Realms opened on threads without a runloop don't auto-refresh, so you'll need to take special care to wrap background Realm access in self-contained @autoreleasepool blocks that don't leak any Realm-related types.

However, we understand that debugging these cases can be difficult, and even if you don't leak Realms, file sizes can be larger than you'd expect.

So I'd like to encourage everyone who's experienced large Realm file sizes to try out the new shouldCompactOnLaunch block on Realm.Configuration, available as of today's 2.6 release: https://realm.io/docs/swift/latest/#compacting-realms

I wanted to provide a default implementation so this would be on-by-default, but struggled to find good one-size-fits-all heuristics. I think some form of "compact on launch" should be Realm's default mode of operation. So I look forward to hearing what settings people end up using in practice and what ends up working best. Please share feedback from this so we can confidently enable something like this for everyone out of the box.

[felipericieri]

Hello. Just a hands up, I have the same problem. I'm on a chat app project & I'm handling Realm 100% in the UI Thread now. Will try the v2.6 as recommended (I was already compacting the Realm on every launch though).

[jpsim]

(I was already compacting the Realm on every launch though).

Please be wary of doing this outside of Realm's provided shouldCompactOnLaunch block, for the reasons outlined in the blog post announcing the feature: https://news.realm.io/news/realm-objc-swift-2.6#compact-on-launch

[jpsim]

As best I can tell with the information provided here, our advice remains the same:

1. Avoid pinning older versions of Realm transactions.
2. Use shouldCompactOnLaunch if 1 isn't sufficient or feasible.

[felipericieri]

1. Avoid pinning older versions of Realm transactions.
2. Use shouldCompactOnLaunch if 1 isn't sufficient or feasible.

1. Invalidate used Realm by realm.invalidate()?

[jpsim]

Explicitly calling invalidate() on Realm instances is one way to accomplish option 1.

[kashifshaikh]

Perhaps realm should provide it's own method of performing background realm operations like CoreData e.g. realm.performBackgroundWrite { realm in
realm.insert(object)
}

This way you control over managing background realm instances (auto-releasing them), and getting the chance to commit background writes in batches.

[clystian]

Someone knows any library or implementation of Realms Manager or suggested methods here?

[jeffreyrobert]

@jpsim really appreciate all info! I think I understand the issue now and I like the solutions you proposed. Can you check I've got this right? Maybe helpful to others.

Issue:
Large Realm file is likely caused by pinning old versions (i.e. retained copies of outdated versions of data)
Pinning old versions is caused by instances of Realm in memory which has an outdated view of the data
Lingering instances are caused because of two things:

1. Background threads do not have a run loop. If they did, the Realm would get auto-refreshed at the start of the run loop just like it does on the main thread
2. Objects instantiated in background threads aren't released until the autorelease pool is closed which happens at some point in the future

Solution:
Use an autorelease pool so that instances of Realm in background threads are released as soon as you're done using them.

Question:
I'm confused about point 2 above, if I'm using GCD I expected that objects would be released as soon as the block is executed. Is that not true? Any idea why?

Implementation:
Since I already thought GCD blocks behaved this way (releasing objects as soon as the block is executed) I decided to just try to make it behave this way.
I replaced my calls to DispatchQueue.global().async with a call to my own function

func asyncWithAutoRelease(execute: @escaping (() -> Void)) { async { autoreleasepool { execute() } } }

Follow up question, how does this relate to shouldCompactOnLaunch. If we have this problem in our app, isn't the problem resolved on app launch? Or is it that on app launch the space in the realm is freed but the file is still large and needs to be compacted?

Really appreciate your help, I love Realm and want to keep using it. Just want to make sure I understand best practices.