Realm-cocoa: Change notifications sometimes miss insertions

[TheHmmka]

Goals

Get notification after every write transaction.

Expected Results

Sometimes I get notification with RLMCollectionChange that contains old value for updates, but RLMResults* result contains up to date data. So I need to have actual RLMCollectionChange to avoid crashes!

Actual Results

What did happened instead?
Let me explain:

...
Receive message
Update UI
Receive message
Update UI
Receive message
Receive message
CRASH!
Invalid update: invalid number of items in section 0. The number of items contained in an existing section after the update (182) must be equal to the number of items contained in that section before the update (180), plus or minus the number of items inserted or deleted from that section (1 inserted, 0 deleted) and plus or minus the number of items moved into or out of that section (0 moved in, 0 moved out).
....

Steps to Reproduce

Don't know how exactly reproduce this issue. I am working with XMPP, so I receive messages:

dispatch_async(MAIN_QUEUE, ^
{
@autoreleasepool
{
NSLog(@"Receive message...");
[The place where I commit write transaction];           
}
});

Than I have notification token:

self.token = [[Message objectsWhere: queryString] addNotificationBlock: ^(RLMResults<Message*>* results, RLMCollectionChange* change, NSError* error)
{
                        NSArray<NSIndexPath*>* insertions = [change insertionsInSection: 0];
            NSArray<NSIndexPath*>* deletions = [change deletionsInSection: 0];
            if (insertions.count > 0)
            {
                weakSelf.dataProvider = results;
                [weakSelf.collectionView insertItemsAtIndexPaths: insertions];

                NSLog(@"Update UI...");
            }

            if (deletions.count > 0)
            {
                weakSelf.dataProvider = results;
                [weakSelf.collectionView deleteItemsAtIndexPaths: deletions];
            }
}]];

And if I receive messages very fast (very very fast) I get crash...

Code Sample

Version of Realm and Tooling

Realm version: 2.0.1

Xcode version: 8

iOS/OSX version: 9

Dependency manager + version: Cocoapods v1.1.0.rc.2

[jpsim]

Could you please include a sample Xcode project that would allow us to replicate this issue? Otherwise it'll be difficult to reproduce this ourselves based on the information you provided.

[kishikawakatsumi]

@TheHmmka For example, moving cells produces deletions and insertions. In that case, you need to wrap insertItemsAtIndexPaths: and deleteItemsAtIndexPaths: in performBatchUpdates:completion: method.

[TheHmmka]

@kishikawakatsumi I understand but I don't need animations.
@jpsim I will try to create a peace of code to recreate this stuff in empty project.

[ab-rdarts]

It seems we have the same issue here. After upgrading to Realm 2.0.2 our application crashes on tableview updates. These crashes disappear after reverting to Realm 1.1.0. For example it has insertion with index 1 while no items in original results

[TheHmmka]

@ab-rdarts Glad that I am not only one, but not cause your problem :-). Still don't have free time to create test project for example :-(.

[AndrewBarba]

Seeing very similar (random) results, and everything is reading from the same results object on the main thread:

Invalid update: invalid number of sections. The number of sections contained in the table view after the update (22) must be equal to the number of sections contained in the table view before the update (21), plus or minus the number of sections inserted or deleted (1 inserted, 1 deleted).

For more context, this is a messaging controller with a Message: Object class. I am seeing this crash rarely when I send a message. It thinks 1 message was inserted but also 1 deleted. There are definitely no deletes happening (I don't support them)

Table view code:

override func numberOfSections(in tableView: UITableView) -> Int {
    return results.count
}

And the animation:

    func animateResultChanges(deletions: [Int], insertions: [Int], modifications: [Int]) {
        guard let tableView = tableView else { return }

        tableView.beginUpdates()

        // Inserts
        var insertIndexSet = IndexSet()
        insertions.forEach { insertIndexSet.insert($0) }
        tableView.insertSections(insertIndexSet, with: insertions.count == 1 ? .top : .automatic)

        // Inserts
        var deleteIndexSet = IndexSet()
        deletions.forEach { deleteIndexSet.insert($0) }
        tableView.deleteSections(deleteIndexSet, with: .automatic)

        // Updates
        var updateIndexSet = IndexSet()
        modifications.forEach { updateIndexSet.insert($0) }
        tableView.reloadSections(updateIndexSet, with: .none)

        tableView.endUpdates()
    }

[nRewik]

We have the issue with Realm's notification after updating to version 2.0.x. I think it might be the same issue.

The problem occurs when realm notification is coalesced. For example, if you create object A and then quickly update object A in the different transaction blocks. Notification block will be called one time.

When I use Realm 1.0.x. RLMCollectionChange has

• insertions.count = 1
• modifications.count = 0
• deletions.count = 0

But after updating to Realm 2.0.x RLMCollectionChange has

• insertions.count = 0
• modifications.count = 1
• deletions.count = 0

[TheHmmka]

@nRewik Yep, thinks it's the same. I have got 1 object in insertions and 1 object in modifications but need to get 2 objects in insertions.

[jpsim]

@TheHmmka @nRewik @AndrewBarba can any of you provide us with a sample Xcode project that reproduces the problem you're facing? We'd love to help, but need to reproduce this to make progress.

[AndrewBarba]

@jpsim Can I send you guys a zip privately? I think it's going to be difficult to create a project from scratch so I'd like to include code from our app that I know can reproduce this. I have a feeling it has to do with the way we write to realms on background threads and only read from them on the main thread and I don't want to lose any of that code, I think it will lead you in a better direction.

[jpsim]

Absolutely, please email a zip of your project to [email protected] and include a link to this issue.

[TheHmmka]

So any movements? :-)

[jpsim]

Still waiting on a sample Xcode project to reproduce this. We received something from @anlaital that may or may not be related to this issue, we're still looking into that, but really the best way for us to move on this would be for people experiencing the issue to share a project that would let us reproduce it too. That'd be you @AndrewBarba and @TheHmmka.

[AndrewBarba]

I'm sorry guys. Halloween in US is one of our busiest times of year. I will get you guys something tomorrow or Wednesday. Side note: writing on background thread and listening for change notifications on main should be an easy way to reproduce. I've had to remove every handler that tried to do table view updates with change notifications because it crashes around 10% of the time. Back to reload data for us. But I will get something over soon

[jpsim]

Side note: writing on background thread and listening for change notifications on main should be an easy way to reproduce

I wish it was that easy! We have lots of tests that follow that pattern in AsyncTests.mm.

[AndrewBarba]

Fair fair. I will get you something this week with UITableView. That's going to be the easiest way for me to get you something that crashes

[nRewik]

Now, I have to use a hacky workaround. By having a gap time between each transaction for 0.2 seconds. Something like this.

transaction_1 -> wait_0.2 -> transaction_2 -> wait_0.2 -> transaction_3

[AndrewBarba]

Okay example project sent, sorry for the delay.

[jpsim]

I was able to trigger the UITableView exception using the project @AndrewBarba sent us, so I've removed the "Reproduction Required" label from this issue.

I have yet to investigate where the problem is: Realm, UITableView or the sample code we received.

[jpsim]

Issue seems to be in Realm. I identified a discrepancy where the count of the collection being observed increased without an accompanying insertion by using the project @AndrewBarba sent us.

It appears that the tg/changeset-skip branch may fix the issue, but keep in mind that those changes are still in progress. Still, I'd appreciate if others experiencing this issue on this thread could try it out.

If you're using CocoaPods, you can build from that branch by specifying the following in your Podfile:

pod 'Realm', git: 'https://github.com/realm/realm-cocoa.git', branch: 'tg/changeset-skip', submodules: true
pod 'RealmSwift', git: 'https://github.com/realm/realm-cocoa.git', branch: 'tg/changeset-skip', submodules: true

[nRewik]

I just try tg/changeset-skip, my app always has realm deadlock. 😂

[jpsim]

That's not too surprising. Work on that branch is still ongoing, which is why it hasn't landed in master yet.

Although it would help us to know how the deadlock is happening. Could you share a sample project that reproduces that @nRewik?

[tgoyne]

I've now fixed a few deadlocks in tg/changeset-skip (and I think there's now actually tests exercising all of the potentially problematic scenarios). It would be helpful if you could retest with the latest version of that branch.

[emuye]

I tried out tg/changeset-skip and still see intermittent crashes updating the tableview after applying the collection changes.

[anlaital]

I talked with @jpsim about us not receiving change notifications on updates when listening on filtered collections. This issue has been fixed in the tg/changeset-skip branch.

[AndrewBarba]

I tested the demo project I sent with the tg/changeset-skip branch and have not seen the crash on that branch. I'm going to test again in our production app later today

[jpsim]

I tried out tg/changeset-skip and still see intermittent crashes updating the tableview after applying the collection changes.

Could you please share a sample project that reproduces the problem?

[emuye]

I sent an email to [email protected] with steps to reproduce this with my application because I was not able to create a reproducible case in isolation.

[pawelkata]

I can confirm the issue still persists when using with collectionViews. My code:

// realm autoupdater
        notificationToken = tickets.addNotificationBlock { [weak self] (changes: RealmCollectionChange) in
            guard let collectionView = self?.collectionView else {
                print("token:invisible")
                return
            }
            switch changes {
            case .initial:
                print("token:reload")
                // Results are now populated and can be accessed without blocking the UI
                collectionView.reloadData()
                break
            case .update(_, let deletions, let insertions, let modifications):
                print("token:update")
                // Query results have changed, so apply them to the UITableView
                collectionView.performBatchUpdates({
                    collectionView.insertItems(at: insertions.map({ IndexPath(row: $0, section: 0) }))
                    collectionView.deleteItems(at: deletions.map({ IndexPath(row: $0, section: 0) }))
                    collectionView.reloadItems(at: modifications.map({ IndexPath(row: $0, section: 0) }))
                }, completion: { status in
                    collectionView.reloadData()
                })
                break
            case .error(let error):
                // An error occurred while opening the Realm file on the background worker thread
                fatalError("\(error)")
                break
            }
        }

The error message says:

*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Invalid update: invalid number of items in section 0.  The number of items contained in an existing section after the update (213) must be equal to the number of items contained in that section before the update (210), plus or minus the number of items inserted or deleted from that section (1 inserted, 1 deleted) and plus or minus the number of items moved into or out of that section (0 moved in, 0 moved out).'

The thing is... this pops up only when app on an app updated from 2.0.3 to 2.0.4. If I do a clean build using v2.4 the problem doesn't exist.

EDIT: The quick fix for the issue is:

case .update(_, let deletions, let insertions, let modifications):
                print("token:update")
                // Query results have changed, so apply them to the UITableView
                collectionView.reloadData()
                break

[jpsim]

@pawelkata have you tried with the tg/changeset-skip branch?

[pawelkata]

@jpsim not yet, sorry. Had to make a release to production and went with v2.0.3 + quick fix mentioned above. I'll test this on Monday and report back.

[jpsim]

Please note that the tg/changeset-skip branch has now been deleted and merged to master in #4253.

We'll be cutting a release including those fixes and features shortly.

[jpsim]

Closing as the problems that this ticket is tracking appear to be resolved with #4253.

@emuye's issue may be different, we're investigating that via email at the moment, and if it turns out to be another bug that was not fixed by the recent improvements, we'll open a new issue.