Realm-cocoa: Deadlock on beginWriteTransaction

[o15a3d4l11s2]

Goals

Begin a write transaction

Expected Results

The transaction is started

Actual Results

Deadlock (not every time, but at the same place)

Steps to Reproduce

Code Sample

[[RLMRealm defaultRealm] transactionWithBlock:block error:nil];

The block itself contains a change of single property.

Version of Realm and Tooling

ProductName:    Mac OS X
ProductVersion: 10.12.4
BuildVersion:   16E195

/Applications/Xcode.app/Contents/Developer
Xcode 8.3.2
Build version 8E2002

/usr/local/bin/pod
1.2.1
Realm (2.6.2)

/bin/bash
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)

/usr/local/bin/carthage
0.16.2
(not in use here)

/usr/local/bin/git
git version 2.10.1

Unfortunately it is close to impossible to send a sample project, but if this might help, I am available for any tests/debugs to do

Might be similar to https://github.com/realm/realm-cocoa/issues/4428

[jpsim]

The stack trace you shared strongly indicates that there's more involved in your app than just a single write transaction. Looks like you also have some notification blocks, which would be significant. What happens when you remove those notification blocks?

Also, are you running more than one process accessing this Realm file at a time?

[kevincador]

We're experiencing the same issue here. There's an unusual long lock when beginning a transaction. We found that when we upgraded to Realm 2.6.2.
The easy fix was to switch to 2.6.1. The bug is not present on 2.6.1 but on 2.6.2 it is present (same code base).

[jpsim]

Have you set up any notification blocks that may be observing changes to objects with circular links? If so, this may be caused by https://github.com/realm/realm-object-store/pull/432

[kevincador]

By circular link you mean an object linking another object which itself links to the first one, right? I think it is the case. We have an object A referencing an object B that itself reference object A.
We have a lot of notification blocks observing a lot of stuff (with rx). So it is possible that we are observing those objects, yes.
What would be your recommendation?

[jpsim]

What would be your recommendation?

To give us a way to reproduce the "unusual long lock" ourselves so we can identify why this is happening.

[TimOliver]

Hi @o15a3d4l11s2 and @kevincador!

A sample app to reproduce this issue would be really valuable to us. Would you be able to isolate your Realm models to show us howe you've set up those circular links?

Thanks!

[semireg]

We can confirm 2.7.0 and 2.6.2 affects us in the same way and yes, 2.6.1 works fine. We are downgrading to 2.6.1 and reevaluating our uses of realm notifications. We will remove them where necessary. This has caused major heartburn with our clients.

[austinzheng]

We're deeply sorry that this issue has caused you problems. Would you be willing to share any information about what the apps are doing when the deadlock occurs, any stack traces, etc? I will attempt to get this escalated with the appropriate engineers as well.

[semireg]

This use case certainly fits the mold of the cyclical parent <--> child relationships. We think we're managing it properly, but the issue with deadlock in notifications makes us less confident.

[austinzheng]

This is great. Is there any chance you can create a reproduction case based on the subset of the code that is causing the problem, so we can poke at a live example?

[semireg]

Unfortunately, no. We're in the same boat as with this - https://github.com/realm/realm-cocoa/issues/4428#issuecomment-281168148

[austinzheng]

Alright, I see you provided some suggestions for creating a similar project. I'll put something together and see what happens.

[austinzheng]

@semireg Thanks for the tips in the linked comment. I put together a test application earlier today and have been playing around with it. Haven't reproduced the deadlock yet, although I understand this type of problem can be tricky to replicate. If I uploaded the project, would you be willing to take a look and see if there is anything blindingly obvious that I need to change to more closely match the conditions of your apps?

[semireg]

You bet.

[austinzheng]

Thank you for your help! You can take a look at the project whenever is convenient. You will have to run pod install to install the dependency. If you prefer I can prepare a version that doesn't use CocoaPods, too.

DeadlockRepro.zip

[semireg]

Working through the code I see that there is use of LinkingObjects. When was LinkingObjects introduced? Our classes date back to October 2014 and use manual inverse relationships. If this gives you enough information to continue working on a repro, great, otherwise, some guidance would be suggested in migrating (in the literal and code sense) away from manual inverse relationships. For example, can we add LinkingObjects and realm can dynamically figure out the parents, or did this declaration need to be here at the time the relationship was created... necessitating a more complex migration?

[austinzheng]

LinkingObjects was introduced about a year ago, in version 0.100. If you add LinkingObject properties to your Realm models and increment your schema version, the backlinks should be calculated automatically without any further intervention on your part. Our documentation goes into a bit more detail.

In terms of the repro case, should I try getting rid of the linking objects and create manual inverse relationships? You did mention circular dependencies, so maybe I should try to set those up (e.g. Person links to Dog, Dog links back to same Person)? Again, many thanks for your time.

[jpsim]

There's no need to increment the schema version when adding linking object properties as those don't affect the representation of the data in the file in any way. LinkingObjects properties are essentially "labels" that name the inverse relationship so that you can query them at runtime.

[semireg]

Sounds good and no problem. I'm glad to help. The way our object hierarchy is setup has one more level of objects. I recommend doing something like: Person has many dogs, dog has many bones. Dog has one owner. Bone's don't know about dogs.

(Fwiw, our setup is: Reading has many Recordings. A Recording has many Measurements. A Recording knows its parentReading. Measurements do not know which Recording they are in)

Additionally, FirstVC could be UITableViewController subclass and the notification should watch the first dog's bones increment in size (count). That's all we're really doing with our TVC that watches a list of recordings and updates the label when measurements get added.

It's worth noting that our code is using a collection (vs. results) based addNotificationBlock.

        _realmToken = [collection addNotificationBlock:^(RLMResults *results, RLMCollectionChange *change, NSError *error) {
            if (error) {
                NSLog(@"Failed to open Realm on background worker: %@", error);
                return;
            }

            [weakSelf collectionChangedWithResults:results change:change];
        }];

and then simply...

-(void)collectionChangedWithResults:(RLMResults*)results change:(RLMCollectionChange*)change {
    UITableView *tv = self.tableView;
    [tv beginUpdates];
    [tv deleteRowsAtIndexPaths:[change deletionsInSection:0] withRowAnimation:UITableViewRowAnimationAutomatic];
    [tv insertRowsAtIndexPaths:[change insertionsInSection:0] withRowAnimation:UITableViewRowAnimationAutomatic];
    [tv reloadRowsAtIndexPaths:[change modificationsInSection:0] withRowAnimation:UITableViewRowAnimationNone];
    [tv endUpdates];
}

[kocherovets]

I have deadlock problem too. It appears in one place almost guaranteed under certain circumstances. When you turn off notifications, it disappears. I can not give you a project, but I can give you a look and debug it on Skype with a Russian-speaking specialist. I use last Realm version.

[Sianay]

I'm also running throught this exact same problem... as long as I remove realm notifications, the problem seems to disappear but this is really frustrating

[weibel]

@austinzheng We also experience this problem in 2.7.0. It is tricky to reproduce the circumstances in a test project as we have a lot of notifications being sent.

The bottom line is that we can end up in a situation where we wait indefinitely for acquiring the lock on m_notifier_mutex inside std::unique_lock<std::mutex> RealmCoordinator::wait_for_notifiers(Pred&& **wait_predicate)**

It's difficult to tell which bit of our code acquired the lock and did not release it. Downgrading to 2.6.1 as @kevincador suggests seems to remove the problem. I'm not familiar enough with the Realm codebase to know where I should set a breakpoint to find out who acquired the lock in the first place. Any hints?

This happens both on beginWriteTransaction and refresh so the issue seems to be somewhat related to https://github.com/realm/realm-cocoa/issues/4768

Hanging forever in beginWriteTransaction or refresh after an error occurs on the notification worker thread is expected, as it's waiting for the notification worker to finish its current set of work

[jpsim]

We need a reproduction case before we can make progress on this. Thanks for your assistance.

[antonkiselev]

I have reproduced this deadlock in a sample project, please check it out. Happens even without notifications, just adds and deletes in main and background threads. It easily reproduces in my real project, which is about to be released.
There is a way to avoid deadlock by moving all work with realm to main thread, but that's not a good way. Hope to hear from you soon.

https://github.com/antonkiselev/realm-2.8-deadlock

[bdash]

Thanks so much for providing the reproducible case. I'll dig into what's happening tomorrow.

[BartBM]

We experience a temporary deadlock problem using Realm 2.6.1 (lock for about 13 seconds). There is a workaround by removing all addNotificationBlock methods.
What I wanted to add is that when we upgrade from realm 2.6.1 to >= 2.6.2 we experience a permanent deadlock and can't get out of this state.

[austinzheng]

Hi @antonkiselev. Thanks again for providing a working reproduction case. We're trialling a new program: to express our appreciation for your time and effort, if you'd like a free Realm T-shirt please email your address and T-shirt size to [email protected], then post back here so we know it's you.

[antonkiselev]

@austinzheng I've sent email few minutes ago. Thanks for supporting community with nice gifts. Waiting for a bug fix in upcoming release :)

[bdash]

As an update, I'm currently investigating this.

[bdash]

@antonkiselev, your issue appears to be different than the problem that was originally reported here.

In your test case, the deadlock occurs when a background thread uses NSAttributedString's HTML parsing functionality from within a Realm write transaction. This HTML parsing delegates to WebKit, which is a main-thread-only API, and waits for the work to complete before returning. At the same time, your main thread is blocked attempting to begin a Realm write transaction. It's unable to do so as a background thread is already within a write transaction. The background thread is also unable to make progress as it requires the main thread to make progress in order to parse the HTML.

My immediate suggestion to address this would be to avoid using NSAtrributedString's HTML parsing functionality within Realm write transactions as you have the potential to hit a deadlock like this if you ever perform a write transaction on the main thread.

---

If anyone else is still able to reproduce the deadlock as originally described and can share a reproducible case with us, we'd be very grateful for it. Feel free to share it privately via email to [email protected].

[BartBM]

@bdash I've emailed a reproducible scenario. Hope it helps!

[weibel]

We have had this issue for a while and have tried different approaches to solve it.

The TL;DR, is that we think we have found a way to circumvent it.
The root cause for the deadlock should still be identified and dealt with inside Realm.

We would have Realm randomly lock up on RealmCoordinator::wait_for_notifiers on either beginWriteTransaction or refresh if there was a Realm notification listener connected to any objects involved in those operations.

First we mitigated the deadlock on beginWriteTransaction by dispatching write transactions to a background queue. We did use an autoreleasepool inside the background work as suggested by the docs. Then after the transaction had finished we would dispatch back to the main queue in case we needed to process any results from the transaction.

Back on the main dispatch queue we would often need to refresh to see the results of the transaction. This is backed up by the following

From https://github.com/realm/realm-cocoa/issues/4837

[…]when dispatch_async’ing to the main thread immediately after a write transaction it's possible for the async block to be dequeued and invoked before Realm's commit notification makes it to the main thread. In this case you may see the Realm as not having refreshed to the latest version, and calling Realm.refresh() is the correct way to deal with that.

This suggest that the auto refresh feature is not synced with the main queue. This could actually be the case. This guy ran some tests https://blackpixel.com/writing/2013/11/performselectoronmainthread-vs-dispatch-async.html and also https://stackoverflow.com/questions/10440412/perform-on-next-run-loop-whats-wrong-with-gcd

At this point refresh would still cause a deadlock with random intervals

This lead us to try to see if we could start execution on the main thread after we knew for sure the Realm had been auto refreshed.

Instead of going with dispatch_async to the main queue we tried using the performSelector family of methods, which are guaranteed to run on the next run loop cycle. So far it's working for us.

This should also mitigate the need to explicitly refresh as auto refresh should have run at this time.

[kharmabum]

Thanks @weibel, much appreciated. FWIW I also alleviated this issue (and improved realm notification related performance) by replacing my object references with primary key references.

#import "NSThread-PerformBlock.h"

@implementation NSThread (PerformBlock)

+ (void)performBlockOnMainThread:(void (^)())block {
    [[NSThread mainThread] orca_performBlock:block];
}

+ (void)runBlock:(void (^)())block{
    block();
}

- (void)performBlock:(void (^)())block {
    if ([[NSThread currentThread] isEqual:self]) {
        block();
    } else {
        [self performBlock:block waitUntilDone:NO];
    }
}
- (void)performBlock:(void (^)())block waitUntilDone:(BOOL)wait {

    [NSThread performSelector:@selector(runBlock:)
                     onThread:self
                   withObject:[block copy]
                waitUntilDone:wait];
}

@end

    fileprivate(set)
    var creator: User? {
        get {
            if isInvalidated { return nil }
            guard let key = _creatorPrimaryKey else { return nil }
            return User.fetch(key)
        }
        set {
            _creatorPrimaryKey = newValue?.key
        }
    }

[nikolsky2]

I've got the same deadlock on Realm Swift 2.9.1 in write transaction. Moving all realm accessing code to main thread, as @antonkiselev said, did not fix the problem. I've tried moving everything to the background threads, as @weibel specified, I got lock inside realm.refresh.

I've used this function for all write transactions on main thread in the app. The first realm write goes through successfully, the second one gets stuck in try self.write(block)

extension Realm {
    func performWriteTransaction(_ block: (() throws -> Void)) {
        precondition(!isInWriteTransaction, "realm.write() was called when isInWriteTransaction == true")
        dispatchPrecondition(condition: .onQueue(.main))

        do {
            try self.write(block)
        } catch {
            print("Realm write error")
        }
    }
}

Also I can add when I disable all notification blocks it doesn't fall in a deadlock.

[plm75]

I'm facing a similar issue where the app "freezes" for quite some time apparently due to the triggering of a batch write operation.

It must be noted that I have never had this issue when running the app in the simulator. Only on real devices.

Can other people in this issue confirm that as well or is this maybe another issue ?

[kharmabum]

@plm75 I did notice that what was appearing as a deadlock was in fact realm notification logic (apparently triggered by the circular references) that would sometimes eventually complete after a very long period of time, O(minutes).

[nikolsky2]

I'm not getting any issues on 2.10.0 anymore. Thank you!

[bdash]

There were changes in v2.10.0 that should mitigate this issue in many cases. I'd love to hear from anyone that's still seeing this after updating to v2.10.0 or newer.

[kharmabum]

Will do! Thank you.

[emuye]

Just ran into a deadlock with several Realm threads trying to write in v2.10.10. I've attached the backtrace. This isn't something that's reproducible but I'm happy to share our schema if that's useful.

backtrace.txt.zip

[bdash]

@emuye, your problem appears to be unrelated to the issue as originally reported here.

Your backtraces show that thread 24 has an open Realm write transaction, and is blocked on a call to dispatch_sync. Similarly, your main thread is blocked on a call to dispatch_sync. Two other background threads appear to be waiting to acquire Realm's write lock, which they cannot do until thread 24 releases it. At least one of these threads looks like it may be executing in the context of the dispatch queue that the various dispatch_sync calls are attempting to target.

This seems like a classic case of inconsistent lock ordering leading to a deadlock. One part of your code acquires Realm's write lock then attempts to dispatch_sync to a specific serial queue, while another part of your code attempts to acquire Realm's write lock while already running on the serial queue.

[emuye]

Thanks so much @bdash! I missed the open Realm write on thread 24. I agree this is unrelated (and completely an issue in our code) :)